31st International Conference on Computers in Education Conference Proceedings
Volume I

Editors:
Ju-Ling SHIH, Akihiro KASHIHARA, Weiqin CHEN, Hiroaki OGATA

Publisher:
Asia-Pacific Society for Computers in Education (APSCE)
Address: Center for Science and Technology for Learning,
National Central University, No. 300, Jhongda Road, Zongli District, Taoyuan
City 32001, Taiwan
Telephone: +886-3-4227151 ext. 35407
Fax: +886-3-4227151 ext. 35407
Email: service@apsce.net
Website: http://www.apsce.net

Date of publication: December, 2023

ISBN: 978-626-968-901-9

Copyright 2023 Asia-Pacific Society for Computers in Education.

All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, transmitted, in any forms or any means, without the prior permission of the
Asia-Pacific Society for Computers in Education. Individual papers may be uploaded
on to institutional repositories or other academic sites for self-archival purposes.

EDITORS
Ju-Ling SHIH, National Central University, Taiwan
Akihiro KASHIHARA, University of Electro-Communications, Japan
Weiqin CHEN, Oslo Metropolitan University and University of Bergen, Norway
Hiroaki OGATA, Kyoto University, Japani

ASSOCIATE EDITORS(in alphabetical order)
Ryan BAKER University of Pennsylvania, USA
Ben CHANG National Central University, Taiwan
Seb DIANATI Charles Darwin University, Australia
Jayakrishnan MADATHIL Indian Institute of Technology Madras, India
Ahmed Mohamed Fahmy YOUSEF Fayoum University, Egypt
Yuqin YANG Central China Normal University, China
Hafed ZARZOUR Souk Ahras University, Algeriaii

MESSAGE FROM THE CONFERENCE CHAIR

Weiqin CHEN
Conference Chair
Oslo Metropolitan University and University of Bergen, Norway

On behalf of the organizing committee, I would like to welcome
all participants of the 31st International Conference on
Computers in Education (ICCE) 2023, the flagship conference
series of the Asia-Pacific Society for Computers in Education(APSCE). After the virtual conferences in 2020 and 2021 and the
hybrid conference in 2022, we have come back to in-person
conference this year.
ICCE is no stranger to Japan. After having successfully hosted ICCE 1999, 2007 and 2014, Japanis once again the host for ICCE, this time in Matsue, the "City of Water". Matsue, withmagnificent castle, beautiful gardens, and breath-taking sunset over Lake Shinji, willundoubtedly give participants a unique experience. The conference theme of ICCE 2023,“Designing new technologies for education in a big social change world” signifies the role oftechnological innovation and adoption in transforming education and addressing societal
challenges.
Four outstanding keynote speakers will share their insights across varying areas in the field ofcomputers in education. Curtis J. Bonk from Indiana University, USA, will focus on achieving
smarter and more innovative forms of learning where digital technologies provide learners of
all ages with open, informal, adaptive, nontraditional, and self-directed learning opportunities.
Tak-Wai Chan from National Central University, Taiwan, will share with us his vision of “Global
Harwell” as the ultimate educational goal and how Seamless Interest-Driven Co-Creator
Theory (SIDC) can contribute to achieving this goal. Davinia Hernández-Leo from Universitat
Pompeu Fabra, Barcelona, Spain, will explicate how technology can support learning design
and the orchestration of complex learning scenarios and thus improve the efficiency andeffectiveness of teachers’ tasks. Masaru Kitsuregawa from the Research Organization ofInformation and Systems, Japan will talk about the research data management platform,
GakuNin RDM, that provides support for publishing and sharing of big data including
educational data, which has strong implications for learning analytics and educational
datamining. There will also be three equally inspiring theme-based invited speeches. Kaushal
Kumar Bhagat from Indian Institute of Technology Kharagpur, India, will present the potential
benefits of game-based learning and how it can be used to create engaging and effective
learning experiences. Brendan Flanagan from Kyoto University, Japan, will discuss challenges
and opportunities of educational data science focusing on reading systems. Daner Sun from
Education University of Hong Kong will talk about the evolution of mobile learningiii

environments and share insights gained from her experiences in research and teaching. These
speeches connect with the essence of the conference theme in different ways and will
stimulate reflections and inspire us to rethink the design of digital technologies and theirimpacts on education and the society.
Indeed, organizing such a large-scale conference requires the orchestrated efforts andunwavering support from the conference organizing committee members and conference
paper reviewers. I would like to express my sincere appreciations to all the individuals who
have rendered their help in every possible way to make this conference a reality. The namesof the hard-working Local Organizing Committee (LOC) chair and team members, International
Program Coordination (IPC) chairs, Sub-conference chairs, Program Committee (PC) members
and reviewers, chairs and organizers of Workshops, Interactive Events, Tutorials, Panels,
Work-In-Progress Posters (WIPP), Doctoral Student Consortium (DSC), Early Career
Workshops (ECW), Executive Summary (ES), APSCE Merit Scholarship, and Showcase of
Advancements in Technology-Enhanced Learning in Underrepresented Countries (SATELUC)
are enlisted in the proceedings. I am also grateful to all the paper authors and registeredparticipants for their exciting academic contributions to the fruitful intellectual exchange inthis conference.
Last but not list, I would like to express my heartfelt appreciation to the Managing Secretaryof APSCE Pham-Duc Tho for his support, the standing committee for being flexible andproactive, and the consultants for sharing their experiences and wisdom and advising us along
the way.
I hope all participants will have opportunities to renew friendships, forge new friendships andprofessional collaborations. I trust that you will have a productive and fun-filled time at thisvery special conference and leave Matsue —a picturesque city with rich and remarkable
heritage—with beautiful, affectionate memories.

Thank you!

iv

MESSAGE FROM
THE INT’L PROGRAM COORDINATION CHAIRS

Ju-Ling SHIH

Akihiro KASHIHARA

Program Coordination Chair
National Central University, Taiwan

Program Coordination Co-Chair
University of Electro-Communications,Japan

The International Conference on Computers in Education (ICCE) is an annual conference series
encompassing a broad range of issues related to using Information and Communication
Technology (ICT) for education, organized by the Asia-Pacific Society for Computers in
Education (APSCE). ICCE 2023 takes place at Matsue, Shimane prefecture, Japan from
December 4-8, 2023. It aims to bring together researchers from all over the world to shareand exchange research and to develop and deploy new ideas that span the field of Computersin Education.
Following the tradition of previous conferences in this series, ICCE 2023 is organized as a metaconference, where there are seven Sub-Conferences, each of which focuses on specialized
themes. Each Sub-Conference is organized by a program committee appointed by therespective Special Interest Group (SIG – see https://apsce.net/sigs). These Sub-Conferences
are:⚫

C1: ICCE Sub-Conference on Artificial Intelligence in Education/Intelligent Tutoring
System (AIED/ITS)⚫

C2: ICCE Sub-Conference on Computer-supported Collaborative Learning (CSCL) and
Learning Sciences (LS)⚫

C3: ICCE Sub-Conference on Advanced Learning Technologies (ALT), Learning Analytics,
Platforms and Infrastructure⚫

C4: ICCE Sub-Conference on Classroom, Ubiquitous, and Mobile Technologies Enhanced
Learning (CUMTEL)⚫

C5: ICCE Sub-Conference on Educational Gamification and Game-based Learning (EGG)v

⚫

C6: ICCE Sub-Conference on Technology Enhanced Language Learning (TELL)⚫

C7: ICCE Sub-Conference on Practice-driven Research, Teacher Professional
Development and Policy of ICT in Education (PTP)The International Program Committee is led by a strong and dedicated team, which includes
the Conference Chair, the Program Coordination Chair and Co-Chair, Sub-Conference Chairs
and Co-Chairs and experts in the field of Computers in Education from many different
countries or economies. Former ICCE local organizing and program coordination chairs haveplayed important roles as consultants in overseeing the organization process of thisconference.
The conference received a total of 256 papers (192 full, 44 short, and 20 posters) from 26
different countries or economies. Table 1 provides the submissions by the country of the firstauthor in each paper.
Table 1. Submission Statistics (based on first author’s country)
Japan
TaiwanChina
India
Hong Kong
Philippines
Singapore
Malaysia
United States

67
44
41
19
17
15
13
6
6

Countries or Economies
Thailand5
Poland4Australia
2
Indonesia
2
Spain2
Tunisia2
Canada1
Croatia1
Ecuador1

FranceGermany
Italy
New ZealandNigeria
NorwayTurkeyViet Nam

1
1
1
1
1
1
1
1

All papers were subjected to a rigorous review process by 3 to 5 reviewers from the respective
Sub-Conference program committees. After the reviews were completed, a meta-review wasprovided for each paper. In total, 740 reviews and meta-reviews were received. After a
discussion period within the individual program committees led by the Sub-Conference
Executive Chairs and Co-Chairs, recommendations were made to the Program Coordination
Committee Chair and Co-Chair, who oversaw the review process and quality for all SubConferences.
This resulted in 44 full, 67 short, and 48 poster acceptances across all of the seven SubConferences. The overall acceptance rate for full papers is 22.92%. The acceptance rate forthe full papers in the individual Sub-Conference closely mirrored the overall acceptance rate.
This is a testimony to the continued maintenance of the quality of presentations in ourconference. The complete statistics of paper acceptance is shown in Table 2.

vi

Submitted as
Full Only

Accepted as
Full

Full Only(%)

Accepted as
Short

Accepted as
PosterOverall
Accepted(%)

C1 - AIED/ITS
C2 - CSCL/LSC3 - ALT/LA/PI
C4 - CUMTEL
C5 - EGG
C6 - TELL
C7 - PTPICCE 2023

Total
Submissions

Table 2. Paper Acceptance Statistics

46
36
49
19
33
27
46
256

37
26
39
14
29
18
29
192

8
6
10
2
7
4
7
44

21.62%
23.08%
25.64%
14.29%
24.14%
22.22%
24.14%
22.92%

5
11
13
5
7
12
14
67

12
8
8
3
7
4
6
48

54.35%
69.44%
63.27%
52.63%
63.64%
74.07%
58.70%
62.11%In addition to full papers, short papers and posters, ICCE 2023 includes various program
components, such as Keynote Speeches, Theme-based Invited Speeches, Workshops,
Interactive Events, Panels, Work-in-Progress Posters (WIPP), Extended Summary (ES),Doctoral Student Consortia (DSC), and Early Career Workshop (ECW). All the papers in these
program components are published in separate proceedings with their own ISBN numbers.
Pre-conference events are held on the first two days of the conference, including 13
workshops, 4 Interactive Events, DSC, ECW, and APSCE Student Wing Workshop.
We would like to thank all who have contributed to making ICCE 2023 a successful conference.
First of all, we would like to thank all paper authors for your contributions and for choosing
ICCE 2023 as an outlet to present your research. We would also like to thank the IPC Executive
Chairs/Co-Chairs and members, who undertook the responsibility of reviewing and selecting
papers that represent research of high quality. Specially thanks to our Keynote and Invited
Speakers for accepting our invitations and bring inspiring research to ICCE 2023 participants.
The Local Organizing Committee deserves a big thank you for their hard work under thetremendous time pressure.
We hope that all participants will find the activities in ICCE 2023 interesting and inspiring, andhave opportunities to meet old friends and establish new professional collaborations.
Furthermore, we hope that participants will enjoy not only the academic activities, but also
the vibrant and exciting culture experience at Matsue, Shimane prefecture, Japan.

vii

MESSAGE FROM
THE LOCAL ORGANIZING COMMITTEE CHAIR
Hiroaki OGATALOC Chair
Kyoto University, Japan

Kon’nichiwa 😊 (Hello!)
On behalf of the local organising committee, I would like to extend my warm welcome to alldelegates of the 31st International Conference on Computers in Education (ICCE 2023), held
for the fourth time in Japan (the first time in Chiba in 1999, the second time in Hiroshima in
2007, and the third time in Nara in 2014). It is my great pleasure and honor to host ICCE 2023
as it takes place fully in-person this year. The theme of the conference, “Designing newtechnologies for education in a big social change world,” aptly reflects what we had
experienced during COVID-19, and digital technologies adoption including AI and big data hastaken an exponential leap for transforming education in the new norm.
It is a great privilege to share our beautiful country with you. I hope you will be able to enjoy
Matsue city, which is often referred to as Little Kyoto. Known as the “City of Water”, Matsue
stands by the Sea of Japan where Lake Shinji and Nakaumi meet, in the middle of Shimane
Peninsula. A former feudal stronghold, Matsue is a true castle town crossed with many canals
and boasts one of the twelve remaining original castles in Japan, and famous for its beautiful
sunsets over Lake Shinji. In addition, Matsue is the birthplace of Japanese culture and originof conventions in Japanese Mythology.
I would like to thank the APSCE Executive Committee for giving us this wonderful opportunity.
Our sincere thanks to the standing committee, the International Program Committee,
reviewers, authors, participants and student volunteers. Especially, I would like to express mygratitude to all the local organizing committee members and sponsors: Uchida Yoko Co. Ltd.,
Photron Limited, Research Council of Evidence-Driven Education, and Learning and
Educational Technologies Research Unit, Kyoto University, Japan. Also, this event is supported
by Allied Telesis K.K., Digital Knowledge Co., Ltd., IPSJ, IEICE, JAEIS, JSAI, JSET, JSiSE, JASLA, andShimane University, Japan. We trust all of you will enjoy the conference, and take home a lotof great memories from Matsue city in Shimane prefecture, Japan.
Arigato! 😊 (Thank you!)viii

ORGANIZATION
Standing committee
Conference Chair
Weiqin CHEN,

Oslo Metropolitan University and University of Bergen, Norway

International Program Coordination Chair
Ju-Ling SHIH,
National Central University, Taiwan
International Program Coordination Co-Chair
Akihiro KASHIHARA,
University of Electro-Communications, Japan
Local Organizing Committee Chair
Hiroaki OGATA,
Kyoto University, Japan
Consultants
Sridhar IYER,
Mas Nida MD KHAMBARI,
Lung Hsiang WONG,

Indian Institute of Technology, India
Universiti Putra Malaysia, Malaysia
Nanyang Technological University, Singapore

Sub Conferences
C1: ICCE Sub-Conference on Artificial Intelligence in
Education/Intelligent Tutoring System (AIED/ITS) and Adaptive
Learning
PC Executive Chair
Ryan BAKER,

University of Pennsylvania, USA

PC Co-Chair
Sébastien LALLÉ,

Sorbonne University, France

SIG Chair
May Marie TALANDRON-FELIPE,

University of Science and Technology, Phillipines

C2: ICCE Sub-Conference on Computer-supported Collaborative
Learning (CSCL) and Learning Sciences
PC Executive Chair
Ben CHANG,

National Central University, Taiwan

PC Co-Chair
Daniel BODEMER,

University of Duisburg Essen, Germany

PC Co-Chair
Gaoxia ZHU,

National Institute of Education, Singapore

SIG Chair
Elizabeth KOH,

National Institute of Education, SingaporeixC3: ICCE Sub-Conference on Advanced Learning Technologies (ALT),
Learning Analytics and Digital Infrastructure
PC Executive Chair
Seb DIANATI,Charles Darwin University, Australia

PC Co-Chair
Zablon PINGO,Charles Darwin University, Australia

PC Co-Chair
Suman LAUDARI,

University of Technology Sydney, Australia

PC Co-Chair
Ashwin DIXIT,

Indian Institute of Technology Bombay, India

SIG Chair
Ramkumar RAJENDRAN,

Indian Institute of Technology Bombay, India

C4: ICCE Sub-Conference on Classroom, Ubiquitous, and Mobile
Technologies Enhanced Learning (CUMTEL)
PC Executive Chair
Yuqin YANG,

Central China Normal University, China

PC Co-Chair
Daner SUN,

Education University of Hong Kong, Hongkong

PC Co-Chair
Gaoxia ZHU,

Nanyang Technological University, Singapore

PC Co-Chair
Lu-Ho HSIA,

National Chin-Yi University of Technology, Taiwan

SIG Chair
Daner SUN,

Education University of Hong Kong, Hongkong

C5: ICCE Sub-Conference on Educational Gamification and Gamebased Learning (EGG)
PC Executive Chair
Hafed ZARZOUR,

Souk Ahras University, Algeria

PC Co-Chair
Jewoong MOON,

The University of Alabama, USA

PC Co-Chair
Junfeng YANG,

Hangzhou Normal University, China

PC Co-Chair
Luiz RODRIGUES

Center for Excellence in Social Technologies – NEES, UFAL, Brazilx

SIG Chair
Ahmed TLILI,

Beijing Normal University, China

C6: ICCE Sub-Conference on Technology Enhanced Language Learning(TELL)PC Executive Chair
Ahmed Mohamed Fahmy YOUSEF, Fayoum University, Egypt
PC Co-Chair
Ahmed Hosny Saleh METWALLY,

Beijing Normal University, China

SIG Chair
Vivian WU,

Asia University, Taiwan

C7: ICCE Sub-Conference on Practice-driven Research, Teacher
Professional Development and Policy of ICT in Education (PTP)PC Executive Chair
Jayakrishnan MADATHIL,

Indian Institute of Technology Madras, India

PC Co-Chair
Rotem ISREL-FISHELSON,

University of Maryland, USA

PC Co-Chair
Shitanshu MISHRA,UNESCO MGIEP, India

SIG Chair
Mas Nida KHAMBARI,

Universiti Putra Malaysia, Malaysia

Workshop/Tutorial/Interactive Event
PC Executive Chair
Di ZOU,The Education University of Hong Kong

PC Co-Chair
Junjie Gavin WU,

Shenzhen Technology University

PC Co-Chair
Rui LI,

Hunan university

PC Co-ChairZi YANG,

Xiamen University

WIPP
PC Executive Chair
Wen YUN,

Nanyang Technological University, Singapore

PC Co-Chairxi

Zhang SI,

Central China Normal University, China

DSC
PC Executive Chair
Vwen Yen Alwyn LEE,

Nanyang Technological University, Singapore

PC Co-Chair
Yanjie SONG,

The Education University of Hong Kong

PC Co-Chair
Jayakrishnan MADATHIL,

Indian Institute of Technology Madras, India

PC Co-Chair
Bo JIANG,

East China Normal University, China

ECW
PC Executive Chair
Hui-Chun CHU,

Soochow University, Taiwan

PC Co-Chair
Chengjiu YIN,

Kyushu University, JapanPC Co-Chair
Kai-Hsiang YANG,

National Taipei University of Education (NTUE)ESPC Executive Chair
Juan ZHOU,Tokyo Institute of Technology, JapanPC Co-Chair
Ruining YANG,

Hunan University, China

Merit Scholarships
PC Executive Chair
Nur Aira ABD RAHIM,

Universiti Putra Malaysia, Malaysia

PC Co-Chair
Ting-Chia HSU,

National Taiwan Normal University, Taiwan

PC Co-Chair
Niwat SRISAWASDI,

Khon Kaen University, Thailand

SATELUCPC Executive Chair
Lung-Hsiang WONG,

Nanyang Technological University, SingaporexiiPC Co-Chair
Mas Nida KHAMBARI,

Universiti Putra Malaysia, Malaysia

PC Co-Chair
Ivica BOTICKI,

University of Zagreb, Crotia

PC Co-Chair
May-Marie TALANDRON-FELIPE,
Philippines, Philippines
PC Co-Chair
Saida ULFA,

University of Science and Technology of Southern

State University of Malang, Indonesia

xiiiLocal Organizing Committee
Local Organizing Chair
Hiroaki OGATA,ACCMS, Kyoto University

Local Coordinator
Tatsunori MATSUI,

Waseda University

Registration Coordinator
Yusuke HAYASHI,

Hiroshima University

Social Event Coordinator
Tomoko KOJIRI,

Kansai University

Printing / Publication Coordinator
Chengjiu YIN,
Kyushu University
Web Site Management/ Social Media Coordinator
Hiroyuki MITSUHARA,
Tokushima University
Technical Support (Physical/Virtual) Coordinator
Naka GOTODA,
Kagawa University
Sponsorship Coordinator
Noboru OKUBO,UCHIDA YOKO Co., Ltd.

Financial CoordinatorIzumi HORIKOSHI,

Kyoto University

Secretary
Noriko UOSAKI,

Kyoto University

APSCE Advisors
Riichiro MIZOGUCHI,

Japan Advanced Institute of Science and Technology (JAIST)

Akihiro KASHIHARA,

The University of Electro-Communications

Masanori SUGIMOTO,

Hokkaido University

Tomoko KOJIRI,

Kansai University

Members
Emmanuel AyedounKansai University

Yiling DAI

Kyoto University

Brendan John FLANAGAN

Kyoto UniversityxivToshitaka FUKAMI

Shimane University

Yoshiko GODA

Kumamoto University

Shinobu HASEGAWA

Japan Advanced Institute of Science and Technology (JAIST)

Toshihiro HAYASHI

Kagawa University

Yugo HAYASHI

Ritsumeikan University

Yuki HAYASHI

Osaka Metropolitan University

Tsukasa HIRASHIMA

Hiroshima University

Tomoya HORIGUCHI

Kobe University

Tatsuya HORITA

Tohoku University

Rieko INABA

Tsuda University

Tomoo INOUE

University of Tsukuba

Kazuhide KANENISHI

Tokushima University

Yasuhisa KATO

Tokyo Online University

Toshihiro KITA

Kumamoto University

Kento KOIKE

Kyoto University

Yusuke KOMETANI

Kagawa University

Tatsuhiro KONISHI

Shizuoka University

Hidenobu KUNICHIKA

Kyushu Institute of Technology

Fusako KUSUNOKITama Art University

Yukie MAJIMA

Osaka Metropolitan University

Rwitajit MAJUMDAR

Kumamoto University

Hiroyuki MASUKAWA

University of the Sacred Heart, Tokyo

Hideya MATSUKAWA

Tohoku University

Takahiko MENDORIKochi University of Technology

Tadashi MISONO

Shimane University

Takashi MITSUISHI

Tohoku University

Makoto MIYAZAKI

Teikyo University

Toshio MOCHIZUKI

Senshu University

Yusuke MORITA

Waseda University

Kousuke MOURI

Hiroshima City University

Masayuki MURAKAMI

Osaka University

Keiichi MURAMATSU

Waseda University

Toru NAGAHAMA

Tohoku University

Kae Nakaya

Tokyo Woman’s Christian University

Junko NEMOTO

Meiji Gakuin University

Chifumi NISHIOKANational Institute of Informatics
xv

Youji OCHI

Kindai University

Yuichi OHKAWA

Tohoku University

Tomoya OHYAMA

Tohoku University

Fumiya OKUBO

Kyushu University

Kiichiro OKUBO

Shimane University

Jun OSHIMA

Shizuoka University

Kazuhisa SETA

Osaka Metropolitan University

Katsusuke SHIGETA

Hokkaido University

Takanori SHIGIUCHIDA YOKO Co., Ltd.

Atsushi SHIMADA

Kyushu University

Masato SOGA

Wakayama University

Takahiro SUMIYA

Hiroshima University

Katsuaki SUZUKI

Kumamoto University

Hideyuki TAKADA

Ritsumeikan University

Kyosuke TAKAMI

Kyoto University

Ryo TAKAOKA

Yamaguchi University

Yasuhisa TAMURA

Sophia University

Yoshimasa TAWATSUJI

Waseda University

Takahito TOMOTOTokyo Polytechnic University

Hiroshi UEDA

Hosei University

Maomi UENOThe University of Electro-Communications

Kenzi WATANABE

Hiroshima University

Yuki WATANABETokyo University of Science

Masanori YAMADA

Kyushu University

Yasuhiro YAMADA

Shimane University

Sho YAMAMOTO

Kindai University

Kazuaki YOSHIHARA

Kindai University

xviSpecial Interest Groups(SIG) 2022-2023
SIG 1: Artificial Intelligence in Education/Intelligent Tutoring Systems/Adaptive Learning(AIED/ITS/AL)
May Marie TALANDRON-FELIPE, University of Science and Technology, Phillipines
SIG 2: Computer-supported Collaborative Learning and Learning Sciences (CSCL)
Elizabeth KOH, Nanyang Technological University, Singapore
SIG 3: Advanced Learning Technologies, Platforms and Infrastructure (ALT)
Eunice SARI, UX, Indonesia
SIG 4: Classroom, Ubiquitous and Mobile Technologies Enhanced Learning
Daner SUN, The Education University of Hong Kong, Hong Kong
SIG 5: Educational Gamification and Game-based Learning (EGG)
TLILI, Beijing Normal University, China
SIG 6: Technology Enhanced Language Learning (TELL)
Vivian WU, Asia University, Taiwan
SIG 7: Practice-driven Research, Teacher Professional Development and Policy of ICT in
Education (PTP)
Mas NidaMDKHAMBARI, Universiti Putra Malaysia, Malaysia
SIG 8: Development of Information and Communication Technology in the Asia-Pacific
Neighborhood (DICTAP)
Patcharin PANJABUREE, Mahidol University, Thailand
SIG 9: Educational Use of Problems/Questions in Technology-Enhanced Learning (EUPQ)
Takahito TOMOTO, Tokyo PolytechnicUniversity, Japan
SIG 10: Learning Analytics and Educational Data Mining (LAEDM)
Ramkumar RAJENDRAN, Indian Institute of Technology Bombay, India
SIG 11: Computational Thinking Education & STEM Education (CTE&STEM)
Chee Kit LOOI, Nanyang Technological University, SingaporexviiC1: AIED/ITS PC Member
Ange Adrienne Nyamen Tato, École de Technologie Supérieure, Canada
Benedict du Boulay, University of Sussex, United Kingdom
Kritya Bunchongchit, Mahidol University International College, Thailand
Michelle Banawan, Asian Institute of Management, Philippines
Ryan Baker, University of Pennsylvania, United States
April Clarke, University of Canterbury, New ZealandJose De-La-Cruz, Universidad de Malaga, Spain
Brendan Flanagan, Kyoto University, JapanClaude Frasson, University of Montreal, Canada
Bastiaan Heeren, Open University, The Netherlands
Erik Harpstead, Carnegie Mellon University, United States
Tsukasa Hirashima, Hiroshima University, Japan
Yuki Hayashi, Osaka Prefecture University, Japan
Tomoko Kojiri, Kansai University, Japan
Cristina Maier, McGraw Hill Education, United States
Tanja Mitrovic, University of Canterbury, Christchurch, New Zealand
Tatsunori Matsui, Waseda University, Japan
Elvira Popescu, University of Craiova, Romania
Valery Psyche, Université TÉLUQ, Canada
Maria Mercedes T. Rodrigo, Ateneo de Manila University, Philippines
John Stamper, Carnegie Mellon University, United States
Kazuhisa Seta, Osaka Metropolitan University, Japan
Stefan Slater, Teachers College, United States
Masaki Uto, The University of Electro-Communications, Japan
Jill-Jênn Vie, Inria Lille, Francexviii

C2: CSCL PC Member
Jürgen Buder, Leibniz-Institut für Wissensmedien, Germany
Sanjay Chandrasekharan, Homi Bhabha Centre for Science Education, India
Ben Chang, National Central University, Taiwan
Ching-Yi Chang, Taipei Medical University, Taiwan
Cheng-Huan Chen, Asia University, Taiwan
Sherry Chen, National Central University, Taiwan
Andrew Clayphan, The University of Sydney, Australia
Pryce Davis, University of Nottingham, United Kingdom
Xueqi Feng, Southern University of Science and Technology, China
Lydia Harbarth, University of Duisburg-Essen, Germany
Jo-Chi Hsiao, National Yang Ming Chiao Tung University, Taiwan
Lingyun Huang, Education University of Hong Kong, China
Gwo-Haur Hwang, National Yunlin University of Science and Technology, Taiwan
Heisawn Jeong, Hallym University, Korea
Vwen Yen Alwyn, Lee, Nanyang Technological University, Singapore
Rose Liang, National University of Singapore, Singapore
Feng Lin, Singapore University of Social Sciences, Singapore
Pei-Yi Lin, National Tsing-Hua University, Taiwan
Sunny Lin, National Yang Ming Chiao Tung University, Taiwan
Kian Keong Aloysiu Ong, National Institute of Education, Singapore
Johanna Pöysä-Tarhonen, University of Jyvaskyla, Finland
Nikol Rummel, Ruhr-Universität Bochum, Germany
Lenka Schnaubert, University of Nottingham, United Kingdom
Osman Tasdelen, Universität Duisburg-Essende, Germany
Yuyao Tong, University of Hong Kong, Hong Kong
Daiyi Wang, Providence University, Taiwan
Hongzhi Yang, The University of Sydney, Australia
Gaoxia Zhu, National Institute of Education, Nanyang Technological University, SingaporexixC3: ALT PC Member
Vishwas Punjaji Badhe, Indian Institute of Technology Bombay, India
Oluwafemi Samson Balogun, University of Eastern Finland, Finland
Gaurang Belekar, Indian Institute of Technology, Bombay, India
Manuel Caeiro Rodríguez, University of Vigo, Spain
Liang Changhao, Kyoto University, Japan
Weiqin Chen, Oslo Metropolitan University, Norway
Yiling Dai, Kyoto University, Japan
Syaamantak Das, Indian Institute of Technology Bombay, India
Seb Dianati, Charles Darwin University, Australia
Yoshiko Goda, Kumamoto University, Japan
Carl Haas, University of Waterloo, Canada
Marc Jansen, University of Applied Sciences Ruhr West, Germany
Nisumba Soodhani K, Indian Institute of Technology Bombay, India
Ulfa Khwaja, Indian Institute of Technology Bombay, India
Mehmet Kokoç, Karadeniz Technical University, Turkey
Siu Cheung Kong, The Education University of Hong Kong, Hong Kong
Suman Laudari, Charles Darwin University, Australia
Huiyong Li, Kyoto University, Japan
Sonsoles López-Pernas, University of Eastern Finland, Finland
Rwitajit Majumdar, Kyoto University, JapanJon Mason, Charles Darwin University, Australia
Ritayan Mitra, Indian Institute of Technology Bombay, India
Regina Motz, Universidad de la República, Uruguay
Shivsevak Negi, Don Bosco Institute of technology, Mumbai, India
Indrayani Nishane, Indian Institute of Technology Bombay, India
Rumana Pathan, Indian Institute of Technology Bombay, India
Herold Pc, Indian Institute of Technology Bombay, India
Zablon Pingo, university Of Technology, Sydney, Australia
Antony Prakash, Indian Institute of Technology Bombay, India
Ramkumar Rajendran, Indian Institute of Technology Bombay, India
Nihar Sanda, Indian Institute of Technology Bombay, India
Ismaila Temitayo Sanusi, University of Eastern Finland, FinlandBill Searle, Charles Darwin University, Australia
Niusha Shafiabady, Charles Darwin University, Australia
Jyoti Shaha, Indian Institute of Technology Bomaby, India
Atsushi Shimada, Kyushu University, Japan
Daevesh Singh, Indian Institute of Technology Bombay, India
María Estrella Sousa Vieira, University of Vigo, Spain
Ashwin T S, Indian Institute of Technology Bombay, India
Kyosuke Takami, Education Data Science Center, National Institute for Educational Policy
Research (NIER), Japan
Manjunath K Vanahalli, National Institute of Technology, India
Samarth Yadannavar, Indian Institute of Technology Bombay, Indiaxx

C4: CUMTEL PC Member
Kaushal Bhagat, Centre for Educational Technology, Indian Institute of Technology,
Kharagpur, India
Ivana Bosnic, University of Zagreb, Croatia
Ivica Boticki, Fakultet elektrotehnike i računarstva, Croatia
Huiying Cai, Jiangnan University, China
Ben Chang, National Central University, Taiwan
Guang Chen, Beijing Normal University, China
Feng-Kuang Chiang, Shanghai Jiao Tong University, China
Chi-Ming Chu, National Ilan University, Taiwan
Haiguang Fang, Capital Normal University, China
Xueqi Feng, Southern University of Science and Technology, China
Maja Gligora Marković, University of Rijeka, Croatia
Martina Holenko Dlab, University of Rijeka, Croatia
Yih-Ruey Juang, Jinwen University of Science and Technology, Taiwan
Tai-Chien Kao, National Dong Hwa University, Taiwan
Chiu-Lin Lai, National Taipei Univeristy of Education, Taiwan
Chen-Yu Lee, Ling Tung University, Taiwan
Jing Leng, East China Normal University, China
Xiuhan Li, Central China Normal University, China
Ma Luo, East China Normal University, China
Igor Mekterović, Fakultet elektrotehnike i računarstva, Croatia
Kuo-Liang Ou, National Tsing Hua University, Taiwan
Yanjie Song, The Education University of Hong Kong, Hong Kong
Daner Sun, The Education University of Hong Kong, China
Yuyao Tong, University of Hong Kong, China
Zhihong Wan, The Education University of Hong Kong, China
Xuefeng Wei, Ludong University, China
Longkai Wu, National Institute of Education, Singapore
Kai-Hsiang Yang, National Taipei University of Education, Taiwan
Xianmin Yang, Jiangsu Normal University, China
Yuqin Yang, Central China Normal Univerisity, China
Ying Zhan, The Education University of Hong Kong, Hong Kongxxi

C5: EGG PC Member
Ahmed Ahmim, Faculty of Exact Sciences and Sciences of Nature and Life University of Larbi
Tebessi, Algeria
Alex Barrett, Florida State University, United States
Abdelmalek Bouguettaya, CRTI, Algeria
Chefrour, université badji mokhtar annaba, Algeria
Chih-Pu Dai, Florida State University, United States
Zhaihuan Dai, University of South Florida, United States
Samia Drissi, univeristé de souk ahras, Algeria
Maazouzi Faiz, Univ annaba, Algeria
Zakaria Gheid, University of Souk Ahras, Algeria, Algeria
Kamel Eddine Heraguemi, M'sila University, Algeria
Hyangeun Ji, Temple University, United States
Amine Khaldi, kasdi merbah university, Algeria
Lukas Liu, The University of Hong Kong, Hong Kong
Soltani Mohamed, Souk Ahras University, Algeria
Yanjun Pan, Florida STATE UNIVERSITY, United States
Marcela Sávia Pessoa, Universidade do Estado do Amazonas, Brazil
Khedairia Soufiane, souk ahras university, Algeria
Luke West, Florida State University, United Statesxxii

C6: TELL PC Member
Michael Adarkwah, Southwest University, China
Ting Da, Beijing Normal University, China
Reza Hadi Mogavi, Sharif University of Technology, Iran
Feifei Han, Australian Catholic University, Australia
Ahmed Hosny, Beijing Normal University, Egypt
Muhammad Yasir Mostafa, Beijing Normal University, China
Stylianos Mystakidis, University of Patras, Greece
Michelle Siao-Cing Guo, National Taipei University of Business, Taiwan
Wanwisa Wannapipat, Khon Kaen University, Thailand
Vivian Wu, Asia University, Taiwan
Dong Yang, Beijing Normal University, China
Ahmed Mohamed Fahmy Yousef, Fayoum University, Egyptxxiii

C7: PTP PC Member
Vishwas Badhe, Indian Institute of Technology Bombay, India
Aparajita Biswal, B H Gardi College of engineering and technology, India
Ivica Boticki, Fakultet elektrotehnike i računarstva, Croatia
Arup Chatterjee, Indian Institute of Technology Madras, India
Rohan Dasgupta, Anjuman-I-Islam's Kalsekar Technical Campus, India
Ajita Deshmukh, MIT-ADT University, Pune, India
Anita Diwakar, Indian Institute of Technology Bombay, India
Lakshmi Ganesh, Indian Institute of Technology Bombay, India
Anchal Garg, University of Bolton, United Kingdom
Arnon Hershkovitz, Tel Aviv University, Israel
Martina Holenko Dlab, University of Rijeka, Croatia
Sajna Jaleel, Mahatma Gandhi University, India
Kapil Kadam, Indian Institute of Technology Bombay, India
Navneet Kaur, Indian Institute of Technology Bombay, India
Najwan Khambari, Universiti Teknikal Malaysia Melaka, Malaysia
Chen-Yu Lee, Ling Tung University, Taiwan
Ganesh Lokhande, Symbiosis International (Deemed) University, India
Jayakrishnan Madathil, Indian Institute of Technology–Madras, India
Hagit Meishar Tal, Holon Institute of Technology (HIT), Israel
Shitanshu Mishra, UNESCO Mahatma Gandhi Institute of Education for Peace andSustainable Development, India
Priscilla Moses, Universiti Tunku Abdul Rahman, Malaysia
Soumya Narayana, Indian Institute of Technology Bombay, India
Lucian Ngeze, Indian institute of Technology Bombay, India
Yogendra Pal, NIIT University, India
Mrinal Patwardhan, Indian Institute of Technology Bombay, India
Prajish Prasad, FLAME University, India
Rajashri Priyadarshini, Indian Institute of Technology Bombay, India
Ashutosh Raina, Indian Institute of Technology Bombay, India
Rekha Ramesh, Mumbai University, India
Vivek Sabanwar, Indian Institute of Technology Bombay, India
Sameer Sahasrabudhe, Indian Institute of Technology Bombay, India
Charu Saini, UNESCO Mahatma Gandhi Institute of Education for Peace and Sustainable
Development, India
Madhuri Srinivas, SMIORE – Education, India
Narasimha Swamy, Indian Institute of Technology Bombay, Mumbai, India
Briju Thankachan, Indian Educational Technology Association, India
Bindu Thirumalai, Tata Institute of Social Sciences in, India
Vikram Vincent, Indian Institute of Technology Bombay, India
Ying Zhan, The Education University of Hong Kong, Hong Kongxxiv

EARLY CAREER RESERCHER AWARD WINNER(2023)
Dr. Rwitajit MAJUMDAR is an Associate Professor at the
Research and Educational Institute for Semiconductors
and Informatics at Kumamoto University. He is attachedto the Graduate School of Social and Cultural Sciences inthe Division of Instructional System Studies. Before joining
Kumamoto University in September 2023, he was a seniorlecturer at the Academic Center for Computing and Media
Studies at Kyoto University since 2021. He joined as a
post-doc researcher in Prof. Hiroaki Ogata's lab andmoved to Japan in 2018.
Rwitajit graduated from the Inter-disciplinary program (IDP)in Educational Technology at the Indian Institute ofTechnology Bombay in India, co-advised by Prof. Sridhar
Iyer of the Computer Science and Engineering department and Prof. Aniruddha Joshi from the
Design School. He did his undergraduate studies and master's from BITS Pilani, India, inEngineering Technology and Design Engineering. He attended doctoral coursework at the
Indian Institute of Science in Bangalore before moving to Mumbai for doctoral research.
Rwitajit's research interests include Learning Analytics, designing data-driven services, andstudying human-data interactions in the context of education. In the last five years, he hasreceived 3 national grants from JSPS as PI and 3 as co-PI for research related to the GOAL
project for designing data-driven platforms to develop learners' self-direction skills and build
knowledge model-based learning infrastructure. At Ogata lab, over the years, Rwitajit worked
directly with 7 Ph.D. and 7 master's students and other research members in various learning
analytics research projects and has co-authored more than 100 international
conference papers and 30 journal publications. He continues to bridge researchers from theEast in Japan, India, Taiwan, and the West to share expertise and perspectives in different
collaborative research projects. In 2023
He has actively participated in the APSCE events, being co-chair for the Advanced Learning
Technologies (ALT), Learning Analytics and Digital Infrastructure sub-conference track of ICCE,
organizing workshops related to Learning Analytics as well as Embodied Learning in ICCE andcontributing to RPTEL journal as an author as well as reviewer. Rwitajit was also awarded theIEEE TCLT Early Career Researcher Award in Learning Technologies during ICALT 2023.
Along with research, Rwitajit likes to travel and experience local cultures and practices. His
hobbies include photography, music, and mending broken potteries with the art of Kintsugi.

xxv

LAST TEN YEARS’ DISTINGUISHED RESEARCHER
AWARD WINNERS
2022 -APSCE Distinguished Researcher Award
Maiga Chang, Athabasca University, Canada
2021 -APSCE Distinguished Researcher Award
Maria Mercedes T. Rodrigo,Ateneo de Manila University, Philippines2020 -APSCE Distinguished Researcher Award
Wenli CHEN, Nanyang Technological University, Singapore
2015 -APSCE Distinguished Researcher Award
Lung-Hsiang WONG, Nanyang Technological University, Singapore
2014 -APSCE Distinguished Researcher Award
Hiroaki OGATA, Kyushu University, Japan
2011 -APSCE Distinguished Researcher Award
Antonija MITROVIC, University of Canterbury, New ZealandChen-Chung LIU, National Central University, Taiwan

xxvi

LAST TEN YEARS’ EARLY CAREER RESEARCHER
AWARD WINNERS
2022 - APSCE Early Career Researcher Award
Daner Sun, The Education University of HongKong, HongKong
2021 -APSCE Early Career Researcher Award
Bo Jiang,East China Normal University, China
2020 -APSCE Early Career Researcher Award
Kaushal Kumar BHAGAT, Indian Institute of Technology, Kharagpur, India
2019 -APSCE Early Career Researcher Award
Cheng-Jiu YIN, Kobe University, Japan2018 -APSCE Early Career Researcher Award
Ting-Chia HSU, National Taiwan Normal University, Taiwan
2017 -APSCE Early Career Researcher Award
Jon MASON, Charles Darwin University, Australia
2015 -APSCE Early Career Researcher Award
Morris Siu-Yung JONG, The Chinese University of Hong Kong, Hong Kong

xxvii

SPEAKERS OF APSCE WEBINAR SERIES(December 2022 – November 2023)
APSCE Webinar #30: The Metaverse and Language Learning
Date: 16 December 2022 (Friday)
Speaker: Prof. Yu-Ju LAN (National Taiwan Normal University, Taiwan)
Moderator: Prof. Vivian Wen-Chi WU (Asia University, Taiwan)
Curated by: APSCE Technology-Enhanced Language Learning (TELL) SIG
APSCE Webinar #31 : Leveraging Deep NLP and Generative AI in Education
Date: 10 March 2023 (Friday)
Speaker: Dr. Michelle Banawan, Asian Institute of Management, Philippines
Moderator: Dr. May Marie P. Talandron-Felipe, University of Science and Technology ofSouthern Philippines, Philippines
Curated by: APSCE Artificial Intelligence in Education / Intelligent Tutoring Systems /Adaptive Learning (AI-Ed) SIG
APSCE Webinar #32 : Graphical organizer-based in-field mobile learning
Date: 21 April 2023 (Friday)
Speaker: Prof. Hui-Chun CHU, Soochow University, Taiwan
Moderator: Prof. Jerry Chih-Yuan SUN, National Yang Ming Chiao Tung University, Taiwan
Curated by: APSCE Advanced Learning Technologies, Platforms & Infrastructures (ALT) SiG
APSCE Webinar #33 : Institutional and Psychological Factors Affecting Online DistantForeign Language Learning Behaviors
Date: 5 May 2023 (Friday)
Speaker: Prof. Yuichi ONO, University of Tsukuba, Japan
Moderator: Prof. Vivian Wen-Chi WU (Asia University, Taiwan)
Curated by: APSCE Technology-Enhanced Language Learning (TELL) SIG
APSCE Webinar #34: Transforming Education with AI and Computational Action
Date: 18 May 2023
PANELIST:
Natalie LAO, Massachusetts Institute of Technology, USA
Mark FRIEDMAN, App Inventor Foundation, USA
Keertan KINI, Stanford University, USA
Chair: Ting-Chia HSU, National Taiwan Normal University, Taiwan
Curated by: APSCE Computational Thinking in Education/STEM (CTE/STEM) SIG
APSCE Webinar #35 : Three Challenges in Implementing Multimodal Learning Analytics inReal Learning Environments
Date: 31 May 2023 (Wednesday)
Speaker: Assoc. Prof. Bertrand Schneider, Harvard Graduate School of Education, USA
Moderator: Dr. Elizabeth Koh, Nanyang Technological University, Singapore
Curated by: APSE Computer-Supported Collaborative Learning / Learning Sciences (CSCL/LS)
SIG
xxviiiAPSCE Webinar #36 (Postponed): Smart Technologies in Education: Policies for Effectiveand Ethical Use
Date: 16 June 2023
PANELIST:
Habibah Ab Jalil, Universiti Putra Malaysia, Malaysia
Vikas Kanungo, World Bank
Lung Hsiang WONG, Nanyang Technological University, Singapore
Moderator: Khaizer Omar Universti Putra Malaysia, Malaysia
Curated by: APSCE Practice-Driven Research, Teachers’ Professional Development & ICT
Policies (PTP) SIG
APSCE Webinar #37: Moving Toward a Mobile Learning Landscape: Effective Device
Integration
Date: 3 July 2023
Speaker: Dr. Helen Crompton, Old Dominion University, USA
Moderator: Dr. Daner Sun, Education University of Hong Kong
Curated by: APSCE Classroom, Mobile and Ubiquitous Technology Enhanced Learning(CUMTEL) SIG
APSCE Webinar #36 (Re-scheduled): Smart Technologies in Education: Policies for Effectiveand Ethical Use
Date: 31 July 2023
PANELIST:
Habibah Ab Jalil, Universiti Putra Malaysia, Malaysia
Maiga Chang, Athabasca University, Canada
Lung-Hsiang Wong, Nanyang Technological University, Singapore
Moderator: Muhd Khaizer Omar, Universiti Putra Malaysia, Malaysia
Curated by: SIG 7 – Practice-driven Research, Teacher Professional Development and Policyof ICT in Education (PTP)APSCE Webinar #38: Human-Centered Learning Technologies and Multimodal Data
Date: 21 July 2023
Speaker: Prof. Michail (Michalis) Giannakos
Norwegian University of Science and Technology (NTNU), Norway
Moderator:Dr. Ramkumar Rajendran, IIT Bombay, India
Curated by: APSCE Learning Analytics and Educational Data Mining (LAEDM) SIG
APSCE Webinar #39: Unlocking Potential: Leveraging Multimodal Learning Analytics for
Collaborative Learning
Date: 11 August 2023
Speaker: Prof. Dragan Gašević, Monash University, Australia
Moderator: Dr. Ramkumar Rajendran, IIT Bombay, India
Curated by: APSCE Learning Analytics and Educational Data Mining (LAEDM) SIG
APSCE Webinar #40: Where now for ‘Smart’? Consequent questions and the co-productionof knowledgexxix

Date: 6 September 2023 (Wednesday)
Speaker: Assoc. Prof. Jon Mason, Charles Darwin University, Australia
Moderator: Prof. Takahito Tomoto, Chiba lnstitute of Technology, Japan
Curated by: APSCE Education Use of Problems/Questions in Technology-Enhanced Learning(EUPQ) SIG
APSCE Webinar #41: Building In-Context Understanding of Learning Behaviors for
Designing Game-Based Assessments
Date: 3 October 2023 (Tuesday)
Speaker: Assist. Prof. Zhichun “Lukas” Liu, The University of Hong Kong, Hong Kong SAR
Moderator: Assoc. Prof. Ahmed Tlili, Beijing Normal University, China
Curated by: APSCE Educational Gamification and Game-based Learning (EGG) SIG
APSCE Webinar #42: Digital Transformation of Higher Education: Challenges and Insights
Date: 25 October 2023 (Wednesday)
Speaker: Prof. Merlin Teodosia Suarez, De La Salle University, The Philippines
Moderator: Assoc. Prof. Patcharin Panjaburee, Khon Kaen University, Thailand
Curated by: APSCE Development of Information and Communication Technology in the AsiaPacific Neighborhood (DIC TAP) SIG

xxx

KEYNOTE SPEAKERS

Curtis J. BONK
Indiana University, USA

Time to Wake Up from Our Innovative Learning
Dreams and Make Smarter Learning a Reality
Abstract: For a half century, educators, psychologists, and researchers have been
predicting that highly intensive, innovative, and individualized learning formats are
only a few years away. Learners of all ages would enter enticing microworlds, highly
engaging learning experience holodecks, fully immersive hands-on scenarios, highfidelity simulations and games, AI-based adaptive microlearning snippets, andcompletely free and open educational resources and courses on any topic. Massive
open online classes were promised one day and then on demand microlearningsnippets were delivered in the next. The learning related dreams we had in past
decades were quickly forgotten as the next wave of learning technology came along.
But all those dreams will prove pointless if they fail to address true problems or issues
that some aspect of society is struggling with. It is time to wake up from such dreams
of a glistening technological future and have our dream machines help us envision aworld filled with open, informal, adaptive, nontraditional, and self-directed learning
opportunities. When that happens, we will truly have arrived in the age of smarter andmore innovative forms of learning where the learner is finally in charge of the dreams.
Biography: Curtis J. Bonk is Professor in the School of Education at Indiana University(IU) teaching psychology and technology courses and Adjunct in the School ofInformatics at IU. He is a former software entrepreneur, certified public accountant,corporate controller, and educational psychologist who presently is an educational
technologist, award-winning writer, highly published researcher, statewide andnational awardee in innovative teaching with technology, and internationally acclaimed
presenter. Curt is the author of over 400 publications including 20 books such as the
Handbook of Blended Learning: Global Perspectives, Local Designs, MOOCs and Open
Education in the Global South, The World is Open: How Web Technology is
xxxi

Revolutionizing Education, and Transformative Teaching Around the World. He has
given close to 2,000 talks around the world, including over 300 keynote and plenary
talks. In 2020, Curt was awarded the IU President’s Award for Excellence in Teachingand Learning Technology and in 2021, he received the David H. Jonassen Excellence in
Research Award. Recently, the American Educational Research Association named him
a 2022 AERA Fellow for his exceptional contributions to, and excellence in, education
research, and the following week, he was honored with the International Engagement
award from the IU School of Education. In 2022, he was also listed in the top 2% ofscientists in the world based on publication citations for career. In 2023, AERA awarded
Curt and his colleague Dr. Min Young Doo from Kangwon National University in Korea
with the Outstanding International Research Collaboration Award. Curt Bonk co-hosts
the weekly award-winning podcast show, Silver Lining for Learning(https://silverliningforlearning.org/). He can be contacted at
homepage is http://curtbonk.com/.

xxxii

and hisKEYNOTE SPEAKERS

Tak-Wai CHAN
National Central University, Taiwan

Global Harmony and Wellbeing——Should it be our Ultimate Educational Goal Worldwide?
Abstract: More than 160 years ago, Dickens wrote in the first sentence of his A Tale of
Two Cities: "It was the best of times, it was the worst of times, it was the age of wisdom,
it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity,
it was the season of Light, it was the season of Darkness, it was the spring of hope, it
was the winter of despair...”. Today, the human lifespan has extended, but millions ofpeople have deceased in a short time before our eyes due to COVID-19. Digitization
enables all of us to connect and communicate, but we constantly quarrel over different
beliefs; online games offer immersive experience, but the specter of children’s
addiction looms large; artificial intelligence promises to enhance our lives, yet it harbors
the potential of great harm to human beings; the advent of metaverse could herald aluminous future, but it may also signal a dark abyss to come; we cheer for technologicaladvancement, but are simultaneously beset by concerns over environmental pollution,
wealth disparity, and other daunting challenges.
During my keynote address at AIED2007, I put forward my observation: our researchcommunity has been experiencing three orientations of research: dream-oriented,
adoption-oriented, and humanity-oriented. Humanity-oriented research, which was
emerging at the time, addresses that learning should go beyond knowledge acquisition,and hence it should cover cognitive, affective, social and attitudinal domains. For thesake of humanity, we should strive to lay the foundations for the future world bybettering yourself, nurturing a caring family, incubating a humane society, fostering apeaceful and collaborative world.

xxxiiiFurthermore, I posed 4 grand challenge problems. The first three were informed by myobservation of research in artificial intelligence in education (AIED), computersupported collaborative learning, mobile learning, and game-based learning, as well as
by the challenges of transforming education at that time. The fourth problem, the‘global educational goal problem’—rethinking the educational goal from the global
perspective—was due to the threats such as nuclear holocaust, earth resource
exhaustion, climate change, societal polarization, and mass extinction of species. Theseissues present profound risks to humanity and the planet's future. There is an increasing
concern about whether our descendants will be able to survive on Earth, let alone live
enjoyably. Recognizing the pressing need to address the global educational goalproblem, I reordered the four problems in my last slide, placing it as the first grandchallenge problem.
I acknowledge that when I first posed the global educational goal problem, I didn't
have any idea about the answer, and I believed that this was the case for most otherresearchers in our field as well. However, the problem is so fundamental that it
underpins why we do what we do.
Given the recent regrettable conflicts in the world, leaders in a prominent society in
our field have recently raised questions and made statements: “Can we develop
partnerships, processes, and learning environments that can reduce divisiveness andengender abilities to talk, work, and learn across differences? Can we design to disrupt
othering and hate and to promote empathy and care? There is growing urgency tocontinue developing theory and practice to address this aim. We have expertise in our
membership, we need to learn from each other, especially from those who have been
thinking about this for a while now. This is where we can make a difference and takeaction as a Society.” The world truly stands on the brink of peril. Can we coexist
harmoniously with our environment? What is the future of humankind? What role dowe, as educators, play in shaping the world of tomorrow?
What is education? It is said that at the end of World War II, a letter was found in a Nazi
concentration camp. It is addressed to Teachers (Note 1).
Dear Teachers:
I am a survivor of a concentration camp. My eyes saw what no man should witness: gaschambers built by learned engineers, children poisoned by educated physicians, infants
killed by trained nurses, women and babies shot and burned by high school and college
graduates.
So, I am suspicious of education.
My request is: help your students become human. Your efforts must never produce
learned monsters, skilled psychopaths, educated Eichmanns. Reading, writing,arithmetic are important only if they serve to make our children more human.
If knowledge and human values are disconnected, can education lead to a morecivilized world? If global society is not harmonious, can wellbeing be achievable?
Mandela once said, “Education is the most powerful weapon which you can use tochange the world.” Indeed, education stands as a beacon of hope for the future. The
education we impart today will shape the destiny of all humans in the years to come.
xxxiv

In two decades or so, today's students in schools and universities will become the pillarsof our society.
Designing the future of education is equivalent to designing the future world.
Educational researchers—particularly those in our community who are engaged withtechnology—bear an even greater responsibility to lead change through global
collaboration.
In this talk, I will share some thoughts based on years of ongoing discussions among agroup of international researchers (Note 2). First, we may define the 'ultimate
educational goal worldwide' as the final, most far-reaching aspiration that people
envision, plan, and commit to achieving in education. It is designed to serve everyone
across all societies and cultures, transcending individual and societal objectives toembrace humanity as a whole. For the sake of brevity, we will refer to the 'ultimate
educational goal worldwide' simply as the 'ultimate educational goal' henceforth.
Second, assuming the ‘ultimate educational goal’ exists, its realization would
necessitate a form of ‘ultimate education.’ Perhaps we can formulate this ultimate
education as follows:
ultimate education = (ultimate educational goal, design theory, future digital world)
In this formulation, the ‘ultimate educational goal’ represents 'why' and 'what' to learn;
design theory informs 'how' to learn; and ‘future digital world’ addresses ‘who’ and'where' to learn, as well as how the ultimate education can be supported by andintegrated into the digital future world, propelling the transformation of education
toward that overarching goal.
We propose 'Global Harmony and Wellbeing' (abbreviated as ‘Global Harwell’) as a
candidate for the ultimate educational goal in this formulation (Note 3). We use'candidate' to acknowledge that, after thorough discussion, other superior options forthe ultimate educational goal may emerge. Additionally, we introduce the InterestDriven Creator (IDC) Theory as a candidate for the design theory—a theory planned to
be a revised version of its predecessor. Moreover, we suggest the term 'Seamless AI
World' is a candidate concept (or notion) that succinctly and accurately captures theessence of the future digital world (Note 4).
Once the ultimate educational goal is identified, to impact education significantly andaccelerate its transformation, we may:1.
Build global and local awareness of the goal,
2.
Establish experimental and model educational sites in various countries andregions,3.
Create a ‘global ultimate educational park,’ and4.
Disseminate the principles of ultimate education worldwide."
Given its potential far-reaching impact on education, along with its inherent intricacy
and the fast-paced nature of the digital world, establishing the ultimate educationworldwide will require concerted efforts of several generations. There's an adage: “Ifyou want to go fast, go alone; if you want to go far, go together.” In fact, if you goalone, you won't get anywhere. However, given the pressing urgency humanity faces
and our planet teetering on the edge of catastrophe, we must unite to achieve swift
xxxv

and substantial progress. We cordially invite you to join us in this fruitful dialogue. To
facilitate this, we plan to host a series of forums, either online or in person, to deliberate
the aforementioned issues at the website: globalharwellgoal.org. We have chosen'Global Harwell Goal' as the name for our website because the proposed goal can, atthe very least, serve as a reference point and stimulate discussions until a more suitablename for the ultimate educational goal is agreed upon.
Note 1: Chee-Kit Looi forwarded the story to me. The letter, often attributed to Haim
Ginott, was published in his book ‘Teacher and Child.’
Note 2: Tak-Wai Chan, Chee-Kit Looi, Siu-Cheung Kong, Wenli Chen, Lung-Hsiang
Wong, Su Luan Wong, Ben Chang, Ju-Ling Shih, Ying-Tien Wu, Fu-Yun Yu.
Note 3: It is worth noting that most of us are researchers, not experts in philosophy,
politics, or religion. Nevertheless, we can still propose possible candidates for theultimate educational goal based on our knowledge and life experiences, without
deliberately considering religious doctrines or ideological beliefs. Furthermore, if
Global Harwell is our ultimate educational goal, its fulfillment would require what wemight call Global Harwell Education.
Note 4: If we accept Global Harwell as our ultimate educational goal, addressing the'why' and 'what'; IDC Theory as our activity design framework, explaining the 'how';and Seamless AI World as the concept describing the digital future, outlining the'where' and 'who'; then 'Seamless IDC Theory' could be a theory for designing Global
Harwell Education.
Biography: Professor Tak-Wai Chan is a trailblazer in digital learning and a global
leader in the field. Almost 40 years ago, at a time when computers and the internet
were not yet mainstream in the mid-eighties, he began researching on AI supported
learning for his doctoral dissertation, proposing a new genus of AI in education systemcalled learning companion system in 1988. This virtual companion system, called
Integration-Kid, was the first artificial companion in the world. In 1989, he and his
students started to build the world’s first dedicated networked learning system forcollaborative learning and learning through competition games, called Distributed
West (1992). In early 2000s, he and his colleagues built the largest online learning
community called EduCity (1.5 million learners with 1,700 schools involved in 2003),
which was also referred as the first learning society in the world. In the same time
period, his team conducted frontier research on mobile learning, intelligent classroom,
future classroom, interactive clicker, e-schoolbag, one-to-one technology enhanced
learning, and so forth. After this series of research, in 2006, working together with alarge group of international researchers mainly from the Western countries, heproposed the concept of Seamless Learning. In 2010s, after some long-term
experiments on reading (MSSR) and writing in one-to-one technology enhanced
classroom, in collaboration with a group of Asian scholars, he proposed the Interest-xxxvi

Driven Creator (IDC) Theory. Again, he and his colleagues are now calling for building
Seamless Interest-Driven Creator (SIDC) Theory with interested researchers.
In addition to his research, Professor Chan has also been a major founder of twosocieties: the Asia-Pacific Society for Computers in Education (APSCE) and the Global
Chinese Society for Computers in Education (GCSCE). These two societies respectively
host annual conference series ICCEs and GCCCEs, as well as the journals RPTEL and
JLCE. Moreover, to cope with the expanding research community of the field, he has
been assisting the establishment of APSCE Theme-Based International Conference
Series (TBICS), including CTE-STEM, ICFULL, MetaACES.

xxxvii

KEYNOTE SPEAKERS

Davinia HERNÁNDEZ-LEO
Universitat Pompeu Fabra, Barcelona, Spain

Computers in Education: how can we support teachers?
Abstract: While it is widely agreed that the role of teachers is key to achieve students’learning, research on how technology can support teachers’ tasks is often
underemphasized. In this talk I will summarise research results leading to practical
implications in the design of technologies that improve the efficiency and effectivenessof teachers’ tasks, caring also for their wellbeing. In particular, I will focus on howtechnology can support learning design and the orchestration of complex learning
scenarios, such as computer-supported collaborative learning in large classrooms. The
technology presented will include authoring tools, teaching community platforms,
enactment systems, orchestration dashboards and data-driven interventions based onlearning analytics. I will also discuss synergies between technological solutions
emphasizing human-in-control and machine-in-control perspectives. During the talk,participants will be able to experience some notions covered by interacting using the
PyramidApp tool and the Integrated Learning Design Environment (ILDE).
Biography: Davinia Hernandez-Leo is Full Professor, Serra Hunter and ICREA Academia
Fellow at the Department of Information and Communications Technologies
Department (DTIC) at Universitat Pompeu Fabra, Barcelona (Spain), where she is the
director of the Interactive and Distributed Technologies for Education research group(TIDE). She obtained a Ph.D. at University of Valladolid, Spain, and has been visiting
researcher at Open University of the Netherlands, Fulbright Scholar at Virginia Techand visiting academic at the University of Sydney. She has published extensively andreceived several awards, including best and most cited scientific paper awards andxxxviii

recognitions for technology contributions. Prof. Hernández-Leo has been VicePresident of the European Association for Technology-Enhanced Learning, a Associate
Editor of the IEEE Transactions of Learning Technologies, and is currently an elected
member of the CSCL Committee within the International Society of the Learning
Sciences and member of the Steering Committee of the European Conference on
Technology-Enhanced Learning. She is very active in research supervision (PhD
students, visitors, etc.), in participation and lead of European and national projects, andin collaborations with companies, non-profit organizations, policy makers and private
foundations. Her research activity is broadly centered on the domain of learningtechnologies, spanning fields such as learning design technology, computer-supported
collaborative learning (CSCL), community platforms and learning analytics.

xxxix

KEYNOTE SPEAKERS

Masaru KITSUREGAWA
Research Organization of Information and Systems,
Japan

Building a Research Data Platform and Education
Abstract: We have entered a highly uncertain, unpredictable age beset by natural
disasters and wars around the world as well global-scale pandemics. However, we must
not despair at this state of affairs and simply wait in hope of better circumstances.
Rather, we must move forward with an eye to the future. The Research Organization ofInformation and Systems (ROIS), consisting of four distinguished research institutes,
aims to solve complex phenomena and issues relating to life, the earth, the naturalenvironment, and human society by reframing these issues from the perspective of
information and systems while advancing data science to conduct integrated researchthat transcends disciplinary boundaries. In line with its mission to support resourcesharing and joint research among all universities, ROIS promotes cutting-edge researchin specialized fields through joint research that transcends university boundaries by
providing researchers nationwide with access to large-scale, state-of-the-art
equipment and facilities, big data, valuable materials, and analytical methods.
Especially, the National Institute of Informatics replaced the previous Science
Information NETwork (SINET) with the world’s fastest ultra-high-speed network
infrastructure, SINET6, which provides transmission speeds of up to 400 Gbps. In
addition to the over 1,000 institutions and universities currently being served, thenetwork will soon be offered to elementary, junior high, and high schools as well. SINET
is also expected to make substantial contributions to industry and continuing
education. The full rollout of the GakuNin RDM research data management platform
not only provides data management support for individual researchers but also
xl

supports the development of open science by providing a platform for the proper
release of research data including educational big data.
Biography: Masaru Kitsuregawa graduated from the Electronics Engineering
Department, Faculty of Engineering, the University of Tokyo in March 1978, completed
his doctorate in information engineering at the same university and received a Ph.D. in

1983. He became a lecturer at the Institute of Industrial Science in April 1983, anassociate professor in 1984, and a professor in 1997, all at the same university. He has
been director general of the National Institute of Informatics since 2013. Currently heis a president of Research Organization of Information and Systems in Japan. Dr.
Kitsuregawa has made numerous distinguished achievements in the database fieldover a long period. He was a leading researcher on the high-speed operation of ahashed relational database. With a conventional simple method, the relational
database operation cost is the square of the number of records. To solve this problem,
he developed the GRACE hash method, which operates a database at a linear cost by
combining a dynamic destaging method, bucket adjustment and different
implementation methods. This method is referred to in Wikipedia as a basic method of
operating a relational database. Today, all major database software programs use ahash algorithm. Dr. Kitsuregawa’s research established the foundation of this algorithm.
In recognition of his achievements in enhancing database performance, including
those mentioned above, he received the ACM SIGMOD E. F Codd Innovations Award,
which is the most prestigious award in database system research. He was the firstrecipient from Asia. Also, he was designated a fellow by IEICE, IEEE, and ACM, and also
received many awards, including Achievement Award from IEICE, Medal with Purple
Ribbon, and Legion d’Honneur, Chevalier.

xli

THEME-BASED INVITED SPEAKERS

Kaushal Kumar BHAGAT
Indian Institute of Technology Kharagpur, India

Game On! Leveraging the Benefits of Game-Based Learning in the
Digital Age
Abstract: In today’s digital age, game-based learning has become an increasingly
popular way to engage students and enhance their learning experiences. Game-basedlearning leverages the engaging and immersive nature of games to create a fun andinteractive learning environment, which can help students to develop critical thinking,problem-solving, and collaboration skills. In this keynote presentation, we will explore
the benefits of game-based learning and discuss how it can be used to meet the needsof today’s learners. We will discuss the importance of incorporating game-basedlearning into the classroom and explore some of the latest research on theeffectiveness of this approach. We will also explore some fundamental designprinciples of successful game-based learning and highlight some of the best practices
that educators can use to create engaging and effective games for their students.
Finally, we will examine some of the challenges and limitations of game-based learningand discuss how educators can work to overcome these obstacles. Overall, this keynote
presentation will provide attendees with a comprehensive overview of game-basedlearning and its potential to transform education in the digital age. Whether you are
an educator, a curriculum developer, or a game designer, this presentation will provide
valuable insights into how you can leverage the benefits of game-based learning tocreate engaging and effective learning experiences.
Biography: Dr. Kaushal Kumar Bhagat is currently working as an assistant professor in
the Advanced Technology Development Centre at the Indian Institute of Technology(IIT), Kharagpur, India. He received his Ph.D. from the National Taiwan Normal
xlii

University in September 2016. He then served a two-year postdoctoral position at the
Smart Learning Institute at Beijing Normal University. In 2015, Dr. Bhagat received
NTNU International Outstanding Achievement Award. He was also awarded the 2017
IEEE TCLT Young Researcher award. In 2020, he received APSCE Early Career ResearcherAward (ECRA) from the Asia-Pacific Society for Computers in Education. He was also
awarded the 2022 Excellence in Distance Education Award (EDEA) by the
Commonwealth of Learning (COL), Canada. He is an associate editor of the British
Journal of Educational Technology (BJET), Regional Associate Editor (Asia) of the
Journal of Learning for Development (JL4D), and editor-in-chief of Contemporary
Educational Technology (CET). He is also an editorial board member of several reputed
international journals. He is a consultant for international organizations like theCommonwealth of Learning, UNESCO, etc. His research area of interest includes
augmented reality, virtual reality, game-based learning, online learning, andtechnology-enhanced learning.

xliii

THEME-BASED INVITED SPEAKERS

Brendan FLANAGAN
Kyoto University, Japan

Challenges and Opportunities of Educational Data Science for
Reading Systems
Abstract: As educational systems are collecting an increasing amount of data on thelearning behavior of students, its analysis has given rise to the fields of Educational
Data Mining, and more recently Learning Analytics. As a result, educational AI that isconstructed from and consumes learning behavior data has become more prevalent inlearning systems and is fueling increased research attention in the field. While many
datasets have been made public to promote research, important issues such as
information privacy have also limited broader analysis and have resulted in data silosand hindered replication studies within the community. This talk will give an overview
of educational data science focusing on reading systems and discuss importantongoing challenges including data analysis for niche learning contexts, data divide, andinsights into methods for promoting collaboration through synthetic data and their
possible limitations.
Biography: Brendan Flanagan is an Associate Professor at the Center for Innovative
Research and Education in Data Science, Institute for Liberal Arts and Sciences, and theData Science Department at the Graduate School of Informatics, Kyoto University. His
research interests include Learning Analytics, Educational Data Science, Computerxliv

Assisted Language Learning, and the Application of Blockchain in Education. He has
also hosted educational data challenges at prominent international conferences formore than 5 years to promote educational data science research. He is currently the
Principle Investigator of several government-funded research projects that investigate
knowledge and learning process analysis, and explainable predictions by learning
systems, and is also part of a Japanese Cabinet Office (NEDO) funded large research
project into educational symbiotic AI systems.

xlv

THEME-BASED INVITED SPEAKERS
Daner SUN
Education University of Hong Kong, Hong
Kong

Exploring the Evolution of Mobile Learning Environments
Abstract: The rapid advancement of technology and the changing landscape of
education have led to significant changes in technology-enabled learning
environments. This presentation will explore the impact of changing situations onmobile technology-enabled learning environments, with the speaker sharing insights
as both a researcher and an instructor. The talk will cover the evolving distribution andadjustment of components in these environments, as well as changes in pedagogy
before, during, and after the Covid-19 pandemic. Additionally, the speaker will
highlight the emerging dominance of new technologies in Hong Kong and worldwide,and propose future research directions for mobile learning.
Biography: Dr Daner Sun is an assistant professor at the Department of Mathematics
and Information Technology, the Education University of Hong Kong (EdUHK), HongKong. Her research interests are mobile learning, STEM education, and higher-order
thinking in technology-supported learning. So far, Dr Sun has published more than 30
SSCI journal papers. She serves as the editor/co-editor for conference proceedings andjournal special issues and acts as a reviewer in the community. Besides being theawardee of the APSCE Early Career Researcher Award (ECRA) 2022, she is also theawardee of Outstanding Performance in Research 2023, Outstanding Performance inKnowledge Transfer (Team) 2020, and Dean’s Research Output Prize 2021 in EdUHK.

xlvi

Table of Content
C1: ICCE Sub-Conference on Artificial Intelligence in Education/Intelligent
Tutoring System (AIED/ITS) and Adaptive Learning

1

An Efficient and Generic Method for Interpreting Deep Learning based Knowledge Tracing
ModelsDeliang Wang, Yu Lu, Zhi Zhang and Penghe Chen

2

Recommending Learning Actions Using Neural Network
Hirokazu Kohama, Yuki Ban, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi,
Akitoshi Itai and Hiroyasu Usami

12

An Agent-Based Modeling and Simulation Tool as a Learning Aid for Diffusion of Innovations
Joseph Benjamin Ilagan, Jose Ramon Ilagan and Maria Mercedes Rodrigo

21

Studying Memory Decay and Spacing within Knowledge Tracing
Cristina Maier, Isha Slavin, Ryan S. Baker and Steve Stalzer

24

Developing a Video-based e-Learning System Incorporating a Fill-in-the-blank Question-type
Concept Map
Takumi Hasegawa and Tessai Hayama

34

Exploring Students' Adoption of ChatGPT as a Mentor for Undergraduate Computing Projects:
PLS-SEM Analysis
Swapna Gottipati, Kyong Jin Shim and Venky Shankararaman

40

Deep Knowledge Tracing is an Implicit Dynamic Multidimensional Item Response Theory Model
Jill-Jênn Vie and Hisashi Kashima

46

Immediate Feedback in Computational Thinking: Generating hints using a Knowledge Graph
Nitesh Kumar Jha, Plaban Kumar Bhowmik and Kaushal Kumar Bhagat

55

Composite Score for ChatGPT Prompt Efficiency: A Computational Linguistic Analysis ofEngineered Chatbot Prompts
Michelle Banawan

58

Large Language Models (GPT) for Automating Feedback on Programming Assignments
Maciej Pankiewicz and Ryan Baker

68

Improved Automated Labeling of Mathematical Exercises in Japanese
Taisei Yamauchi, Ryosuke Nakamoto, Yiling Dai, Kyosuke Takami, Brendan Flanagan and
Hiroaki Ogata

78

Can We Ensure Accuracy and Explainability for a Math Recommender System?
Yiling Dai, Brendan Flanagan and Hiroaki Ogata

88

Activity Analysis Support System by Causal Relation Check Between Sub-activities
Kota Kunori and Tomoko Kojiri

98

Table of Content
Method for Estimating Learning Strategiesfrom Tools Using Bayesian Network

101

Kento Kuwajima, Atsushi Ashida and Tomoko Kojiri
ExGen: Ready-To-Use Exercise Generation in Introductory Programming Courses
Nguyen Binh Duong Ta, Hua Gia Phuc Nguyen and Swapna Gottipati

104

Object Identification Training Support System for Object-Oriented Design with Cooking Recipes
Daiki Maeda, Kota Kunori and Tomoko Kojiri

114

Development of Learning Support System for Critical Reading of Academic Papers
Aota Nishida, Kazuhisa Seta and Yuki Hayashi

120

Overcoming Barriers to Sustainable Dissemination of L2 Learning Resources: An Integrated
Framework for Creating and Distributing Dialogue Scenarios

123

Emmanuel Ayedoun, Yuki Hayashi and Kazuhisa Seta
Learning Support System to Understand Others Through Dramatic Script Reading and Its
Evaluation
Hanano Okamoto, Yuki Hayashi and Kazuhisa Seta

126

Development of Estimation Method for Learner’s Emotional Concealment During Learning
Using Biometric Information and Feedback Model
Koichi Shinohara, Keiichi Muramatsu and Tatsunori Matsui

132

Development of a Learning Support System for playing Ryuteki in Gagaku for Beginners
Yasushi Ueno and Masato Soga

135

An Adaptive Learning Support System based on Ontology of Multiple Programming Languages
Lalita Na Nongkhai, Jingyun Wang and Takahiko Mendori

141

Examination of the Robot's Role as a Helper in Learning Situations
Makoto Shiraishi and Tatsunori Matsui

144

An Improved Ant Colony Optimization Algorithm for Multi-Granularity Learning Path
Recommendation in E-Learning
Yaqian Zheng, Deliang Wang, Yaping Xu, Ziqi Mao, Yaqi Zhao and Yanyan Li

147

Developing Effective Educational Chatbots with ChatGPT: Insights from Preliminary Tests in a

150

Case Study on Social Media Literacy
Cansu Koyuturk, Mona Yavari, Emily Theophilou, Sathya Bursic, Gregor Donabauer, Alessia
Telari, Alessia Testa, Raffaele Boiano, Alessandro Gabbiadini, Davinia Hernandez-Leo, Martin
Ruskov and Dimitri Ognibene

Table of Content
C2: ICCE Sub-Conference on Computer-supported Collaborative Learning(CSCL) and Learning Sciences

153

Tackling Unserious Raters in Peer Evaluation: Behavior Analysis and Early Detection with
Learner Model
Changhao Liang, Izumi Horikoshi, Rwitajit Majumdar and Hiroaki Ogata

154

Future City: A Simulation for Making SDGs Action Decisions
Pin-Chen Chen, Yu-Hao Lu and Ju-Ling Shih164

Multimodal Assessment of an Ultra-Brief Practice of Progressive Muscular Relaxation Adapted
for the Classroom
Marc Beardsley, Batuhan Sayis and Marta Portero Tresserra

167

The Development and Evaluation of the Platform for Online High-Level Cooperative Games
Feng-Lung Liu, Geng-De Hong, Ju-Ling Shih and George Ghinea

177

A Thematic Analysis Exploring Flexibility in Programming-based Mathematical Problem Solving
Huiyan Ye, Oi-Lam Ng and Zhihao Cui

183

Epistemic Network Analysis to assess collaborative engagement in Knowledge Building
discourse
Aloysius Ong, Chew Lee Teo, Alwyn Vwen Yen Lee and Guangji Yuan

189

Investigating Trustworthiness and Conflict in Historical Multiple Texts: From Eye-Tracking Data

199

of Source and Content Processing
Zheng-Hong Guan and Sunny S. J. Lin
Multimodal Analysis of Learners' Communications in CSCL of a Mathematical Proof
Masataka Kaneko, Hironori Egi and Takeo Noda

206

Development and Evaluation of a Matching System to Facilitate Online Collaborative Learning
Haruka Tada and Fumihide Tanaka

209

Assessing college students’ sense of community for advancing community knowledge
Chih Hui Seet, Yi-Ning Tsai, Guo-Tsai Hung and Huang-Yao Hong219

Teachers Developing More Creative Learning Views via Online Perspective Taking Activities

222

Mei-Ju Chen, Chao-Yu Guo and Huang-Yao Hong
Unveiling University Students' Data Literacy: A Case Study on Modeling Reasoning in Data

225

Mining Projects
Tianqi Zhang
Mathematic Learning-by-teaching: Video Creation and Cross-Schools Staging
Te-Yang Chou, Yen-Cheng Yeh and Hong-Ming TU

235

Table of Content
The Role of Individual Preparation for Knowledge Construction in Collaborative Argumentation:238

an Epistemic Network Analysis
Wenli Chen, Junzhu Su, Qianru Lyu, Siew Cheng Aileen Chai, Xinyi Li, Guo Su and Eng Eng Ng
Students Know more than They can Tell: Understanding Learners’ Ideas of Heat Transfer via
Model Revision Activities
Rajashri Priyadarshini, Chandan Dasgupta and Sahana Murthy

244

Impact of 360°VR on Pre-Service Teachers’ Empathy——Taking Educational Equity as an
exampleYanjun Chen, Yiling Hu and Bian Wu

250

Enhancing Student Teachers’ Collaborative Interdisciplinary Design Through Knowledge-

253

Building Activities
Pei-Yi Lin
From Individual Ideation to Group Knowledge Co-Construction: Comparison of High- and Lowperforming Groups
Wenli Chen, Guo Su, Xinyi Li, Qianru Lyu, Junzhu Su, Siew Cheng Aileen Chai and Eng Eng Ng

256

Discussion Support Agent System to Promote Equalization of Speech among Participants
Ryunosuke Nishimura, Risa Iharada, Yuya Sugamoto, Yutaka Ishii, Toshio Mochizuki and
Hironori Egi

266

Experimental Verification of "Peer-ness" Formation by a Learning Companion Robot

272—

Possibility of inducing a sense of competition through long-term nonverbal interaction—
Koki Honda, Yoshimasa Tawatsuji and Tatsunori Matsui
Study on The Development of Computational Thinking Decomposition Strategies for SeniorPrimary Students
Mengtao Li, Yaxin Guan and Guang Chen

278

Remembering The Knowledge of Experts and Novices in Computer-Supported Collaborative
Learning: A Multinomial Processing Tree Approach
Oktay Ülker and Daniel Bodemer

286

Exploring Group Formation Strategies in Computer-Supported Collaborative Learning: A

292

Systematic Review
Jiamin Tang, Huihan Zhou, Yajing Tan and Guang Chen
A Comparative Analysis on the Effects of Cognitive Tools in Data Inquiry Cultivation

295

Hui Zhang, Bian Wu, Yiling Hu and Yujie Xu
Argumentative Knowledge Construction and Certainty Navigation: A Comparison between
Individual and Group Work
Wenli Chen, Eng Eng Ng, Guo Su, Junzhu Su, Xinyi Li, Siew Cheng Aileen Chai and Qianru Lyu

301

Table of Content
C3: ICCE Sub-Conference on Advanced Learning Technologies (ALT),
Learning Analytics and Digital Infrastructure

307

ChatGPT’s Performance in Spreadsheets Modeling Assessments based on Revised Bloom’s
Taxonomy
Michelle Cheong308

Promoting Middle School Students' Achievement and Attitude toward Science Learning throughSphere Recognition-Based AR Application
Ruixue Liu, Lijun Liang and Xiaodong Wei

318

Preparation for Future Lockdowns: A Comparison of Student LMS Activity During and After
COVID-19

328

Ezekiel Adriel Lagmay and Maria Mercedes T. Rodrigo
Using Augmented Reality to Facilitate Music Learning for Preschool Children
Xiaodong Wei, Rui Qiu and Ruixue Liu

338

Comparing Perceived Cognitive Load while Learning Online with AI Chatbots, Pre-recorded
Videos, and Live Lectures
Haixi Sheng, Xinran Zhou, Yue Zhao and Guoqing Zhao

344

Do the Same Rules Apply? Transferring MOOC Success Behaviors to University Online
Learning
Clarence James Monterozo and Maria Mercedes Rodrigo

350

Visual Attention Patterns in Processing Compiler Error Messages
Christine Lourrine Tablatin and Maria Mercedes Rodrigo

356

Process Evaluation for Concept Map Building and Its Experimental Evaluation
Ridwan Rismanto, Aryo Pinandito, Banni Satria Andoko, Yusuke Hayashi and Tsukasa Hirashima

362

On-demand lectures with humor and questions using avatars
Satoshi Toyota and Asuka Terai

372

Adapting Noticing Framework to Analyze Learner's Reasoning in VR-simulated complex
scenarios

382

Herold Pc and Chandan Dasgupta
Analysis of Algorithmic Strategy Development in the Development of Computational Thinking ofUpper Elementary School Students

388

Xiaowen Wang, Pinqi Hu and Guang Chen
Toward a Virtual Human Exhibit for Public AI Education
Ning Wang, Timothy Hurt, Ari Krakowski, Eric Greenwald, Omkar Masur, Boxi Fu and Chirag
Merchant

394

Table of Content
Analyzing Learning Patterns and Potential Interventions in First-Year Compulsory Course at an

397

Online University
Yasuhisa Kato
A Comparative Study of Traditional and Augmented Reality-Based Engineering Drawing
Instruction: Effects on Visualization Skills and Cognitive Load
Ajay Shankar Tiwari and Kaushal Kumar Bhagat

400

Measuring Self-regulated Learning Processes in Computer Science Education
Elizabeth Cloude, Ryan Baker and Maciej Pankiewicz

406

Construction of a Japanese Language Learning Support System for Learning Semantic
Negotiation

409

Satoru Kogure, Akira Yoshida, Yasuhiro Noguchi, Koichi Yamashita, Tatsuhiro Konishi and
Makoto Kondo
Creating Meaningful Connections: The Role of Simultaneous Multi Situational Learning in
Knowledge Contextualization and Application
Chih-Yen Chen, Su-Hang Yang, Meng-Xuan Xie, Yi-Chuan Fan, Jen-Hang Wang and Gwo-Dong
Chen

412

ECLAIR: A Centralized AI-Powered Recommendations System in a Multi-Node EXAIT System
Isanka Wijerathne, Brendan Flanagan, Yiling Dai and Hiroaki Ogata

422

Sharing Learning Log while Maintaining Privacy over Blockchain: Heuristic Evaluation of

429

BOLL
Patrick Ocheja, Rwitajit Majumdar, Brendan Flanagan and Hiroaki Ogata
Program Visualization System Supporting Teacher-Intended Stepwise Refinement
Koichi Yamashita, Hiroki Soma, Satoru Kogure, Yasuhiro Noguchi, Raiya Yamamoto, Tatsuhiro
Konishi and Yukihiro Itoh

435

Automatic Detection of Negotiation in Collaborative Complex Problem Solving Interactions
Daevesh Singh, Ulfa Khwaja, Sahana Murthy and Ramkumar Rajendran441

A page jump recommendation model based on digital textbook contents and student log data
Wenhao Wang, Natsumi Yamamoto, Fuzheng Zhao, Etsuko Kumamoto, Zicheng Kang and451

Chengjiu Yin
Concept Map Recomposition Approach for Advanced Formative Assessment in Large-Scale
Online Course

457

Sho Yamamoto, Aryo Pinandito and Tsukasa Hirashima
Towards Automated Evidence Extraction: A Case of Adapting SAM to Real-World Educational
Data
Koki Okumura, Izumi Horikoshi, Kento Koike and Hiroaki Ogata

463

Table of Content
What does process mining of feedback-seeking reveal about problem-solving in chemistry472

undergraduates?
Anveshna Srivastava and Chandan Dasgupta
ICCE 2023 Learning Outcomes of Computer Programming and Information Technology Integrated Courses for Non-Computer Science Majors: Case Study of a Public Research
University in Taiwan
Che-Yu Hsu, Feng-Nan Hwang, Tseng-Yi Chen and Chia-Hui Chang

475

DLOT: An Open-Source Application to Assist Human Observers
Ashwin T S, Danish Shafi Shaikh and Ramkumar Rajendran478

Supporting Learning Through Affordance-Based Design: A Comparative Analysis of "BioVARse"484

and a Standard Textbook Companion Application in Biology Education
Devanshu Saindane, Sunny Prakash Prajapati and Syaamantak Das
Investigating Programming Performance Predictability from Embedding Vectors of Coding
Behaviors
Ikkei Igawa, Yuta Taniguchi, Tsubasa Minematsu, Fumiya Okubo and Atsushi Shimada

487

Unveiling Learners’ Interaction Behavior in Virtual Reality Learning Environment
Antony Prakash and Ramkumar Rajendran

490

Keeping Teams in the Game: Predicting Dropouts in Online Problem-Based Learning
Competition

500

Aditya Panwar, Ashwin T S, Ramkumar Rajendran and Kavi Arya

C4: ICCE Sub-Conference on Classroom, Ubiquitous, and Mobile
Technologies Enhanced Learning (CUMTEL)

510

Fostering Students’ Dialogic Engagement with the Use of Visual Learning Analytics as a
Teaching Assistant Tool in Primary School Classrooms
Fan Chen, Pengjin Wang, Deliang Wang and Gaowei Chen

511

Advancing Education through Stakeholder Engagement: An Evaluation of the Learning Butler
Chatbot's Impact on Instructors, and Learners
Chih-Yang Peng, Su-Hang Yang, Pei-Yu Ho, Jen-Hang Wang and Gwo-Dong Chen

521

Building Students’ Learning Habits on Slack: An Application of the IDC Theory
Veenita Shah, Sahana Murthy and Sridhar Iyer

527

Using the Self-regulated Based Personalized Online Learning System for Learning Factorizationin Mathematics
Thanyaluck Ingkavara, Patcharin Panjaburee and Wararat Wongkia

537

Table of Content
The Effects of Visualization Strategies on Students’ Learning Outcomes in Augmented Reality

543

Contexts
Pei-Shan Tsai
Teaching Analytics with xAPI: Learning Activity Visualization with Cross-platform Data
Izumi Horikoshi, Yuko Toyokawa, Kohei Nakamura, Changhao Liang, Rwitajit Majumdar andHiroaki Ogata

548

Effects of Audio and Tactile Biofeedback Based on EEG Attention Levels on University Students’
Relaxation
Shih-Jou Yu, Wai Ki Rebecca Cheng, Yi-Hsuan Chen and Jerry Chih-Yuan Sun

554

Estimating Physical Interactions with Neighboring Student for Detecting Active Learners in the

560

Computer Classroom
Takahiro Yoshino, Shin Ueno and Hironori Egi
Effect of Active Breaks during e-Learning and Mental Arithmetic Tasks
Masaki Kodaira, Tatsuya Hamada and Hironori Egi

563

Chronotypes of Learning Habits in Weekly Math Learning of Junior High School
Chia-Yu Hsu, Mandukhai Otgonbaatar, Izumi Horikoshi, Huiyong Li, Rwitajit Majumdar andHiroaki Ogata

566

C5: ICCE Sub-Conference on Educational Gamification and Game-based
Learning (EGG)

569

Relationship Between Students’ Minecraft Re-engagement Metrics and STEM Interest
Maria Mercedes T. Rodrigo, Jonathan Casano and Mikhail Fuentes

570

Enhancing Learning Experience in University Engineering Classes with Kahoot! Quiz Games
Zilu Liang

573

Design and Implementation of an Educational Game for Teaching Artificial Intelligence to HighSchool Students
Ning Wang, Ryan Montgomery, Eric Greenwald and Maxyn Leitner

579

Executive Functions Training-oriented Digital Games: Effectiveness and Experience589

Chi-Fang Huang, Hui-Ling Hsu and Zhi-Hong Chen
Incorporating Tangible Rewards into Gamification Increases Students’ Identified Regulation in
Fully Online Learning

595

Ya Xiao and Khe Foon Hew
Effects of Different Embodied Scaffoldings on Students' Spatial Abilities in Digital Game-Based
Learning
Junyi Zhou, Jialing Zeng, John B. Black and Junjie Shang

601

Table of Content
Exploring the Impact of Designing a Robot as a Pet with Interdependence Theory on Long-Term

611

Relationships and Learning Performance
Vando Gusti Al Hakim, Su-Hang Yang, Jen-Hang Wang, Yu-Chen Chang, Hung-Hsuan Lin and
Gwo-Dong Chen
Analysis to Creation: Using the ADDIE Model to Develop an Educational Game for Children
Yufan Zhang, Nurul Nadwa Zulkifli, Ahmad Fauzi Mohd Ayub and Zewen Shang

621

Case Study-Based Research on Understanding App User Engagement to Develop Environmental
Literacy of Urban Residents
Ewa Duda

624

Design and Development of A Game to Improve Self-Efficacy: A Case Study of Addressing

627

Modes Learning
Fuzheng Zhao, Danqing Luo, Etsuko Kumamoto and Chengjiu Yin
Investigating Cognitive Biases in Self-Explanation Behaviors during Game-based Learning about
Mathematics
J. M. Alexandra Andres, Elizabeth Cloude, Ryan Baker and Seiyon Lee

637

GaMINLab - Meaningful Gamification to Engage Students in Science Inquiry Practices throughSimulation Labs
Archana Rane, Sahana Murthy and Sasikumar M.

643

A Skill Tracing Model for Player Character Control in STG

646

Peizhe Huang, Wanxiang Li, Wen Gu, Kouichi Ota and Shinobu Hasegawa
The Design and Practice of Scientific Inquiry Activities for Children Aged 5-6 Based on an AR
Flashcard Environment
Tian-Tian Gou, Min-Sheng Fan and Bin-Li Wang

649

Toward Game-Based Learning of Japanese Writing for Elementary School Students
Kazumasa Omura, Kei Kubo, Frederic Bergeron and Sadao Kurohashi

655

The Impact of Digital Game-based Learning with a Mathematical Game Application on
Calculation Abilities of Grade 4 Students
Yinbei Liu and Alex Wing Cheung Tse

661

Develop and validate STEM education activities using the “6E Design Teaching Model”: Taking“Dynamics and Energy Conversion in Sail Car Design” as an example
Lu Chen, Yang Yang and Guang Chen

671

The Impact of Gamified Assessment on the Learning Burnout of Undergraduate Computing
Students: a Quasi-experimental Research
Beilei Zhang and Alex Wing Cheung Tse

686

Table of Content
Enhancing Learner Satisfaction in Simulation-Based Learning: The Impact of Learner

696

Characteristics and Expectancy
Steven Ck Ng, Li Fern Tan and Poh Nguk Lau
Exploring the Impact on Student Reading Preferences in Gamified Reading Portfolio
Hsiao-Tung Yang, Chang-Yen Liao, Ciao-Min Syu and Tak-Wai Chan704

Improving Teamwork through a Decision-Theoretic Coach in a Minecraft Search-and-Rescue
Game
David Pynadath, Nik Gurney, Sarah Kenny, Rajay Kumar, Stacy Marsella, Haley Matuszak, Hala
Mostafa, Pedro Sequeira, Volkan Ustun and Peggy Wu

707

C6: ICCE Sub-Conference on Technology Enhanced Language Learning(TELL)

710

Using Flipped Classroom Approach to Integrate SRL Instruction in Classical Chinese Reading
Instruction: Insights from the First-Year Results
Kit Ling Lau

711

Relations between Instructional Factors and Student Acceptance of Flipped Learning in Chinese
Language Learning
Kit Ling Lau and Quan Qian

717

The AI-Supported Instructional Design in PBL Integrating Chinese Language Learning andMultimedia Creation

726

Satoko Sugie
Experimental Comparison of Promotion Effect for EFL Reading Comprehension between
Conventional Summarization and Toulmin Argument Reconstruction
Banni Satria Andoko, Rosa Andrie Asmara, Vivin Ayu Lestari, Deasy Sandhya Elya Ikawati, Arief
Prasetyo, Tsukasa Hirashima and Yusuke Hayashi

732

Effects of a Machine Learning-empowered Chinese Character Handwriting Learning Tool on
Rectifying Legible Writing in Young Children: A Pilot Study
Lung-Hsiang Wong, Guat Poh Aw, He Sun, Ching-Chiuan Yen, Chor Guan Teo and Yun Wen

742

Using TAASSC to Investigate Fine-Grained Complexity in Reading Texts of Two High-Stakes

748

English Tests in China
Shengshu Lin
Using Multidimensional Analysis to Investigate the Extrapolation Inference of a High-Stakes Test
Shengshu Lin

754

Enabling Visually Impaired People’s Chinese Literacy learning through Information Technology
Shelley Shwu-Ching Young and Jen-Li Wang

760

Table of Content
Tracking Knowledge for Learning Japanese as a 2nd Language

766

Tomoko Okimoto, Matthew Johnson, Huy Nguyen, Steven Moore, Michael Eagle and John
Stamper
The Effect of Timing Differences in Online Corrective Feedback on Adult Verbal English
Learners' Learning Engagement: A Micro-genetic Study
Wanying Liang, Guang Chen and Wei Cheng

769

Using Self-Regulated Digital Storytelling in Primary Students’ English Learning: An Exploratory
Factor Analysis
Yunsi Tina Ma, Siu Cheung Kong and Daner Sun

779

Constructing a Natural Conversation Learning Activity to Improve Students’ English-Speaking

789

SkillsHong-Min Tu, Chang-Yen Liao and Te-Yang Chou
Analysis of Topic Sentences of Classification Paragraphs and Development of a Diagnostic
Function
Kandai Ishikawa and Hidenobu Kunichika

792

Construction of an English Grammar Quiz Recommendation System Using Explanation by a
Knowledge Map
Kensuke Takii, Naomichi Tanimura, Brendan Flanagan and Hiroaki Ogata

795

Educational System of English Tense for Japanese Learners by Forming Temporal Constraints on

801

Tense
Haruto Nagata, Tatsuhiro Konishi and Makoto Kondo
Exploring the Use of Chatbot to Promote Online EFL Students’ Behavioral, Cognitive, andEmotional Engagements
Xinyi Luo, Weijiao Huang, Khe Foon Hew, Chengyuan Jia and Xiangjie Cao

807

Impact of Self-analysis Behaviors in GOAL for Japanese High School EFL Learners
Zixu Wang, Rwitajit Majumdar and Hiroaki Ogata

813

AI-powered Collaborative Activities for Chinese Vocabulary Learning
Xinyu Guo and Yun Wen

819

Design and Development of a Sentence Construction Game for Deaf and Hard of Hearing (DHH)
Users: A Qualitative Usability Study
Arjun Prasad, Sunny Prajapati, Utkarsh and Vishwas Badhe

825

Using T-Robot Board Games to Enhance Learning Gains of Rural Elementary School Students
Yu-Wei Wu, Wen-Yu Ye, Wen-Chi Vivian Wu and Rong-Jyue Wang

831

Table of Content
C7: ICCE Sub-Conference on Practice-driven Research, Teacher Professional
Development and Policy of ICT in Education (PTP)834

How Teachers’ Conceptions of Student Engagement Influenced their Actual Strategy
Implementation, and Student Online Engagement
Gulipari Maimaiti and Khe Foon Hew

835

ICCE 2023 Exploring the Social Media Discourse: the Impact of ChatGPT on Teachers' Rolesand Identity
Yuchun Zhong, Davy Tsz Kit Ng and Samuel Kai Wah Chu

838

Designing Faculty Development Programs by a Team from Different Majors
Yukari Kato, Yukari Yukihiko, Hisashi Hatakeyama and Toshiya Oishi

849

What Digital Tools Teachers are Ready to Use in Kindergarten – International Comparative Studywith Early Childhood Pre-Service Teachers
Natalia Demeshkant, Siri Sollied Madsen, Aleksander Janeš, Andreja Klančar, Rita Brito, Ahmet
Sami Konca, Sergey Krasin, Heidi Iren Saure, Jane O'Connor, Mustafa Jwaifell, Steinar
Thorvaldsen and Sławomir Trusz

855

Democratising AI Education: Teaching Autoencoders to Out-of-School Children from Lowincome Backgrounds
Saumay Garg

861

Informatics Education for University Students based on Text Input Time

867

Yuko Murakami and Tomohiro Inagaki
Investigating Student Teachers’ Learning Experience and Choice of Participation Modalities in a
HyFlex Course: A Mixed Methods Approach
Liang Jing Teh, Su Luan Wong, Mohd Zariat Abdul Rani, Mas Nida MD Khambari and Sai Hong
Tang

873

Assessment of Intelligent Teaching Preparation of EFL Teachers: Based on Two-Year Data
Comparison
Xin An, Xi Shen, Jiannan Bai and Yushun Li

884

Using Learning Design Technologies for Teachers’ Practice-Driven Research

892

Marc Beardsley, Davinia Hernandez-Leo and Roberto Sánchez-Reina
The Impact of Preschool Teacher Training on STEAM Education: Professional Preparedness and898

Confidence
Wan-Chen Chang
Matching Intervention Messages Considering Complex Personality Types of High SchoolStudents
Taisei Yamauchi, Yuta Nakamizo, Kyosuke Takami, Rwitajit Majumdar and Hiroaki Ogata

901

Table of Content
Design-Based Implementation Research: A Collaborative Approach to Educate Out-of-School907

ChildrenFaisal Badar and Jon Mason
A Study of Versatile Tutor Training Programs for Universities in Japan
Yumi Ishige, Kazuhiro Kabeya, Kayoko Nagao and Hirotoshi Tanigawa

910

Support for Fitting Chromebooks to the Child with Cerebral Palsy: A Practical Study on
Incorporating Advice from ICT Specialists
Tomohito Yamazaki and Toyokazu Mizuuchi

913

Exploring the Relationship between 21st Century Skills and Motivation: A Study Using
Contextual Inquiry Project-based Learning

916

Jirapipat Thanyaphongphat, Preeyada Tapingkae, Kannika Daungcharone and Krittawaya
ThongkooData-Driven Competency Assessment Supporting System for Teachers
Taito Kano, Izumi Horikoshi, Kento Koike and Hiroaki Ogata

926

Visualization of Instructional Patterns from Daily Teaching Log Data
Kohei Nakamura, Izumi Horikoshi, Rwitajit Majumdar and Hiroaki Ogata

936

Sustaining Students’ Interest in an Instructional System Design Course by Leveraging InterestDriven Creator Theory
Vishwas Badhe, Sunita Raste, Sahana Murthy and Sridhar Iyer

946

Supporting Science Teachers to Select Quality Edtech Learning Solutions for their Context
Shruti Jain, Sheeja Vasudevan, Madhuri Mavinkurve and Sahana Murthy956

A Preliminary Study: Exploring Teachers’ Perspectives on the Role of Gathering Information in
Supporting Teachers’ Digital Learning Agility
Kamilah Abdullah, Mas Nida Md Khambari, Nur Dania Mohd Rosli, Su Luan Wong, Noor
Syamilah Zakaria, Siti Raba'Ah Hamzah and Priscilla Moses

966

The Role of Flexibility in Shaping Teachers’ Digital Learning Agility: A Preliminary Study
Nur Dania Mohd Rosli, Kamilah Abdullah, Mas Nida Md. Khambari, Su Luan Wong, Noor
Syamilah Zakaria, Priscilla Moses and Siti Raba'Ah Hamzah

972

Incorporation of Robotics in AI Education: Barriers and Enablers as Perceived by K-12 Teachers
Miao Yue, Morris Siu-Yung Jong, Yun Dai and Tsun-Hin Luk

978

Collaborative Design of a Simulation-based Math Classroom: Contradictions and Solutions
between Teaching and Research
Wenxin Guo, Bian Wu and Dong Li

981

The Impact of Development-questioning Activities on Students' Pre-writing Ideas
Jia Ling Hong, Chang-Yen Liao, Tak-Wai Chan and Jui-Fen Chang

991

Table of Content
Promoting Teachers’ Digital Literacy Achievement: A Nationwide Survey of Education

997

Informatization in China
Ziyan Che, Jiumei Yang, Longkai Wu and Di Wu
Learning with Conversational AI and Personas: A Systematic Literature Review
Antun Drobnjak, Ivica Boticki, Peter Seow and Ken Kahn

1003

Supporting Content Creators in Creating Accessible Digital Materials in Higher Education
Eirik Hansen and Weiqin Chen

1009

Panel

1015

Towards a Collaborative Vision for Redesigning Education for Harmonious and Thriving

1016

Educational Futures in Asia and Beyond: Will Seamless IDC Theory lead us there?
Chee-Kit Looia, Siu-Cheung Kong, Ronghuai Huang Jon Mason, Hiroaki Ogata, Hyo-Jeong So,
Lung-Hsiang Wong
Mobile Learning: Reflections on the Past and Visions for the Future
Lung-Hsiang Wong, Daner Sun, Hiroaki Ogata, Hyo-Jeong So, Xiaoqing Gu, & Ting-Chia Hsu

1020

AIED/ITS1

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

An Efficient and Generic Method forInterpreting Deep Learning based
Knowledge Tracing ModelsD濙濠濝濕濢濛 WANGa,c, Yu Lu a,b*, Zhi ZHANGa & Penghe CHEN a,b
aSchool of Educational Technology, Faculty of Education, Beijing Normal University, ChinabAdvanced Innovation Center for Future Education, Beijing Normal University, China
cFaculty of Education, The University of Hong Kong, Hong Kong, China*luyu@bnu.edu.cn
Abstract: Deep learning-based knowledge tracing (DLKT) models have been regarded
as the promising solution to estimate learners’ knowledge states and predict their future
performance based on historical exercise records. However, the increasing complexityand diversity make DLKT models still difficult for users, typically including both learners
and teachers, to understand models’ estimation results, directly hindering the model’s
deployment and application. Previous studies have explored using methods fromexplainable artificial intelligence (xAI) to interpret DLKT models, but the methods havebeen limited in their generalizing capability and inefficient interpreting procedures. To
address these limitations, we proposed a simple but efficient model-agnostic
interpreting method, called Gradient*Input, to explain the predictions made by thesemodels in two datasets. Comprehensive experiments have been conducted on theexisting five DLKT models with representative neural network architectures. The
experiment results showed that the method was effective in explaining the predictionsof DLKT models. Further analysis of the interpreting results revealed that all five DLKTmodels share a similar rule in predicting learners’ item responses, and the role of skill
and temporal information was found and discussed. We also suggested potential
avenues for investigating the interpretability of DLKT models.
Keywords: Knowledge tracing models, deep learning, explainable artificial intelligence

1. Introduction
The ability to automatically identify learners’ knowledge states is crucial for personalized
learning, and it plays a fundamental role in sustaining learning motivation (Pelánek et al.,2017) and improving academic performance (Koedinger & Aleven, 2016). To achieve this,researchers have developed a range of knowledge tracing (KT) models by leveraging onlearners’ historical exercise records to predict their future performance. With theadvancement of artificial intelligence (AI), KT models that employ deep learning techniques
are considered effective due to their strong capability to capture inherent information.
However, the complex structures and large number of variables in the deep learning-basedknowledge tracing (DLKT) models make them difficult for users (e.g., teachers, students,and education researchers) to understand the models’ decisions (Tsai & Gasevic, 2017),
which may reduce users’ trust and accordingly hinder the DLKT models’ deployment andapplication, as indicated in the case of automated recommendation systems (Dietvorst et al.,2015). Additionally, blindly trusting the incorrect decisions would cause a wrong diagnosis ofknowledge status and accordingly reduce learning efficiency.
To address the interpretability issue in deep learning knowledge tracing (DLKT)models, researchers have started exploring solutions. One approach is the incorporation
component of the model remains a black box. Another approach is using the explainableartificial intelligence (xAI) techniques to interpret DLKT models, such as the model-specificlayer-wise relevance propagation (LRP) method (Lu et al., 2020; Lu et al., 2022), but it is2

only applicable to the specific DLKT models and hard to be generalized to other DLKTmodels. The DeepSHAP method has been also proposed as a more generic method forinterpreting DLKT models (Wang et al., 2022), but it heavily relies on reference samples,
which are normally scarce in the real cases. In addition, it is still in lack of systematic study
on interpreting the existing representative DLKT models, typically including the recurrent
neural networks (RNNs), memory-augmented neural networks (MANNs), attention, graphneural networks (GNNs), and convolutional neural networks (CNNs).
We thus propose a cost-effective and generic method to interpret the diverse DLKTmodels, where the model-agnostic interpreting method is designed to explain the predictions
made by the five representative DLKT models. The comprehensive experiments havevalidated the effectiveness of the proposed method. Further investigations show that despite
their different configurations, the five DLKT models follow a similar rule for estimating
learners’ knowledge states and making predictions.

2. Related Work
2.1 Deep Learning based Knowledge Tracing (DLKT) Models
DKT (Piech et al., 2015) was the pioneer DLKT model, and it utilized RNNs to achieve
superior performance compared to traditional Bayesian knowledge tracing (BKT) (Corbett &Anderson, 1994). Researchers subsequently improved upon the DKT model by incorporating
additional input features such as question difficulty (Sonkar et al., 2020) and extending themodel’s structure such as adding another layer of RNN (Su et al., 2018). Other types of deepneural networks have also been adopted for use in DLKT models. For example, because
DKT summarizes the states of all concepts in a single vector, memory-augmented neuralnetworks (MANNs) were used to build the DKVMN (Zhang et al., 2017), SKVMN(Abdelrahman & Wang, 2019), and EKT (Liu et al., 2019) models, which store the status ofeach concept in a separate space. Due to the limited performance of RNN-based and
MANN-based KT models on concepts with little data, the attention mechanism has been
employed to extract the similarity between different concepts and questions, leading to thedevelopment of the SAKT (Pandey & Karypis, 2019) and AKT (Ghosh et al., 2020) models.
To capture the interconnected relationship between concepts and questions, graph neuralnetworks (GNNs) have been used to build relationship graphs, resulting in the creation of the
GKT (Nakagawa et al., 2019) and GIKT (Y. Yang et al., 2020) models. Additionally,
convolutional neural networks (CNNs) have been explored for use in DLKT models, resulting
in the development of the CKT (Shen et al., 2020) and CAKT (S. Yang et al., 2020).

2.2 Interpreting Methods for Deep Learning Models
Researchers have developed multiple interpreting methods to understand the internal
workings and individual decisions of non-transparent deep learning models. These
interpreting methods can be classified into model-specific and model-agnostic approaches(Adadi & Berrada, 2018). The former ones are tailored to the models with specific structures,
while the latter ones can be applied to a wider range of models. Model-agnostic interpreting
techniques include visualization (Goldstein et al., 2015), knowledge extraction (Hinton et al.,2015), influence methods (Cortez & Embrechts, 2011), and example-based explanations(Wachter et al., 2017). Influence methods are commonly used to explain models by
estimating the importance or contribution of a feature to the model’s prediction. Popularmethods in this category include perturbation-based and backpropagation-based
approaches, such as Gradient (Simonyan et al., 2013) and DeepSHAP (Lundberg & Lee,2017).
Although there are a large number of interpreting methods in the field of explainable AI(xAI), currently only two methods (i.e., LRP and DeepSHAP) have been utilized to interpret
DLKT models (Lu et al., 2020; Wang et al., 2022). These methods are limited in theirgeneralizing capability or their high computational complexity, leading to the need for simpler
yet effective methods to provide explanations of the diverse and complex DLKT models.

33. Interpreting Method
3.1 Method
We propose to utilize an existing xAI method, called Gradient*Input (Arras et al., 2019), forinterpreting DLKT models. Compared to other methods previously investigated for DLKTmodels (i.e., LRP and DeepSHAP), Gradient*Input has two advantages. First, it can beapplied to a wide range of models regardless of their internal structure. Second, it has
relatively low computational complexity, requiring only one forward pass and one backward
pass to obtain an explanation, and does not rely on auxiliary information, such as thereference samples used in DeepSHAP. And this method has been demonstrated to beeffective in various tasks (Arras et al., 2019; Shrikumar et al., 2017).
The Gradient*Input method explains the model prediction by decomposing it into thecontributions of the input features. Formally, given an input ‫ ݔ‬with ‫ ܭ‬features, the predictionof a deep learning model on class ܿ is a highly non-linear function ݂௖ (‫) ݔ‬. Based on Taylor
expansion (Li et al., 2016; Montavon et al., 2017), the non-linear prediction ݂௖ (‫ ) ݔ‬isapproximated by a linear function, as shown in Equation 1:
డ௙೎ (௫)݂௖ (‫ ) ݔ‬ൎ σ௄‫ݔ‬௜ + ܾ
(1)
௜ୀଵ
డ௫೔

where ݅ represents the i-th feature in the input, σ௄௜ୀଵడ௙೎ (௫)డ௫೔

represents the partial derivative of݂௖ (‫ ) ݔ‬with respect to ‫ݔ‬௜ , and ‫ݔ‬௜ reflects the contribution of feature ‫ݔ‬௜ to the prediction.

((a)) Forward Prediction(b) Compute Feature Derivatives(c) Element-wise Product(d) Perform a Sum Operation

Figure 1. The interpreting procedure of Gradient*Input for DLKT models.

3.2 Explaining DLKT Models
Figure 1 illustrates the procedure of using Gradient*Input to interpret the predictions of DLKTmodels. Specifically, given a set of question-answer records from learners, denoted as {(‫ݍ‬ଵ ,ܽଵ ), (‫ݍ‬ଶ , ܽଶ ), ..., (‫ݍ‬௡ , ܽ௡ )}, DLKT models can make predictions about their future performance,e.g., the probability ݂(݊ + 1) of correctly answering a future question ‫ݍ‬௡ାଵ , as Figure 1(a)shows. To explain the prediction, Gradient*Input uses backpropagation to calculate thepartial derivative of the prediction with respect to the input features, as depicted in Figure1(b). Then, the element-wise product between the feature and its derivative is performed,
which allows it to obtain the feature relevance (i.e., contributions) for each question-answer
record, as Figure 1(c) indicates. Because each record is represented as a vector (e.g., anembedding or a one-hot vector with m dimensions), the sum of the feature relevance in aquestion-answer record can be used to determine the overall relevance of the record to theprediction (i.e., QA relevance), as shown in Figure 1(d).

44. Evaluation
4.1 Construction of DLKT Models
We select five typical and representative existing DLKT models for the experiment, including(1)澳DKT (Piech et al., 2015), the first DLKT model that adopts RNN as its inner structure; (2)
DKVMN (Zhang et al., 2017), a DLKT model that adopts MANN to store concept status; (3)AKT (Ghosh et al., 2020), a DLKT model that employs attention to extract similarity between
concepts and questions; (4) GKT (Nakagawa et al., 2019), A DLKT model that uses GNN tobuild skill relationship graphs; (5) CKT (Shen et al., 2020), A DLKT model that utilizes CNN
to model individualization.
4.1.1 Dataset
We adopt two commonly used KT datasets, ASSISTment2009 and ASSISTment2015 (Fenget al., 2009). Specifically, the datasets were preprocessed to eliminate repetitive data anddata without skill or question tags, and the length of the learner answer sequence was set torange from 10 to 200. After preprocessing, ASSISTment2009 contained 325,637 records on
110 skills and ASSISTment2015 contained 682,223 records on 100 skills. 80% of the data
was randomly chosen for training, while the remaining 20% was used for testing.
4.1.2 Model Training
For all five models, we uniformly set the optimizer, dropout rate, mini-batch size, initial
learning rate, and iteration epoch to Adam, 0.5, 64, 0.005, and 100, respectively. For the
DKT model, the hidden dimension was set to 64. For the DKVMN model, the state
dimension was set to 64 and the memory size was set to 110 for the ASSISTment2009
dataset and 100 for the ASSISTment2015 dataset. For the AKT model, the hidden
dimension was set to 256 and the number of heads was set to 8. For the GKT model, thehidden dimension was set to 64 and the number of heads was set to 4. For the CKT model,
the hidden dimension was set to 64. The performance (i.e., AUC and accuracy) of these five
DLKT models in ASSISTment2009 and ASSISTment2015 can be seen in Table 1. Note thatgiven the focus of this work is not on the model performance, we do not optimize theaccuracy for each model.
Table 1. The performance of five DLKT models in ASSISTment2009 and ASSISTment2015.
Dataset
RNN-based DKTMANN-based DKVMNAttention-based AKT
GNN-based GKT
CNN-based CKTMetric
AUC
ACC
AUC
ACC
AUC
ACC
AUC
ACC
AUC
ACC

ASSISTment2009
0.74
0.72
0.76
0.74
0.75
0.72
0.74
0.73
0.75
0.72

ASSISTment2015
0.72
0.73
0.72
0.75
0.73
0.75
0.72
0.74
0.72
0.74

4.2 Method Validation
We first validate the capability of the proposed method on interpreting the decisions of DLKTmodels. We split the test data for both ASSISTment2009 and ASSISTment2015 intosequences of 15 question-answer records, resulting in 48,670 sequences for
ASSISTment2009 and 97,637 sequences for ASSISTment2015. In each sequence, the first5

14 records were used as input to predict the correctness of the last record, allowing us to
identify correctly-predicted sequences. The number of positive and negative predictions
made by all five DLKT models in each dataset is shown in Table 2. By applying the proposed
method, we could calculate the relevance of each question-answer record to the prediction.
Table 2. The number of positive and negative predictions in correctly-predicted sequences.
DLKT Models
RNN-based DKTMANN-based DKVMNAttention-based AKT
GNN-based GKT
CNN-based CKTASSISTment2009
Positive Negative
26,2588,777
27,477
7,660
27,271
7,628
27,005
8,066
26,6298,679

Sum35,03535,13734,89935,07135,308ASSISTment2015
Positive Negative
60,322
10,86562,975
8,694
61,787
9,314
59,690
11,22963,101
8,992

Sum
71,187
71,66971,101
70,91972,093Figure 2. Question-answer record deletion results for DLKT models in ASSISTment2009.

Figure 3. Question-answer record deletion results for DLKT models in ASSISTment2015.
To validate the effectiveness of the relevance of each question-answer record, we
further conducted the experiment by removing question-answer records with high relevancefrom each sequence and observing the resulting change in accuracy. Specifically, we
deleted question-answer records in the descending order of relevance for positive
predictions (i.e., predictions of a correct response in the last question), and in the ascendingorder of relevance for negative predictions (i.e., predictions of a wrong response in the last
question). By observing the change in accuracy, we could evaluate whether question-answer

6

records with high relevance were more important for model predictions. We also performed
random removal of records for comparison.
Figure 2 and 3 show the results of deleting question-answer records for all five DLKTmodels in the ASSISTment2009 and ASSISTment2015 datasets, where DLKT-GI represents
the proposed method and DLKT-R means random deletion. As can be seen, compared torandom deletion, the proposed method leads to a significant drop in accuracy. For example,
in the case of the DKT model, removing 5 question-answer records based on relevancecauses the prediction accuracy to drop from 100% to around 57%, while random deletion
only slightly reduces the accuracy to around 95%. Similarly, for other DLKT models,deletions based on relevance cause the accuracy to drop to a range between 65% and 71%,
while random deletions only reduce the accuracy to around 95%. Overall, the results
suggest that the proposed method is effective in explaining the decisions of all five DLKTmodels.

4.3 Interpreting Model Rules
With the validated QA relevance, we further interpret how the DLKT models makepredictions. We mainly analyze how the model input question-answer records influence themodel output, i.e., the predictions on learners’ future performance. In particular, we consider
the effects of both the skill information (i.e., the specific skill being tested) and temporal
information (i.e., when the exercise was completed) on the model’s decision.
We selected correctly-predicted sequences and set the skill and position of the lastrecord in each sequence as the target skill and prediction position. Then each questionanswer record in the sequence was tagged as either Same skill or Different skill based onwhether its skill matched the target skill, and Recent (i.e., the first half QA records) andDistant (i.e., the second half QA records) based on its distance from the prediction position.
This resulted in four groups (i.e., 2 × 2): Distant & Same skill, Distant & Different skill, Recent& Same skill, and Recent & Different skill. For each group, the absolute relevance of eachquestion-answer record was summed to compute the mean. We computed and compared
the mean of four groups among all correctly-predicted sequences across the five DLKTmodels in the ASSISTment2009 and ASSISTment2015 datasets.

(a) ASSISTment2009(b) ASSISTment2015

Figure 4. Skill-distance relevance comparison for RNN-based DKT.

(a) ASSISTment2009(b) ASSISTment2015

7

Figure 5. Skill-distance relevance comparison for MANN-based DKVMN.

(a) ASSISTment2009(b) ASSISTment2015

Figure 6. Skill-distance relevance comparison for Attention-based AKT.

(a) ASSISTment2009(b) ASSISTment2015

Figure 7. Skill-distance relevance comparison for GNN-based GKT.

(a) ASSISTment2009(b) ASSISTment2015

Figure 8. Skill-distance relevance comparison for CNN-based CKT.
The results, shown in Figure 4, 5, 6, 7, and 8, indicate that for all five DLKT models,
the recent-same-skill records (i.e., records that are close to the prediction position and on
the target skill) have the highest relevance to the prediction, while those distant-different-skill
records (i.e., records that are far from the prediction position and on non-target skills) havethe lowest relevance. For example, in the case of AKT, the relevance of recent-same-skill
records is approximately 0.036 in both the ASSISTment2009 and ASSISTment2015
datasets, while the relevance of distant-different-skill records is only about 0.005. The otherfour DLKT models show similar results for these two groups. Furthermore, it is difficult todifferentiate between the relevance of recent-different-skill records or distant-same-skill
records, as these groups show similar relevance in some cases (e.g., DKT and CKT) anddiffering relevance in others (e.g., DKVMN in ASSISTment2009 and ASSISTment2015).
We also find that the same skill records are more sensitive to distance (i.e., time)compared to different skill records. Specifically, for all five DLKT models in both
ASSISTment2009 and ASSISTment2015, the average relevance difference between recentsame-skill records and distant-same-skill records is much larger than that between recent-8different skill records and distant-different-skill records. In contrast, distance seems to have asmall effect on the relevance of different skill records for model prediction. For instance, therelevance of recent-same-skill records for the DKT model in ASSISTment2015 is about
0.055, 0.045 higher than distant-same-skill records, while the difference between recentdifferent-skill records and distant-different-skill records is close to 0.002. Other DLKT models
show similar results.
Based on the findings discussed above, we conducted an additional experiment tofurther investigate the influence of skill and distance information on the decisions of DLKTmodels. Specifically, for each input sequence of length 15, we excluded the records thatwere distant from the prediction position and on non-target skills (i.e., distant-different-skill
records) and thus only kept the records that were close to the prediction position or on thetarget skill (i.e., recent-same-skill records, recent-different-skill records, and distant-sameskill records). The leftover records made up 73% of the raw records in the ASSISTment2009
dataset and 63% in the ASSISTment2015 dataset. We then compared the model prediction
performance with cases using all the raw records.
Figure 9 illustrates the experiment results: we see that for all five DLKT models,despite containing significantly fewer data points, the models with the leftover records have
similar prediction accuracy to the models with raw full sequences. For example, DKVMN
achieves 0.734 in accuracy with 1,464,555 records in ASSISTment2015. After excluding
about 37% of data, the accuracy of DKVMN remains at 0.734 with 932,799 records. The
experiment results partially validate the findings and the rules that how the DLKT models
utilize the skill and distance information to make the decision.

(a) ASSISTment2009(b) ASSISTment2015

Figure 9. The influence of skill and distance on the decisions of DLKT models.

5. Conclusion
In this work, we propose a simple and efficient xAI method to address the interpretability
issue of increasingly complex and diverse DLKT models. The method only requires a lower
complexity and can be used to explain a wide range of DLKT models. The experimentresults on five DLKT models in two datasets validate the effectiveness of the proposed
method. Further analysis of the explanations reveals that all five DLKT models use a similar
rule when making the decision: question-answer records that are close to the predictedquestion and on the same skill as the predicted question are found to be the most relevant
indicators, while records that are distant and on different skills are the least important. We
also have found that records of the same skill are more sensitive to changes in distancecompared to records of different skills. Additionally, it is also observed that using fewer but
relevant question-answer records to make predictions can achieve similar accuracy as using
full sequences, which supports the findings about the decision rules of DLKT models.
This work has significant impacts on practice. First, the rules obtained can be
integrated into intelligent tutoring systems that utilize DLKT models. Automatically identifying
learners' knowledge states and providing explanations can potentially increase their trust in
the system and help them adjust their learning behavior when receiving incorrect diagnosesof their knowledge states. Second, the findings about the decisions of DLKT models

9contribute to making these models more transparent and provide valuable insights forresearchers to design more interpretable KT models. It can be a promising direction for
future research to evaluate the impact of these explanations on education and consider theeffect of skill and temporal information when designing DLKT models.

Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (no.
62077006, 62177009) and in part by the Fundamental Research Funds for the Central
Universities.

References
Abdelrahman, G., & Wang, Q. (2019). Knowledge tracing with sequential key-value memory networks.
Proceedings of the 42nd International ACM SIGIR Conference on Research and Development inInformation Retrieval, 175–184.
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial
intelligence (xai). IEEE access, 6, 52138–52160. https://doi.org/10.1109/ACCESS.2018.2870052
Arras, L., Osman, A., Müller, K.-R., & Samek, W. (2019). Evaluating recurrent neural network
explanations. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting
Neural Networks for NLP, 113–126.
Corbett, A.T., & Anderson, J.R. (1994). Knowledge tracing: Modeling the acquisition of proceduralknowledge. User modeling and user-adapted interaction, 4(4), 253–278.
https://doi.org/10.1007/BF01099821
Cortez, P., & Embrechts, M. J. (2011). Opening black box data mining models using sensitivity
analysis. 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), 341–
348.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid
algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126.
https://doi.org/10.1037/xge0000033
Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the assessment challenge with an online
system that tutors as it assesses. User Modeling and User-Adapted Interaction, 19(3), 243–266.
https://doi.org/10.1007/s11257-009-9063-7
Ghosh, A., Heffernan, N., & Lan, A. S. (2020). Context-aware attentive knowledge tracing.
Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data
mining, 2330–2339.
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015). Peeking inside the black box: Visualizing
statistical learning with plots of individual conditional expectation. Journal of Computational andGraphical Statistics, 24(1), 44–65. https://doi.org/10.1080/10618600.2014.907095
Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the knowledge in a neural network. stat, 1050, 9.
Koedinger, K. R., & Aleven, V. (2016). An interview reflection on “intelligent tutoring goes to school inthe big city”. International Journal of Artificial Intelligence in Education, 26(1), 13–24.
Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2016). Visualizing and understanding neural models in nlp.
International Conference of the North American Chapter of the Association for Computational
Linguistics, 681–691.
Liu, Q., Huang, Z., Yin, Y., Chen, E., Xiong, H., Su, Y., & Hu, G. (2019). Ekt: Exerciseaware
knowledge tracing for student performance prediction. IEEE Transactions on Knowledge andData Engineering, 33(1), 100–115. https://doi.org/10.1109/TKDE.2019.2924374
Lu, Y., Wang, D., Chen, P., Meng, Q., & Yu, S. (2022). Interpreting deep learning models forknowledge tracing. International Journal of Artificial Intelligence in Education, 1–24.
https://doi.org/10.1007/s40593-022-00297-z
Lu, Y., Wang, D., Meng, Q., & Chen, P. (2020). Towards interpretable deep learning models forknowledge tracing. International Conference on Artificial Intelligence in Education, 185–190.
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advancesin neural information processing systems, 30.

10Montavon, G., Lapuschkin, S., Binder, A., Samek, W., & Müller, K.-R. (2017). Explaining nonlinear
classification decisions with deep taylor decomposition. Pattern recognition, 65, 211–222.
https://doi.org/10.1016/j.patcog.2016.11.008
Nakagawa, H., Iwasawa, Y., & Matsuo, Y. (2019). Graph-based knowledge tracing: Modeling student
proficiency using graph neural network. 2019 IEEE/WIC/ACM International Conference On WebIntelligence (WI), 156–163.
Pandey, S., & Karypis, G. (2019). A self-attentive model for knowledge tracing. 12th International
Conference on Educational Data Mining, EDM 2019, 384–389.
3HOiQHN53DSRXãHN-ěLKiN-6WDQLVODY9 1LåQDQ-  (OR-based learner modelingfor the adaptive practice of facts. User Modeling and User-Adapted Interaction, 27(1), 89–118.
https://doi.org/10.1007/s11257-016-9185-7
Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., & Sohl-Dickstein, J. (2015).
Deep knowledge tracing. Advances in Neural Information Processing Systems, 28.
Shen, S., Liu, Q., Chen, E., Wu, H., Huang, Z., Zhao, W., Su, Y., Ma, H., & Wang, S. (2020).
Convolutional knowledge tracing: Modeling individualization in student learning process.
Proceedings of the 43rd International ACM SIGIR Conference on Research and Development inInformation Retrieval, 1857–1860.
Shrikumar, A., Greenside, P., & Kundaje, A. (2017). Learning important features through propagating
activation differences. International conference on machine learning, 3145–3153.
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013). Deep inside convolutional networks: Visualising
image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
Sonkar, S., Lan, A. S., Waters, A. E., Grimaldi, P., & Baraniuk, R. G. (2020). Qdkt: Question-centric
deep knowledge tracing. Proceedings of the 13th International Conference on Educational Data
Mining, EDM 2020, Fully virtual conference, July 1013, 2020.
Su, Y., Liu, Q., Liu, Q., Huang, Z., Yin, Y., Chen, E., Ding, C., Wei, S., & Hu, G. (2018). Exerciseenhanced sequential modeling for student performance prediction. Proceedings of the AAAI
Conference on Artificial Intelligence, 32(1).
Tsai, Y. S., & Gasevic, D. (2017). Learning analytics in higher education---challenges and policies: areview of eight learning analytics policies. Proceedings of the seventh international learning
analytics & knowledge conference, 233-242.
Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening theblack box: Automated decisions and the gdpr. Harv. JL & Tech., 31, 841.
Wang, D., Lu, Y., Zhang, Z., & Chen, P. (2022). A generic interpreting method for knowledge tracing
models. International Conference on Artificial Intelligence in Education, 573–580.
Yang, S., Zhu, M., & Lu, X. (2020). Deep knowledge tracing with learning curves. arXiv preprint
arXiv:2008.01169.
Yang, Y., Shen, J., Qu, Y., Liu, Y., Wang, K., Zhu, Y., Zhang, W., & Yu, Y. (2020). Gikt: A graphbased interaction model for knowledge tracing. Joint European Conference on Machine Learning
and Knowledge Discovery in Databases, 299–315.
Yeung, C.-K. (2019). Deep-irt: Make deep learning based knowledge tracing explainable using item
response theory. arXiv preprint arXiv:1904.11738.
Zhang, J., Shi, X., King, I., & Yeung, D.-Y. (2017). Dynamic key-value memory networks forknowledge tracing. Proceedings of the 26th international conference on World Wide Web, 765–
774.

11

Kohama, H. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Recommending Learning Actions
Using Neural Networka

Hirokazu Kohama, Chubu University, Japanb
Yuki Ban, Chubu University, Japanc
Tsubasa Hirakawa, Chubu University, Japand
Takayoshi Yamashita, Chubu University, Japane
Hironobu Fujiyoshi, Chubu University, Japanf
Akitoshi Itai, Chubu University, Japang
Hiroyasu Usami, Chubu University, Japanatuna0724@mprg.cs.chubu.ac.jpbbanyuu@mprg.cs.chubu.ac.jpchirakawa@mprg.cs.chubu.ac.jpdtakayoshi@isc.chubu.ac.jpefujiyoshi@isc.chubu.ac.jpfitai@isc.chubu.ac.jpgusami@isc.chubu.ac.jp

Abstract: Many studies applying neural networks to the field of education havefocused on student performance prediction and explainability of their decisions. While
those studies introduced neural networks into educational settings, such networks
cannot directly support student learnings in place of teachers. Therefore, we present
a method that uses a general Transformer encoder to recommend appropriate
learning actions for improving student performance. By considering the attention
weight of a low-performing student to be close to that of a high-performing student,
our method recommends the learning materials and actions for learning the materials.
To evaluate the effectiveness of our method, we trained a deep neural network (DNN)
on a private dataset of student operations (e.g., NEXT, PREV, OPEN) on digitallearning materials obtained from a Japanese university. The number of operations
divided by each learning material and by type of operation are input to the DNN, andthe DNN outputs the student’s grade on 5-point scale. We applied our method withthis trained DNN to samples that successfully predicted grades, and the number ofoperations increased on the basis of the recommended learning materials andactions. By re-inputting modified sample into the DNN, we then observe how thestudent performance changes. The results of this simple experiment indicate that
more students improved their performance with both the material-based andoperation-based recommendations than with random recommendations. The
percentage of students whose grades improved tended to be larger for those with low
grades. Specifically, the improvement ratio for students with the two lowest grades
was over 90% by operation-based recommendation. This is consistent with ourintuition that low-performing students are more likely to improve.
Keywords: Student Performance Prediction, Explainable AI, Transformer

1. Introduction
Deep neural networks (DNNs) have been actively studied in various fields such as image
processing and natural language processing. They are expected to be used for education in
the current remote educational environment triggered by the COVID-19 pandemic (Adedoyin& Soykan, 2020). In an educational environment where there is physical distance between

12teachers and students, it is difficult for teachers to provide individual feedback, so support
from DNNs is necessary.
There have been many studies introducing DNNs into educational environments(Piech et al., 2015; Imran et al., 2019; Abdelrahman & Wang, 2019; Xing, & Du, 2019).
These studies showed that we can use DNNs to predict a student’s performance andpossibility of early school dropout. However, it is necessary to incorporate explainability andaccountability in educational DNNs design (Webb et al., 2021). There many studies
explained the basis for DNN predictions (Baranyi et al., 2020; Mu et al., 2020; VultureanuAlbişi & Bădică, 2021; Hasib et al., 2022; Swamy et al., 2022). However, these studies did
not focus on DNNs replacing teachers, thus could not directly reduce the burden onteachers.
We propose a method with which artificial intelligence (AI) instead of teachers
recommend appropriate learning actions (e.g., selection of learning materials, operations onlearning materials) for improving student performance (see Section 2). We developed thismethod to misidentify the output of AI predicting student grades as a good grade byminimally perturbing the input of a student with a low grade. The pipeline for integrating our
method into an educational environment is shown in Figure 1. We conducted an experimentto evaluate our method, and the results are presented in Section 3. Here, the predictedgrades are denoted by S (excellent) , A (good), B (satisfactory), C (pass), and F (fail).

(YHQW/RJV
Step 1:6WXGHQWV/HDUQZLWK'LJLWDO0DWHULDOV*UDGH3UHGLFWLRQ
6

hƐĞƌ
/

ĐƟŽŶ

Step 2:0RGHO&RQVWUXFWLRQ$%&

Step 3:
Analysis of Learning Actions

)

dŝŵĞ

ZĞĐŽŵŵĞŶĚĞĚ
ĐƟŽŶ

)HHGEDFN

Figure 1. Pipeline for Deploying DNN into an Educational Environment.

2. Proposed Method
Given a DNN and dataset including student learning actions on digital-learning materials, wedeveloped a method for recommending learning actions to improve grades in an input-data
dependent manner. We first introduce the criterion to identify learning actions.

2.1 Identification of Learning Actions that Significantly Affect Predictions
When an input vector ࢞ is perturbed ࢾ, the network loss ࣦ changes as:ሺͳሻȟࣦሺ࢞ሻ ൌ ȁࣦሺ࢞ሻ െ ࣦሺࢾ ٖ ࢞ሻȁǡ
where ٖ denotes the Hadamard product. Next, to compute the impact of each input,
Equation (1) is transformed with first order Taylor expansion by focusing on a single element‫ݔ‬௜ ‫࢞ א‬:ȟࣦሺ‫ݔ‬௜ ሻ ൌ ȁࣦሺ‫ݔ‬௜ ሻ െ ࣦሺߜ‫ݔ‬௜ ሻȁ߲ࣦሺߜ‫ݔ‬௜ െ ‫ݔ‬௜ ሻ െ ࣩሺԡ‫ݔ‬௜ ԡଶ ሻฬൌ ฬࣦሺ‫ݔ‬௜ ሻ െ ࣦሺ‫ݔ‬௜ ሻ െ
߲‫ݔ‬௜
߲ࣦ
ሺʹሻ
ሺߜ‫ݔ‬௜ െ ‫ݔ‬௜ ሻฬ ǡ؆ฬ߲‫ݔ‬௜where ࣩ denotes terms of higher order than the 2nd degree. We evaluate the input data on
material or operation in accordance with Equation (2).

13Materials: If the number of learning materials is ‫ ܯ‬and the type of operations (e.g., NEXT,
PREV, OPEN) is ܰ, the input data are denoted as ࢄ ‫ א‬Թெൈே . When ࢄ is input, the score ofmaterial ݉ is expressed by using Equation (2) as the following equation:ே߲ࣦܵ௠ ሺࢄሻ ൌ ෍ ቤ൫ߜ‫ݔ‬௠ǡ௡ െ ‫ݔ‬௠ǡ௡ ൯ቤ ǡ߲‫ݔ‬௠ǡ௡ሺ͵ሻ௡ୀଵwhere ࢞௠ǡ௡ represents the number of operations per type of operation in ݉. We select thematerial with the highest score and recommend it to students.
Operations: The score for each operation is expressed as:ெ߲ࣦܵ௡ ሺࢄሻ ൌ ෍ ቤ൫ߜ‫ݔ‬௠ǡ௡ െ ‫ݔ‬௠ǡ௡ ൯ቤ Ǥ߲‫ݔ‬௠ǡ௡ሺͶሻ௠ୀଵ

We identify the type of operation that is effective for learning by using Equation (4). As withthe material criterion, we recommend to students the operation with the highest score.

2.2 Optimization for Transformer
The theory introduced in Section 2.1 is applied independently of DNN structure. However,calculation of automatic differentiation using the loss between the correct label and output
incurs large computational cost. Therefore, we introduce an additional lightweight criterion
optimized for a Transformer (Vaswani et al., 2017).
A Transformer introduces an attention mechanism to determine the dependencies ofvarious ranges (e.g., shorter, or longer range) within a sequence. In this attention
mechanism, attention weight ࡭ in the first layer is derived using input ࢄ ‫ א‬Թெൈே as:ࡽࡷ்ሺͷሻ࡭ ൌ  ቆቇ ‫ א‬Թெൈெ ǡඥ݀௞‫ݏ‬Ǥ ‫ݐ‬Ǥ ࡽ ൌ ࢄࢃ௤ ǡ ࡷ ൌ ࢄࢃ௞ ǡ
where ݀௞ is a scaling coefficient, and ࢃ ‫ א‬Թௗൈெ denotes learnable parameters. By
considering the interrelationships between input sequences using Equation (5), we candetermine which inputs are contributing to the prediction.
Given an ideal attention weight ࡭Ԣ of a high-performing student, we can reduce thecomputational cost from Equation (3) and (4). The method is simply to replace the loss inEquations (3) and (4) by the squared loss between ࡭Ԣ and ࡭:ெெଶࣦ ൌ ෍ ෍൫ܽᇱ ௜ǡ௝ െ ܽ௜ǡ௝ ൯ ǡሺ͸ሻ௝ୀଵ ௜ୀଵ

ᇱ‫ݏ‬Ǥ ‫ݐ‬Ǥ ܽ௜ǡ௝‫࡭ א‬ᇱ ǡ ܽ௜ǡ௝ ‫࡭ א‬ǤThis optimization method makes it possible to recommend learning actions only throughprocessing by the first layer of Transformer encoders.

3. Experiment
We evaluated the recommendation of learning actions on a private dataset obtained in a real
educational environment. To verify that the ideal attention weight is obtained from thetraining data, we first investigate the attention weight per grade. We then investigated thevalidity of recommending learning actions.

3.1 Dataset
We use log data of real student learning actions and the grades collected at a Japanese
university. These log data were compiled for 21 types of operations (e.g., NEXT, PREV,
OPEN) executed on 12 types of materials in a single course. The number of training data14

was 114, and the number of test data was 51, because we used one year of such data forevaluation collected for a period of three years 2020, 2021 and 2022.
Figure 2 illustrates how the data were preprocessed. We treated the number ofoperations for each type of operation as a histogram for each learning material. Thus, theinput for a student was ࢄ ‫ א‬Թଵଶൈଶଵ . The values of these histograms were normalized by the
maximum number of operations calculated using the histograms of all students.

ʁNormalized Number of Operations
Material̏

OPEN

NEXT

ADD
MEMO

CLOSE

Material 2

OPEN

NEXT

ADD
MEMO

CLOSE

Material 12

OPEN

NEXT

ADD
MEMO

CLOSE

Figure 2. Preprocessing of The Data of One Student.

3.2 Experimental Setup
We construct a success-prediction model that can accurately classify the correctedlabel (final grade) ୱ for student  from learning actions, as shown in Figure 2. We
rely on a DNN architecture that is based on Transformer encoders, which can
combine high with reduced computational cost due to Equation (6). An overview ofthe model structure is shown in Figure 3. The features input to the model are linear
projections from student data such in Figure 2. The number of Transformer encoders‫ ܮ‬was 4. The model was trained using Adam (Kingma & Ba, 2015) for 200 epochswith an initial learning rate of 0.0001 and batch size of 32. The loss function uses
cross-entropy loss.

Linear Projection

Grade
S, A,…

Global Average PoolingTransformer Encoder

Embedding
Vectors

Figure 3. Overview of Model Structure for Grade Prediction.
3.3 Prediction of Student Grades
To coordinate the number of embedding dimensions ࢊ and heads ࢎ, we compare15their prediction accuracy with Transformer models of various scales.
Table 1 shows the top-1 accuracy of grades on the test dataset. The numberof embedding dimensions was investigated in the range from 64 to 256, and thehighest test accuracy was achieved at 128. The highest accuracy was then obtainedwith one head in all comparisons. Using the results of this experiment as a guide, we
recommend learning actions using a model with 128 embedding dimensions and onehead.
3.4 Attention Weight Differences Between Grades
Table 1. Test Accuracy of Grade Prediction݄݀dŽƉͲϭĐĐƵƌĂĐǇ΀й΁

ϲϰ
ϲϰ
ϭϮϴ
ϭϮϴ
Ϯϱϲ
Ϯϱϲ
Ϯϱϲ

ϭ
Ϯ
ϭ
Ϯ
ϭ
Ϯ
ϰ

ϯϵ͘Ϯϭ
ϯϵ͘Ϯϭ
ϱϮ͘ϵϰ
ϰϳ͘Ϭϲ
ϰϱ͘ϭϬ
ϰϱ͘ϭϬ
Ϯϳ͘ϭϮ

Attention is conducted multiple times in parallel, which results in higher performance
than just using a single head. However, our results were the exact opposite (seeSection 3.3). This means that mixing features between materials does not require
such complex patterns. We hypothesize that if the mixing between materials isrepresented by a simple pattern, there is an ideal attention weight, as described in
Section 2.1, common to high-performing students. To confirm this hypothesis, weinvestigated the attention weight by grade.
Figure 4 shows the attention weights of the first layer, calculated using thetraining dataset. In Figure 4, high values of attention weight are shown in light colors
and low values in dark colors. These attention weights are averaged per grade.
Since the number of teaching materials was 12, the size of attention weight was
12x12. We observed that the attention weights of the high-performing students, suchas those with grades S and A, had a biased attention to fewer materials. The
attention of the lowest-performing students are plotted across many learningmaterials. Therefore, we use the attention weight of students with grade S as an
ideal attention weight to reduce the computational cost of recommending learning
actions.

Figure 4. Average of Attention Weight per Grade Obtained
from First Layer of Transformer Encoder.

16

3.5 Recommending Learning Actions
Our method recommends appropriate materials and operations to a student using
the trained DNN.
Material Recommendation: Our method recommends the materials that students
should be learning to improve their grades. To investigate such improvement, we
increased the number of operations included in recommended materials by a factorof 1.5. We then re-input the sample with an increased number of operations into the
DNN to check for changes in performance. Figure 5 shows the ratio of performance
improvement for the sample that successfully predicted grade before modification
when our method was used to recommend materials to be learned. For comparison,
Figure 5 plots the improvement ratio with dashed lines when learning materials wererandomly recommended (the results plotted with solid lines are from our method).
The vertical axis is the ratio of predicted improved performance after re-input, andthe horizontal axis is the number of materials that were modified using our method or
randomly.
High rate of improvement was observed in the lowest-performing students
with grades C and F. Their performance tended to improve more in proportion to thenumber of materials modified. These results are consistent with the intuition thatlower-performing students are more likely to improve with a small amount oflearning, and that students who learn more improve their performance. Then, theimprovement ratio with our method is higher than that with random, which confirms
the effectiveness of our method.

:S
Improvement Ratio

:A
:B

:C
:F
: Ours: Random

Number of Materials
Figure 5. Ratio of Performance Improvement for Each Material Modified.
Operation Recommendation: We then identified the types of operations thatshould be used to improve student performance. By modifying the number ofoperations by a factor of 1.5 for our method, or for a randomly recommended
operation, we can compare the changes in grade predictions. Figure 6 shows theratio of performance improvement for the sample that successfully predicted grade
before modification when our method was used to recommend operations to belearned. The vertical axis is the percentage of predicted grades that improved after17

re-input, and the horizontal axis is the number of operations that were modified using
our method or randomly.

:S
Improvement Ratio

:A
:B
:C
:F
: Ours: Random

Number of Operations
Figure 6. Ratio of Performance Improvement for Each Operation Modified.
The operations recommended with our method were more accurate thanthose randomly recommended. From these results, the performance of students withlow grades improved significantly with our method.
4. Discussion
Our results showed that the suggested learning actions improved student
performance. However, our method is highly dependent on the quality and quantityof the learning data and may not work well when the number of students is small or
when there are not enough learning logs (our method may be very accurate in thepresence of a large number of learning logs). Figure 7 shows the top-5 mostfrequently recommended operations and their average scores. Many basic
operations (e.g., NEXT, PREV, OPEN) are recommended, and operations such as“ADD MEMO” are not included. While the basic operations are performed manytimes and are understandably effective for estimating student learning density, most
realistic teachers would recommend leaving notes and bookmarking importantpassages. Training on a larger dataset is necessary to achieve the same
performance as a teacher using a DNN. Therefore, a larger dataset would beneeded to construct a DNN comparable with a teacher. Since suggesting learning
actions using too small a dataset may invade the privacy of certain highestperforming students, one should be careful when deploying a pipeline such as thatshown in Figure 1.

18

Operations

Average Operation Scores
Figure 7. Top-5 Recommended Operations and Their Scores.

5. Conclusion
We showed that considering the impact of inputs on grade prediction can recommend thelearning actions needed to improve grades. Learning actions are recommended on a permaterial or per-operation basis, allowing the DNN to provide detailed feedback to thestudents. In particular, the operation recommendations confirmed a higher improved ratio ofgrades than the learning material recommendations. Recommending learning actions tostudents is a necessary component for neural networks to directly assist teachers andencourage the introduction of AI into the educational environment. In future work, we plan touse data augmentations suitable for event logs to recommend learning actions that are morein line with our intuition.

"DL OPX MFEHFNFOUTThis work was supported by JST CREST Grant Number JPMJCR22D1, Japan.

References
A. S. Imran, F. Dalipi, & Z. Kastrati. (2019). Predicting student dropout in a MOOC: An
evaluation of a deep neural network model. 5th International Conference on
Computing and Artificial Intelligence.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, & I.
Polosukhin. (2017). Attention Is All You Need. Advances in Neural Information
Processing Systems.
A. Vultureanu-Albişi & C. Bădică. (2021). Improving students’ performance by interpretable
explanations using ensemble tree-based approaches. IEEE 15th International
Symposium on Applied Computational Intelligence and Informatics.
C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, & J. Sohl-Dickstein.
(2015). Deep Knowledge Tracing. Advances in Neural Information Processing
Systems.
D. P. Kingma & J. Ba. (2015). Adam: A Method for Stochastic Optimization. 3rd International
Conference on Learning Representations.

19G. Abdelrahman & Q. Wang. (2019). Knowledge tracing with sequential key-value memory
networks. 42nd International ACM SIGIR Conference on Research and Development
in Information Retrieval.
K. M. Hasib, F. Rahman, R. Hasnat, & M. G. R. Alam. (2022). A machine learning andexplainable AI approach for predicting secondary school student performance. IEEE
12th Annual Computing and Communication Workshop and Conference.
M. Baranyi, M. Nagy, & R. Molontay. (2020). Interpretable deep learning for university
dropout prediction. 21st Annual Conference on Information Technology Education.
M. E. Webb, A. Fluck, J. Magenheim, J. Malyn-Smith, J. Waters, M. Deschênes, & J. Zagami.
(2021). Machine learning for human learners: opportunities, issues, tensions andthreats. Educational Technology Research and Development.
O. B. Adedoyin & E. Soykan. (2020). Covid-19 pandemic andonlinelearning: the challenges
and opportunities. InteractiveLearning Environments (2020).
T. Mu, A. Jetten, & E. Brunskill. (2020). Towards suggesting actionable interventions forwheel-spinning students. International Educational Data Mining Society.
V. Swamy, B. Radmehr, N. Krco, M. Marras, & T. Käser. (2022). Evaluating the Explainers:
Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs.
International Educational Data Mining Society.
W. Xing & D. Du. (2019). Dropout prediction in MOOCs: Using deep learning for
personalized intervention. Journal of Educational Computing Research.

20" !   !" " "   #" !
#" !"$  #" ! #"                                 
2*14)/)6)<-6-7-,=  '30/411><498914889?,=498</0<.;4-0<39@80@=0.38969240<<:;0,/=3;9>23
,:9:>6,=498208=-,<0/79/06482,8/<47>6,=498!&.9?0;48=0;,.=498<,7982
,>=98979>< ,208=< ,8/ =30 ,8,6B<4< 91 070;208= 9>=.970< 1;97 =30 -03,?49;<;0,.=498< ,8/ 48=0;,.=498< 91 =30<0 ,208=< A4<=482 <=>/40< ><482 !& ,47 =9
466><=;,=0 /B8,74. ,208= -03,?49; ,8/ 48=0;,.=498 @4=3 9=30; ,208=< ,8/ =30
08?4;98708=48=30.98=0A=91/411><498914889?,=498<9@0?0;=30B/989=0A=08/=30
><0 91 =30 <47>6,=9;< =9 =0,.3482 ,8/ 60,;8482 '34< <=>/B /0<.;4-0< ,8 !& ->46=
><482 "0= 929 =3,= ,669@< <=>/08=< =9 0A:69;0 =30 47:,.= 91 ?,;49>< ,208=
.3,;,.=0;4<=4.<-03,?49;<,8/48=0;,.=498<98,/9:=48280@4889?,=498<&=>/08=<.,8
7,84:>6,=0.0;=,48:,;,70=0;<48?96?482=30.3,;,.=0;4<=4.<91=30<0,208=<<>.3,<=304;60?06914889?,=498:;9:08<4=B<9.4,64816>08.0,8/.9880.=4?4=B=9<0039@=30<0
1,.=9;<4816>08.0=30,/9:=49891=304889?,=498'309>=.970910,.3;>84<69220/
,8,6BC0/,8/:;0<08=0/=9=30<=>/08=<,<70,84821>6100/-,.5,8/<>220<=498<19;
<>::60708=,;B 60,;8482 1;97 ,8 !& 48 :;0:,;,=498 19; <>..00/482 <47>6,=498
4=0;,=498<'30<47>6,=9;:;0<0;?0<,208=,>=9897B,8/,/,:=,-464=B@3460,669@482=30
<=>/08=<=9:6,B@4=379/06:,;,70=0;<<=30<47>6,=9;=;,8<4=498<=979;007:4;4.,6/,=,19;;>60<29?0;8482=30-03,?49;<91,208=<1>=>;0?0;<498<91=30<47>6,=4987,B
48.9;:9;,=0,//4=498,6><0;48=0;1,.0,8/-,<0/<47>6,=9;060708=<  ,208=-,<0/ 79/06482 ,208=-,<0/ 79/06 <47>6,=498 .97:>=0;
<47>6,=49808=;0:;080>;<34:0/>.,=498/411><498914889?,=498< 
'0-,1..=;1767.1667>)<176;,-;+:1*-;07?6-?1,-);)6,<-+06747/1-;;8:-),<0:7=/0)
878=4)<176%7/-:;  6<0-+76<-@<7.)+7584-@;7+1)4;A;<-5<0-,1..=;1767.
1667>)<176;<0-7:A:-.4-+<;),78<176<1516/)6,,-+1;176;5),-*A+76;=5-:/:7=8;
+4);;1.1-,*);-,76<0-.7447?16/),78<176<1516/+4);;1.1+)<176<0-5- 1667>)<7:; -):4A
),78<-:; -):4A5)27:1<A 4)<-:5)27:1<A)6,4)//):,;%7/-:;  1..=;1767.
1667>)<176;1;:-4->)6<16-6<:-8:-6-=:;018-,=+)<1767?->-:<:),1<176)4<-)+016/
)88:7)+0-;<7-6<:-8:-6-=:;0180)>-*--6*);-,76*=;16-;;5)6)/-5-6<-,=+)<176
?01+01;16.4-@1*4-)6,16-..-+<1>-)A744-)144A
758=<-:;15=4)<176%7*-:<;
0-48;,->-478<-)+0)6,<-;<-6<:-8:-6-=:;018<0-7:A)6,0-48;-6<:-8:-6-=:;
=6,-:;<)6,8:-,1+<)6,-@84)16<0-87<-6<1)4+76;-9=-6+-;7.<0-1:,-+1;176;/-6<*);-,
57,-416/)6,;15=4)<176 &).7:57.+758=<-:;15=4)<176+7>-:;16<-:)+<176;)576/
16,-8-6,-6<)/-6<; )+)4!7:<0)6,<0-)6)4A;1;7.-5-:/-6<7=<+75-;.:75<0-*-0)>17:;)6,16<-:)+<176;7.<0-;-)/-6<;):+1)
'01;8)8-:1;8):<7.)4):/-:;-<
7.;<=,1-;=;16/)/-6<*);-,57,-416/)6,;15=4)<176.7:4)/)6
'0-.7+=;1;76
,1..=;1767.1667>)<176;););15=4)<176;+-6):17<7=6,-:;<)6,<0-5-+0)61;5;<0)<,:1>-<0-),78<176)6,;8:-),7.6-?1,-);)6,<-+06747/1-;'0-;<=,A)4;7),,:-;;-;<0-
87<-6<1)4.7:16<-/:)<16/):<1.1+1)416<-441/-6+-)<<0-)/-6<4->-47?;07=4,<0-;15=4)<7:
)447?;<=,-6<;<784)A):7=6,?1<0,1..-:-6<+0):)+<-:1;<1+;7.)/-6<;158)+<16/<0-),78<176
7.1667>)<176;16<0-;15=4)<176?014-8:-;-:>16/<0-)/-6<;C)=<7675A<7)447?.7:-5-:/-6<

21

7=<+75-;%$ 7?+)6)=<7675A)6,),)8<)*141<A7.)/-6<;*-1584-5-6<-,?014-/1>16/;<=,-6<;<0-.:--,75<7-@8-:15-6<?1<0>):17=;57,-48):)5-<-:;%$ (0)<;7.<?):-
):+01<-+<=:-1;6--,-,.7:<0- ;15=4)<176<70)>-)584->)41,1<A.7:4-):616/<0-+76+-8<;7.,1..=;1767.1667>)<176;%$ 6?0)<?)A;+)6<0- ;15=4)<176;=887:<
:<1.1+1)46<-441/-6+-)<<0-)/-6<4->-4  
(1<0<0-,1..=;1767.1667>)<176;);<0-;15=4)<176;+-6):17;<=,-6<;5)A-@8-:15-6<?1<0>):17=;<-+06747/A;<:)<-/1-;.:758=:;=16/<-+06747/A4-),-:;018);)61667>)<7:<7*-0)>16/);)6-):4A),78<-:7:).);<.7447?-:'0-A+)6-@8-:1-6+-<0-:-;8-+<1>-),>)6<)/-;)6,,:)?*)+3;7.<0-;-;<:)<-/1-;&<=55-:1-;416/
&8-+1.1+;3144;
158):<-,*A<0-;15=4)<176-6>1:765-6<:-4)<-,<7,1..=;1767.1667>)<176;);16;81:-,*A
&<=55-:1-;416/):-5):3-<-6<:A/47*)48:7,=+<4)=6+0;<:)<-/1-;8:1+16/;<:)<-/1-;8:7,=+<41.-+A+4-)6,87:<.74175)6)/-5-6<)6,<-+06747/A;<:)<-/A!-<7/7
'1;=-(14-6;3A
1;)678-6;7=:+-8:7/:)5516/4)6/=)/-)6,57,-416/
-6>1:765-6<;8-+1.1+)44A,-;1/6-,.7:)/-6<*);-,57,-416/(014-;->-:)4;<=,1-;0)>-
)4:-),A.-)<=:-,,1..=;1767.1667>)<176;1584-5-6<-,16 &676-0)>-*--6-@841+1<4A
<1-,<7<-)+016/7:4-):616/-6<:-8:-6-=:;018 &1;):/=-,<7*--1<0-:);=*.1-4,
1;<:1*=<-,"C&=441>)6)34)A7:)6)4<-:6)<1>-7.<0)<<)3-;),>)6<)/-7.
<0-:-;=4<16/-5-:/-6+-7.16<-:)+<176;)6,<0-;<7+0);<1+:-8:-;-6<)<1767.16<-441/-6+-
=/-: )+016-4-):616/ )6,<-+0619=-;5)A)4;7-60)6+-<0-*-0)>17:)6,
+)8)*141<1-;7.)/-6<;);8:787;-,*A%)6,




 '0-8):)5-<-:;7.<0-)/-6<57,-4:-8:-;-6<16/,1..-:-6<;-6;1<1>1<1-;7:16,-@-;.-)<=:-;
8:1+-8:757<176)6,;7+1)416.4=-6+16/1667>)<176),78<176):-*);-,76<0-?7:37.
&+0:)55-<)4
'0-:=4-;)6,.7:5=4);=;-,16<01;;<=,A):-;15841.1-,>-:;176;7.
<07;-8:-;+:1*-,*A);;  
'0-;-16+4=,-<0-+76;=5-:),78<176<0:-;074,
').=6+<1767..-)<=:-;8:1+-8:757<176;7+1)416.4=-6+--<+'0-8:7,=+<8:1+-<-6,;
<7,:787>-:<15-'0--@8-+<-,&+=:>-:-4)<-,<7),)8<176)6,<0-,15161;016/8:1+-;;0)44*-:-.4-+<-,/:)801+)44A)6,)615)<-,7>-:<15-'0-764A.7:57.:-4)<176;0181;?0-6)6)/-6<1;80A;1+)44A6-):)67<0-:6<-/:)<176?1<0)-):616/ )6)/-5-6<&A;<-5 &+)6*-,76-<0:7=/0.14-<:)6;.-:7.!-<7/747/;7:#+)447=<;.7:+76<-6<.:75 &?0-:-
-)+0;15=4)<176:=61;47//-,)6)4AB-,)6,8:-;-6<-,<7<0-;<=,-6<;);5-)616/.=4.--,*)+31/=:- ;07?;<0-;15=4)<7:C;;<)<-*-.7:-)6,).<-:<0-:=6'0-4)A7=<7.
+76;=5-:)/-6<;:-;-5*4-;)*-44+=:>-+0):<76<0-:1/0<;07?;)6&+=:>-.7:<0-),)8<1767.<0-1667>)<176;?1<0)44/:--616,1+)<16/.=44;)<=:)<17667<0-:+0):<;07?;
:->-6=-8-:8-:17,16,1+)<16/<)8-:16/7..:->-6=-;);<0-1667>)<1765)<=:-;)6,
->-6<=)44A,-+416-;  &15=4)<7:;<)<-*-.7:-)6,).<-:<0-:=6

22
'0- 7*14-;?1<+0)447?;)/-6<;<757>-):7=6,<0-;+:--6<7)447?-6+7=6<-:;?1<0
7<0-:)/-6<;)6,<716.4=-6+-<0-1:6-1/0*7:;&<=,-6<;+)65)618=4)<-)/-6<;<77*;-:>-
07?<0-;-.)+<7:;16.4=-6+-<0-),78<1767.<0-1667>)<176/-6<;?144*-0)>-)=<76757=;4A*);-,76<0-8:7/:)55-,:=4-;*-0)>17:;)6,8):)5-<-:;;-<*A<0-;<=,-6<;   
=<=:->-:;176;7.<01;;15=4)<176?144)447?,1:-+<:-4)<176;018;)6,?14467<*-+76;<:)16-,<7
8:7@151<A);<0-*);1;.7:*-16/)*4-<716.4=-6+-7<0-:;(014-<01;8)8-:+7>-:;764A:=4-;
*);-,5-<07,;.7:)/-6<*-0)>17:5)+016-4-):616/ )6,<-+0619=-;16<=:6?144
-60)6+-<0-*-0)>17:)6,+)8)*141<1-;7.)/-6<;%)6,
16.=<=:->-:;176;7.<01;
;15=4)<7:;<0-;15=4)<7:<:)6;1<176;<757:--581:1+)4,)<).7::=4-;/7>-:616/<0-*-0)>17:;7.)/-6<;.=<=:->-:;176;7.<0-;15=4)<1765)A16+7:87:)<-),,1<176)4=;-:
16<-:.)+-)6,*);-,;15=4)<7:-4-5-6<;6),,1<176:)<0-:<0)6:-4A16/76.7:5=4);.7:
+76;=5-:)6,)/-6<*-0)>17:-@<-:6)4,)<).-,16<7)6 57,-4:-41-,76*A)/-6<;?144
)447?57:-),)8<1>-*-0)>17:16<0-;15=4)<7:67<0-:;07:<+7516/7.<01;;15=4)<1761;<0)<<0-*-0)>17:7.+758-<1<7:;1;.1@-,)6,4)+316/(1<0<0-16<:7,=+<1767. )6,
<-+0619=-;),)8<1>-+758-<1<7:*-0)>17:?1445)3-<0-;15=4)<7:57:-16<-:-;<16/<7<0-<-+06747/A-6<:-8:-6-=:;018;<=,-6<;);<0-A?144;--+76;=5-:)6,8:7,=+-:+758-<1<7:*-0)>17:'0-:-)4;71;)67887:<=61<A<7<)3-),>)6<)/-7./-6-:)<1>-)6,4):/-
4)6/=)/-57,-4; ;<7-6)*4-)+76>-:;)<176)4-4-5-6<<7<0--,=+)<176)416<-:.)+-,<<  !    '30 ;06,=498<34: -0=@008 /411><498 ;,=0< 0A:0;408.0 .>;?0< ,8/ /07,8/
06,<=4.4=40<19;.98<>70;/>;,-60=0.3896924.,64889?,=498<$)&#!$ )' #''& D&,<<  !   80@ :;9/>.= 2;9@=3 19; 79/06 .98<>70; />;,-60< #"#(  # +<>::60708=  D  
,B9660   ,466B   ;97 .;,1= =9 <.408.0 '0,.3482 79/06< ,8/ 60,;8482 :;9.0<<0< 48
08=;0:;080>;<34: 0/>.,=498 $)&#! $ )&$%# #)'(& ! & # #  D 
3==:</949;2     ,;.4, %  (<0< 91 ,208=-,<0/ 79/06482 48 4889?,=49880@ :;9/>.= /0?069:708= ;0<0,;.3$)&#!$&$)(##$*( $##"#(  D 
6,2,8
'30/0<428,8/><091,208=-,<0/79/06482.97:>=0;<47>6,=49819;=0,.3482
=0.389692B08=;0:;080>;<34:&$ #'$((#(&#( $#! $#&#$# $"%)(&'#)( $# D >20;    &(   ! #(!! #  (&)()&' # '(&( ' $& $"%!+ %&$!" '$!* #
$0,;<980/>.,=498
!,.,6  !  "9;=3 !   208=-,<0/ 79/06482 ,8/ <47>6,=498 !& 0A,7:60<&$ #'
 #(& ")!( $#$#&#D
3==:</949;2 @<.    #E&>664?,8,56,B!
208=-,<0/79/06<,8/48/4?4/>,64<7<=30@9;6/,208=-,<0/
#* &$#"#(#!## # 
D%,8/)
!,.348060,;8482700=<,208=-,<0/79/06482)30889==929=9,-,; $#&#$#$ ! #(' ')!('#&$'%('%9-0;=<&
'>=9;4,698=30<47>6,=4989130,6=3.,;0<B<=07<&$ #' #(& ")!( $#$#&#
D
3==:</949;2 @<.  %920;<!   )' $#$ ##$*( $#'
&.3;,77!';,489;&3,850;!>!*8,208=-,<0//411><49879/06@4=3
.98<>70;,8/-;,8/,208=<  ' $#)%%$&(,'("' D&=>770;   40<6482   8 ,208=-,<0/ 7,;50= <47>6,=498 19; 08;4.3482 4889?,=498
7,8,20708=0/>.,=498 #(&!)&$%#$)&#!$%&( $#''& D  
'4<>0&)4608<5B("0=6929<47:6008?4;98708=19;79/06482.97:60A4=B#(&#( $#!
$#&#$# $"%!+,'("' D 23" !   !" " "   #" !
#" !"$  #" ! #"          
  /)01),"0(" 4", 1%3%!
)."$$,+"'& )."$$,+"'& &"-)*"+/'&&*/$-&" )."$$,+"'& 
3B9CD9>1 =195B=854E31D9?> 3?=  <=E:3253(@/17<53AB7;/B3A/ABC23<BLA9<=E:3253=</A3B=4A97::A/<2>@3271BAE63B63@B63ABC23<BE7::/<AE3@1=@@31B:G7457D3</?C3AB7=<:7<932B=AC0A3BA
=4AC16A97::A+31=<2C1B/<7<23>B6/</:GA7A=<B6303ABE/GAB=/>>:GB631=5<7B7D3
A173<13 >@7<17>:3A AC16 /A ;3;=@G 231/G /<2 A>/17<5 E7B67< 9<=E:3253 B@/17<5>@=>=A7<5 <3E /:5=@7B6;A !3;31 /<2 !3;31 '>/17<5 B= 2= A= +3 3F>:=@3
27443@3<B;3B6=2A=4;=23:7<5B63@/B3/<2E3756B=4231/GE7B6/<2E7B6=CBA>/17<5/<2/</:GH3B637@7;>/1B=<>@3271B7<5>3@4=@;/<137<@3/:E=@:22/B/*/@7/B7=<A=4B63
;=23:/@31=;>/@3203BE33<3/16=B63@/AE3::/A/5/7<AB=B63@3F7AB7<5/:5=@7B6;A <=E:3253B@/17<5;3;=@G231/GA>/17<534431B:3/@<7<5AGAB3;A$" %$ %>?G<5475DB139>71DD5=@DCD?5CD9=1D5CDE45>D=1CD5BI?>1C5D?6C;9<<C6B?=D859B
@5B6?B=1>35 -85?ED@EDC?6;>?G<5475DB139>781F5C5F5B1<EC5C 9>3<E49>7EC59>2581F9?B1<
1>4C5<6 B57E<1D54<51B>9>7=?45<C B5@?BDCD?9>CDBE3D?BCG9D813D9?>12<59>C978DC 1>44B9F9>7
=1CD5BI<51B>9>7G9D89>141@D9F5<51B>9>7CICD5=C*5<K>5;   <D8?E78D85=1:?B9DI?6
D85 =?CD B535>D G?B; 9> ;>?G<5475 DB139>7 81C 9>F5CD971D54 B569>5=5>DC D? 455@ <51B>9>7
1<7?B9D8=C*95385D1<   D85B581C1<C?255>9>D5B5CD9>45F5<?@9>7;>?G<5475DB139>7
1<7?B9D8=C D81D <5F5B175 69>49>7C 6B?= 3?7>9D9F5 C395>35 8?669> 5D 1<    *1F<9; 5D 1<   1>4 CDE4I9>7 8?G 2B?14<I 1@@<9312<5 D85I 1B5 ,38=E3;5B  '9D385<<     $> D89C
@1@5B  G5 9>F5CD971D5 D85 9=@<5=5>D1D9?> ?6 3?7>9D9F5 C395>35 @B9>39@<5C G9D89> ;>?G<5475
DB139>79>45D19< C@5396931<<I6?3EC9>7?>=5=?BI4531I1>4D85C@139>756653DG85>1@@<954
D?F1B91>DC?6C?=53?=@?>5>DC?6&?79CD93%>?G<5475-B139>7&%-*1F<9;5D1<  G9D8
1>5=@81C9C?>D853?=@?>5>DC6B?=*5B6?B=1>35!13D?B>1<IC9C*!*1F<9;5D1<  
1>4+535>D *5B6?B=1>35!13D?B>1<IC9C+ *!"1<I1B4D"?<49>   /538?C5D?
CDE4I=5=?BI4531I1>4C@139>7G9D89>D89C6B1=5G?B;2531EC5?6*!PC9>D5B@B5D129<9DI1>4
9DCCE@@?BD?6=E<D9 C;9<<9D5=C -?CDE4ID85C53?>35@DC G59>DB?4E351>5G1<7?B9D8=D81D
9>3?B@?B1D5C=5=?BI4531I9>D?D85;>?G<54755CD9=1D9?>1>49>F5CD971D58?GD85C@139>7
56653D 1>4 49665B5>D G1IC ?6 7?F5B>9>7 D85 9>3B51C5 9> =5=?BI 4531I 9>6<E5>35 =?45<9>7
CDE45>D@5B6?B=1>35 /53?>C945BDG?=5D8?4C6?B=?45<9>7=5=?BI4531I?>5D81DEC5C
D85 @B13D935 ?B45B 1C 9> "1<I1B4D  "?<49>     1>4 1 C53?>4 D81D EC5C 1 D9=5 G9>4?G
1@@B?138  $>3?B@?B1D9>7 D85C5 3?>35@DC 1<<?GC 6?B D85 49665B5>D91D9?> 25DG55> 1 CDE45>DPC
3EBB5>D;>?G<5475F5BCEC@B5F9?EC3?=@B585>C9?>D81D=1I81F5255><?CD?F5BD9=5 $>,53D9?>G549C3ECCD85=5=?BI4531I1>4C@139>756653DD85?B95C 1>445C3B925
D85 G5<< ;>?G> ;>?G<5475 DB139>7 1<7?B9D8=  *5B6?B=1>35 !13D?B >1<IC9C *! *1F<9; 5D
1<   ,53D9?>3?>D19>C+5<1D54/?B;9>F5CD971D9>79CCE5CCEBB?E>49>7=5=?BIG9D89>
;>?G<5475 DB139>7  $> ,53D9?>  G5 9>DB?4E35 1 >5G 1<7?B9D8=  '5=53  1>4 1 F1B91>D '5=53 ,@139>7  D81D 9>3?B@?B1D5C D85 C@139>7 56653D  ,53D9?>  @B5C5>DC D85 B51< G?B<4
41D1C5DEC546?BD855H@5B9=5>DC >1<IC9C1>45H@5B9=5>D1<B5CE<DC1B5@B5C5>D549>,53D9?>
 !9>1<<I 3?>3<EC9?>C1>469>1<B5=1B;C1B549C3ECC549>,53D9?> 24 &" "( $ $"*CI38?<?7931< 56653DC 9>6<E5>35 CDE45>D <51B>9>7 9> 3<1CCB??=C 1>4 F9BDE1< <51B>9>7
5>F9B?>=5>DC )>5CE38@B9>39@<59CD854531ID85?BI 1@B9>39@<5?66?B75DD9>7G8938CD1D5C
D81D=5=?BI6145CG9D8D85@1CC175?6D9=5 .><5CCB59>6?B3542IB5@5D9D9?> D859>6?B=1D9?>
G5<51B>9C6?B7?DD5>?F5BD9=5 /9D8?ED9>3?B@?B1D9>74531I9>D?;>?G<54755CD9=1D9?> 9D9C
496693E<D D? 133?E>D 6?B C9DE1D9?>C G85B5 1 CDE45>D 6?B75DC  '5=?BI 4531I 81C 255> G945<I
CDE49541>4=?45<542I3?7>9D9F5C395>D9CDC1>4<51B>9>7C395>D9CDC5 7 '?J5B5D1<  *1F<9;>45BC?>   !?B5H1=@<5 9>'' =5=?BI4531I9C9>3?B@?B1D54D8B?E781
@?G5B6E>3D9?>9>G893851389D5= C@539693=5=?BIDB1354531IC5H@?>5>D91<<I'?J5B5D1<   -89C=?45<G1CD85>EC54D?45D5B=9>5?@D9=1<@B13D935CD?I95<4D8589785CDB5D5>D9?>
?6 =1D5B91< 9> 1 3<1CCB??=  ,9=9<1B<I  1 CDE4I G1C 3?>4E3D54 D? =51CEB5 1>4 3?=@1B5 D85
B5<51B>9>7 ?6 6?B7?DD5> =1D5B91< 2I D8B55 3?=@ED1D9?>1< =?45<C  1<< ?6 G8938 9>3?B@?B1D5 1
3?=@?>5>D?64531I?F5BD9=59>9DC@B5493D9?>5AE1D9?>/1<C85D1<   -85B5 9C 9>3B51C9>7 5F945>35 D81D ;>?G<5475 B5D5>D9?> 31> 25 5>81>354 D8B?E78
=5D8?4C CE38 1C 49CDB92ED54 <51B>9>7  G8938 C5AE5>35C <51B>9>7 CE38 D81D 3?>35@DC 1B5
@B13D9354 9> 1 49CDB92ED54 C3854E<5 ?F5B D9=5 B1D85B D81> G9D8 =1CC54 C3854E<5C G85B5
@B13D9351DD5=@DCB5<1D54D?D85C1=5C5D?6C;9<<C?33EB9>AE93;CE335CC9?> +5C51B389>D89C
1B5181C45=?>CDB1D54D81D2?D8D9=55<1@C5425DG55>@B13D935C?6D85C1=5=1D5B91<1>4
D9=55<1@C5425DG55>69>1<CDE4I5@9C?451>41>5H1=16653D69>1< D5CDB5D5>D9?>/9C5851BD
5D1<   /?B;81C1DD5=@D54D?EC5=?45<CD?45D5B=9>5?@D9=1<C@139>7 CE381C'?J5B
5D1<  *1F<9;>45BC?>   5138?6G89389>DB?4E3541=?45<D81D31>@B5493DD85
9>6<E5>35 ?6 C@539693 CDE4I C3854E<5C ?> B5D5>D9?> 6?B C@539693 9D5=C  %>?G<5475 DB139>7
1<7?B9D8=C9>3?B@?B1D9>7C@139>756653DC9>D?<51B>5B=?45<C5 7 /1<C85D1<  *1F<9;5D
1<  81F5255>EC546?B1>9>3B51C9>7>E=25B?61@@<931D9?>C 9>3<E49>7D85141@D9F5
C5AE5>39>7?6@B13D935 7<9>7D?>*1F<9;    $>D?D1< D85B59C3?>C945B12<55F945>35D81D
C@139>79C9=@?BD1>D6?B<?>7 D5B=B5D5>D9?>?6;>?G<5475 /9D89>D89C@1@5B G59>F5CD971D5
49665B5>DF1B91D9?>C?>8?GD?B5@B5C5>DD85C@139>756653D9>;>?G<54755CD9=1D9?> 1>1<IJ9>7
8?GD85C5=?45<C@5B6?B=6?B49665B5>DB5@B5C5>D1D9?>C?6=5=?BI4531I D8B?E781<D5B9>7D85
B1D51>4CDB5>7D8?64531I?F5BD9=5   "  $!!
*5B6?B=1>35 !13D?B >1<IC9C *! 9C 1 ;>?G<5475 DB139>7 1<7?B9D8= *1F<9; 5D 1<  D81D31>25EC54G9D8=E<D9 C;9<<9D5=C 9>3?>DB1CDD?C?=5?69DC5HD5>C9?>CG8938?><I
G?B;6?BC9>7<5 C;9<<9D5=C"1<I1B4D"?<49>   *!81C255>CE335CC6E<1D@B5493D9>7
2?D8;>?G<5475G9D89>D85<51B>9>7CICD5=1>4<1D5>D;>?G<547531BB954?EDC945D85CICD5=
,3BE77C5D1<    )EB@B?@?C541<7?B9D8=1>45H@5B9=5>DCEC5*!1C121C5<9>5=?45< -85?B979>1<*!=?45<@B5493DCD85@5B6?B=1>35?61CDE45>D?>179F5>9D5= @B?2<5= 
1D179F5>D9=5 $D4?5CD89C2IEC9>7D85CDE45>DPC@1CD>E=25B?6CE335CC5C =E<D9@<9542I1
G5978D  69D D? 5138 ?6 D85 9D5=PC C;9<<C 1 CDE45>DPC @1CD >E=25B ?6 619<EB5C  =E<D9@<954 2I 1
G5978D 69DD?5138?6D859D5=PCC;9<<C1G5978D G8938B5@B5C5>DCD85496693E<DI?61>9D5= 
5@5>49>7?>D85F1B91>D?6*!  9C59D85B1@@<95413B?CC1<<3?>D5HDC 13B?CC1<<9D5=C<9>;54
D?D853EBB5>DC;9<<D85=?CD3?==?>1@@B?1381>4G81DG5G9<<EC585B5 13B?CC1<<9D5=C
?6 D85 C1=5 N9D5= DI@5O  ?B 6?B 9>49F94E1< 9D5=C  -85C5 651DEB5C 1B5 9>@EDD54 9>D? 1 <?79CD93
6E>3D9?>D??2D19>1@B5493D9?> KL G893879F5CD85@B?2129<9DI?6CE335CC6?B179F5>CDE45>D?>179F5>6EDEB59D5= -85*!6?B=E<19C79F5>25<?G
'K$ %   , "L E J K  C  , C  " L *K'L E
 C ! 

G85B5$B5@B5C5>DC1CDE45>D 1B5D85;>?G<54753?=@?>5>DC9 5 C;9<<C<9>;54D?D859D5= %B5@B5C5>DC1C;9<< *1B1=5D5BC    1>4 1B5D85<51B>54@1B1=5D5BC6?BC;9<<% -85*!=?45<9>9DC?B979>1<6?B=4?5C>?D9>3?B@?B1D5D85>?D9?>?6=5=?BI4531I <<@B5F9?EC@B13D935C1B579F5>D85C1=5G5978D B571B4<5CC?6D85D9=5?B?B45B9>G8938D85I

25

D??; @<135  C 49C3ECC54 12?F5  C?=5 @B5F9?EC CDE495C 81F5 9>3?B@?B1D54 D85 >?D9?> ?6
=5=?BI4531I9>D?D859B;>?G<5475DB139>7=?45<C ?9>7C?144B5CC5CD85@85>?=5>?>D81D
=5=?BI6145CG9D8D85@1CC175?6D9=5 $7>?B9>74531I=1I25D5=@?B1B9<IC165G85>@B13D935
C5CC9?>C1B5=1CC541C9>C?=59>D5<<975>DDED?B9>7CICD5=C2EDG9<<<514D?<5CC133EB1D5
9>65B5>35G85>D85CDE45>DPCG?B;?>1C;9<<9CC@B514?ED?F5BD9=5 $ " F1B95DI ?6 ;>?G<5475 DB139>7 6B1=5G?B;C 1>4 =?45<C 81F5 255> @B?@?C54 1>4
CDE4954  -89C 9>3<E45C =5D8?4C 21C54 ?> #9445> '1B;?F '?45<C CE38 1C 1I5C91>
%>?G<5475 -B139>7 %- ?B25DD  >45BC?>    G8938 =?45<C 1 CDE45>DPC <1D5>D;>?G<5475?6;>?G<54753?=@?>5>DC1C1C5D?629>1BIF1B912<5CG8938B5@B5C5>D=1CD5BI?B<13;?6=1CD5BI?651383?>35@D 1>4>5EB1<>5DG?B;=?45<CCE381C55@%>?G<5475-B139>7*95385D1<   G8938EC5C+((CB53EBB5>D>5EB1<>5DG?B;CD?<51B>3?>35@D@1DD5B>C
EC9>7 <?>7 C8?BD D5B= =5=?BI &,-' G9D8?ED 8E=1> 1>>?D1D9?>C  -85B5 1B5 1<C? B535>D
566?BDCD?3?=29>5;>?G<5475DB139>71>4$D5=+5C@?>C5-85?BI)541C19  %81:18
5D1<  2G9D814531I56653D =?45<541C5<1@C54D9=51>416?B75DD9>7@1B1=5D5B $> D85 <1CD 65G I51BC  5HD5>C9?>C D? D85C5 31D57?B95C ?6 =?45<C 81F5 1DD5=@D54 D?
9>3<E459D5=B535>3I 4531I +5C51B385BC45=?>CDB1D54D81D2I1449>716?B75DD9>7@1B1=5D5B
D?%- D851<7?B9D8=9C=?B5C5>C9D9F5D?D8556653D?69>D5BC@5BC54DB91<C%81:185D1<   
>?D85BF1B91>D?6%-31<<54'E<D9CD1D5%-B71G1<5D1<   9>3B5=5>D1<<I9>3B51C5C
D85 G5978D ?6 >5G5B 1DD5=@DC  $> DD5>D9F5 %>?G<5475 -B139>7 "8?C8 5D 1<     
B5C51B385BC9=@<5=5>D541=?>?D?>931DD5>D9?>=5381>9C=D81DEC5C1>5H@?>5>D91<4531I
3EBF5D?4?G>G5978D@1CDAE5CD9?>C ,#8?669>5D1<  =?45<542?D8<51B>9>71>4
6?B75DD9>7 3EBF5C  EC9>7 613D?B9J1D9?> =1389>5C D? 81>4<5 =E<D9@<5 C;9<<C D1779>7  &51B>9>7
*B?35CC 3?>C9CD5>D%>?G<5475-B139>7&*%-,85>5D1<  =?45<CD85CDE45>D<51B>9>7
@B?35CC 1C 1 C5D ?6 DE@<5C G8938 9>3<E45C D85 D9=5 C5B95C 9>6?B=1D9?> ?6 D85 1CC97>=5>DC 
D85B52I5=25449>72?D8D851>CG5BD9=51CG5<<1CD859>D5BF1<25DG55>13D9F9D95C *B5F9?ECB5C51B3881C9>F5CD971D54F1B91D9?>C?6*!D81D9>3?B@?B1D54531I9>D?D859B
=1CD5BI@B5493D9?>C )>5CE38=?45< *! 531I"?>75D1<   D??;@B13D935?B45B9>D?
133?E>DEC9>714531I613D?BR RB19C54D?1@?G5BB5@B5C5>D9>7D8549CD1>359>
@B13D935>E=25B 1>4=E<D9@<954D?D853?E>DC?6CE335CC5C1>43?E>DC?6619<EB5C >?D85B
=?496931D9?> ?6 *!  +535>D *! + *!  9>3?B@?B1D54 =5=?BI 4531I 9>D? D85 =?45<PC@5B6?B=1>35 @B5493D9?> 9> D85 6?B= ?6 1 G5978D54 @B?@?BD9?> ?6 CE335CC  G9D8 G5978D 259>7
45@5>45>D?>D85B535>3I?6D85@B13D935"1<I1B4D"?<49>   #?G5F5B + *!4?5C
>?DD1;5D9=5C@5396931<<I9>D?133?E>D9D:ECD3?>C945BC@B13D935?B45B + *!=?49695CD85*!
6?B=E<1 2I B5@<139>7 D85 D?D1< >E=25B ?6 619<54 @B13D935C "  G9D8 D85 D?D1< >E=25B ?6 1<<
@B13D935CD8EC61B5CC5>D91<<I5AE1<9>7" C ,  1>4B5@<139>7D85D?D1<>E=25B?6CE335CC6E<@B13D935C, G9D813?=@?>5>D $ D81D9>3?B@?B1D5CD85>?D9?>?6=5=?BI4531I
*$"+
$
"
" 
$ E$*$"+" G85B5  K MB5@B5C5>DCD854531I613D?B 1>4" B5@B5C5>DCD85?ED3?=5?6@B13D935(96CE335CC6E<1>4 96619<546?BCDE45>D"?>C;9<<# $>D85?B979>1<+ *!@1@5B 1ED8?BCEC54
D8B5578?CD CI>D85D93@B13D935C"1<I1B4D"?<49>   (?D5D81D9>D8569BCD+ *!@1@5B51389D5=G1C<9>;54D?1C9>7<5C;9<< <?C9>7?>5?6D85=19>?B979>1<25>569DC?6*! -?D8525CD?6?EB;>?G<5475 D85B59C>?@B5F9?ECG?B;?>
5HD5>49>7 + *! D? 3B51D5 1 F1B91D9?> G8938 31> 81>4<5 41D1 3?>D19>9>7 =E<D9 C;9<< 9D5=C #?G5F5B B535>3I81C255>3?>C945B54 9>3<E49>71449>7D9=5 21C54G5978DCD?3?=@?>5>DC?65H9CD9>7=?45<C1>49>3?B@?B1D9>71G5978D54@B?@?BD9?>6?B619<EB5C*1F<9;5D1<   !EBD85BG?B;9CC55>9>&?79CD93%>?G<5475-B139>7&%-*1F<9;5D1<   1<?79CD93
B57B5CC9?> 21C54 6B1=5G?B; D81D 31> 5>12<5 =E<D9@<5 3?=@?>5>DC 6B?= 49665B5>D 5H9CD9>7
=?45<C 9>3<E49>7*!*1F<9;5D1<   *! 531I"?>75D1<   + *!"1<I1B4D
"?<49>   1>4** /1<C85D1<   &%-C8?G31C541CE9D5?63?=@?>5>DCD81D
3?E<4253?=29>54D?6?B=>5G=?45<C C?=5?6G89389>3?B@?B1D54D85>?D9?>?6B535>3I1>4
4531I  -G? ?6 D85 3?=@1B9C?> =?45<C G5 9=@<5=5>D  <7 1>4 <7 45C3B9254 6EBD85B 9>

26

,53D9?> 1B521C54?>3?=@?>5>DC45C3B92549>&%- $=@?BD1>D3?=@?>5>DC6B?=*1F<9;
5D1<  D81DG5EC549>?EB3?=@1B9C?>=?45<C1B51C6?<<?GC• $>D5B35@D6?B5138% C;9<< G89389C1C9=@<5<9>51B=?45<9>D5B35@D .C549><71>4<7 • <7&?7@5B6?B=1>35<?7CE3 <?7619< G89389CD85<?7 DB1>C6?B=54@5B6?B=1>35613D?B
D85D?D1<CE335CC5C?B619<EB5C B5@B5C5>D9>7453<9>9>7=1B79>1<B5DEB>C 5 7 <>*<> • <7 H@?>5>D91<4531I?6@B?@?BD9?> G8938EC5CD85@B9?B@B?2129<9DI3?BB53D6?B5138
;>?G<5475 3?=@?>5>D  1C 9> D85 + *! =?45<  -89C EC5C 1 @1B1=5D5B D? 45C3B925 D85
5H@?>5>D91<B1D5?64531I ?BB535>3I 6?B?2C5BF1D9?>C?61;>?G<54753?=@?>5>D • <7-85&%-F5BC9?>?6+535>3I 4569>541CD85@?G5B<?74531I1@@<954D?D85D9=5
9>D5BF1<C9>35D85@B5F9?EC5>3?E>D5BG9D8D85% -89C651DEB53?>C945BC?><ID85:ECD@B9?B
?2C5BF1D9?>1>4C9=E<1D5C@5B6?B=1>359=@B?F5=5>DG85>D85@B9?B@B13D935G1CB535>D $>D89C@1@5BG5@B?@?C51=?45<31<<54'5=53D81D31@DEB5C=5=?BI4531I1>4
C@139>7  1>4 3?=@1B5 9DC @5B6?B=1>35 D? *!  + *!  1>4 3?=@?>5>DC ?6 &%-  G89<5
133?E>D9>7 6?B =E<D9@<5 C;9<<C @5B 9D5=  /5 9=@<5=5>D 1>4 1>1<IJ5 DG? =5D8?4C ?6
9>3?B@?B1D9>7 4531I D8B?E78 @B13D935 ?B45B 1>4 D8B?E78 D9=5 G9>4?GC  -89C @B?@?C54
1<7?B9D8=3?>C945BCD8556653DC?6C@139>725DG55>@B13D935C D8EC=?45<9>7D85C@139>756653D
G85>@B5493D9>7CDE45>D@5B6?B=1>35 $D1<C?CDE495C=E<D9@<5G1IC?69>4E39>74531I G85B51C@B5F9?ECCDE495C6?3EC?><I?>?>5=5D8?4 =?CD<I@B13D935 ?B45B '?B5CDE4I?6D85C@139>7
56653D9>;>?G<5475DB139>79CG1BB1>D54 /89<5165G1<7?B9D8=C81F5ED9<9J54D85>?D9?>?6
D9=5G9>4?GC8?669>5D1<  &9>4C5I5D1<   D859BD9=5G9>4?GC?F5B<1@1>41B5
4569>549>1B5<1D9F5<IB5CDB93D9F51>4<9=9D54=1>>5B 449D9?>1<<I 8?669>5D1<  &9>4C5I
5D1<  4?>?D3?>C945BD9=55<1@C5425DG55>@B13D935CG85>=?45<9>7C@139>7 G85B51C
?EB@B?@?C54=?45<'5=53,@139>73?>C945BCD85D9=55<1@C5425DG55>51383EBB5>D1>4
@B5F9?EC @B13D935  G8938 G5 25<95F5 9C 3BE391< 6?B =?45<9>7 C@139>7 133EB1D5<I  <C?  9>
'5=53 D89C9C4?>56?B2?D8@B13D935 ?B45B1>4D9=5 G9>4?GF1B91D9?>C !?B'5=531>4
'5=53,@139>7G9D8D9=5G9>4?GC G54531ID85G5978D?61@B13D93521C54?>D85D9=5
G9>4?GD85@B13D93561<<C9>D? .C9>75AE9F1<5>D 49C:?9>DD9=5G9>4?GC1<<?GC6?B3?>C9CD5>3I
9>D855H@?>5>D91<B1D5?64531ID8B?E78D9=5 449D9?>1<<I G53?>4E3D1>9> 45@D81>1<IC9C
?>D859=@13D?6381>79>7D854531I613D?BC1>4D85>E=25B?678?CD@B13D935C    "$   $ $/5@B?@?C51>5G=?45< '5=53 1F1B91D9?>?6D85*!1<7?B9D8=9>C@9B542I+*! 3?=@?>5>DC  D81D 31> 1<C? 25 C55> 1C 69DD9>7 G9D89> D85 &%- 6B1=5G?B;  $> '5=53 
=5=?BI4531I9C1@@<954D?2?D8CE335CC6E<1>4619<54@B13D935C 1>4D85=?45<31>25EC54
G9D8=E<D9 C;9<<9D5=CG85B51CD85?B979>1<+ *!?><ICE@@?BDCC9>7<5 C;9<<9D5=C $>1449D9?>D?D89C G52E9<4?>D851@@B?1389>'195B5D1<   G8938C@<9DCC;9<<C
9>D?N3?==?>O?BNB1B5O31D57?B95C C?D81D9D31>251@@<9546?B<51B>9>7CICD5=CG85B5C?=5
C;9<<C1B5B1B5 +1B5C;9<<C31>?33EBG85>M6?B5H1=@<5 9D5=C1B5D17754G9D8C;9<<CD81D
B5@B5C5>D @B5B5AE9C9D5C >?D D1E78D 9> D85 3?EBC5G1B5  /85> DB19>9>7 *! G9D8 41D1C5DC
3?>D19>9>7B1B5C;9<<C C5F5B1<381<<5>75C9>3<E49>74575>5B1D5@1B1=5D5BC31>?33EB'195B
5D1<   5@5>49>7?>8?GB1B5C?=5C;9<<C1B5 D85B5=978D>?D255>?E7841D1@?9>DC
D?@B539C5<I5CD9=1D5@1B1=5D5BCG85>DB19>9>71=?45< '195B5D1<  @B?@?C5C1*!
F1B91>DD81DC@<9DC?ED3?==?>1>4B1B5C;9<<C /85>DB19>9>71=?45< 51383?==?>C;9<<81C
9DC?G>C5D?6@1B1=5D5BC G89<51<<B1B5C;9<<C1B53?=29>549>D?1C9>7<53?==?>C5D?64561E<D@1B1=5D5BC 9=@B?F9>7@B5493D9?>C1>4B54E39>7=?45<4575>5B13I'195B5D1<   -856?B=E<16?B'5=539C79F5>25<?G>?D5D81D1C6?B*!1>4+ *! 'G9<<25
9>@EDD549>D?1<?79CD936E>3D9?>D??2D19>1@B5493D9?>
*K'L      








     

27

 

      $ E








 





 

$ E 







( ) 



/85B5$B5@B5C5>DC1CDE45>D %B5@B5C5>DC1C;9<< - 9CD853EBB5>DDB91<9 5 @B13D935>E=25B
CDE45>D$9C?>G9D8C;9<<% " B5@B5C5>DCD853?BB53D>5CC?6D85@B13D9359 5 9D9C96CDE45>D$PC @B13D935 ( G9D8 C;9<<%G1C CE335CC6E<  1>4  ?D85BG9C5  ?>CD1>D#  K M9C D85 4531I
B1D5 6?B CE335CC6E< @B13D935C  1>4 3?>CD1>D  K MB5@B5C5>DC D85 4531I 613D?B 6?B 619<54
@B13D935C +,1>4+!B5@B5C5>DD85B535>3I G5978D54@B?@?BD9?>C?6@1CDCE335CC5C1>4@1CD619<EB5C  B5C@53D9F5<I  -85 F1<E5C 6?B1 2 1>43 1B5 @1B1=5D5BC D81D 1B5 <51B>54 6?B 5138
C;9<<%4EB9>7D85DB19>9>7 *1B1=5D5BC1 2 1>43 1B5D854561E<D@1B1=5D5BC<51B>546?BD85
B1B5C;9<<C )><I?>5F1<E59C<51B>546?B5138@1B1=5D5B6?BD85C5D?6B1B5C;9<<C &9;5+ *! '5=5331>9>3?B@?B1D578?CD CI>D85D93@B13D935C9=179>1BI@B13D935C
D81D 9=@B?F5 =?45< @5B6?B=1>35 9>9D91<<I 9> D85 +, 1>4 +! 6?B=E<1C  -? 1<<?G 6?B 78?CD@B13D935C G5CD1BD(6B?=1>571D9F5>E=25B9>CD514?6CD1BD9>76B?=  $>D85?B979>1<+ *!
6?B=E<1  D85 1ED8?BC @B?@?C54 D8B55 78?CD @B13D935C 1<< 619<54 @B13D935C  C 9> &%-  G5
9>F5CD971D5EC9>7>?78?CD@B13D935C DG?78?CD@B13D935C?>5CE335CC6E<1>4?>5619<54 1>4
D8B5578?CD@B13D935C1<<619<54 -85=19>49665B5>35C25DG55>+ *!1>4D85'5=5321C5
F1B91>D9CD81D'5=534?5C>?DEC51D?D1<D5B=9>CD514 9D3?>D19>C13?=@?>5>DD81DD1;5C
9>D? 3?>C945B1D9?> D85 G5978D54 @B?@?BD9?> ?6 619<54 @B13D935C  79F9>7 =?B5 G5978D D? B535>D?>5C <C? '5=5331>81>4<5=E<D9 C;9<<9D5=CG85B51C+ *!9C45C97>546?B?><IC9>7<5
C;9<< 9D5=C  449D9?>1<  + *! ?><I 3?>C945B54 1 @B13D935 ?B45B 1@@B?138  G85B51C ?>5
'5=53F1B91>D3?>C945BC1D9=5G9>4?G1@@B?138 $> '5=53  1C 9> =?CD =?45<C D81D 9>3?B@?B1D5 D85 >?D9?> ?6 4531I  D85 ?B45B ?6
@B13D935C@<1IC1>9=@?BD1>DB?<5  F5BID9=5D85CDE45>D3?=@<5D5C1>5G@B13D935 D85=?45<
9>DB?4E35C=?B54531ID?@B5F9?EC@B13D935C -89C=51>CD81DD85=?B5@B13D935CD85CDE45>D
81C D85=?B54531I9C1@@<954D??<45B@B13D935C $>D89C1@@B?138 G89<54531I9C9>3B5=5>D54
2I=?B5@B13D935 D85D9=55<1@C5425DG55>@B13D935C4?5C>?D16653DD8531<3E<1D9?>?64531I *B13D935C31>25C5@1B1D542I49665B5>D1=?E>DC?6D9=5 6B?=C53?>4CD?=?>D8C $D9C
E><9;5<I D81D CE2CD1>D91< 6?B75DD9>7 G9<< ?33EB G9D8 C=1<< 1=?E>DC ?6 5<1@C54 D9=5C  CE38 1C
C53?>4C?B=9>ED5C -89C3?E<4251<9=9D1D9?>D?D85@B13D935 ?B45B1@@B?138 96D851=?E>D?6
5<1@C54D9=531>F1BI3?>C945B12<I -8EC G5@B?@?C51F1B91D9?>?6'5=53D81DEC5C1D9=5
G9>4?G 13?>CD1>D4EB1D9?>?6D9=56?B5H1=@<541I9>G89389D5=C1>CG5B54G9D89>D85
C1=5 D9=5 G9>4?G 1B5 79F5> 5AE1< 4531I  -89C 133?E>DC 6?B =5=?BI 4531I >?D ?33EBB9>7
9>CD1>D1>5?EC<I *B13D935C1>CG5B549>D9=5G9>4?GC61BD85B6B?=D853EBB5>D@B13D93531>25
5H@53D54 D? 81F5 4531I54 =?B5 D81> @B13D935C 6B?= =?B5 B535>D D9=5 G9>4?GC  !?B D89CF1B91>D '5=53C+,1>4+!6?B=E<1C253?=5$ $ E

" 
$  $% !%($*$ +$*"+)
"
#


$ 

$ E$% !%($*$ +$*"+)


"
#

$% !%($*$ +$*"+)


"


F D " H $  $% !%($*$ +$*"+)
"



 

-85C@139>756653D81CD?4?G9D8D85D5=@?B1<49CDB92ED9?>?6@B13D935C<9>;54D?D85
C1=5C;9<< $6=9>9=1<D9=581C5<1@C5425DG55>@B13D935C D85<51B>9>79CC194D?25=1CC54 
H9CD9>7B5C51B38CE775CDCD81D96@B13D935C1B5C@1354?ED 9>6?B=1D9?>9CB5D19>54<?>75B9>
=5=?BI*1F<9;>45BC?>   $>D89CC53D9?> G59>F5CD971D51>5HD5>C9?>?6'5=53
D81D 9>3?B@?B1D5C D85 >?D9?> ?6 D85 C@139>7 56653D 9>D? ;>?G<5475 5CD9=1D9?>  -85 '5=53
=?45<G1C14:ECD54D?EC5# F1<E5CD81D1B531<3E<1D5421C54?>16?B=E<1G8938D1;5C9>D?
133?E>D8?GC@13541@1BD@B13D935C1B5 -85F1<E5C?6 1B531<3E<1D549>1C9=9<1B=1>>5B

28

# F- H E N# $"()*+!.*+-$!, )+!&*,! -$'! E 
F# C  ' K-$'!F- H D -$'!K- D HL # & L )-#!+/$,!

 $"()*+!.*+-$!, )+!&*,! -$'! E  F- H E NF C  ' K-$'!F- H D -$'!K- D HL L )-#!+/$,!
&

G85B5#   K M1>4# &  K MB5@B5C5>DD85=9>1>4D85=1HF1<E5CD81DG51<<?G6?B# ?>CD1>DC  K M1>4   K MB5@B5C5>DD85=9>1>4D85=1HF1<E5CD81D31>
25 EC54 6?B   ?>CD1>D 0 9C D85 21C5 ?6 D85 <?71B9D8=  1>4 5H@B5CC9?> -$'!KF- H D-$'!F- D HL31<3E<1D5CD855<1@C54D9=525DG55>D853EBB5>D@B13D935G9D8C;9<<#1>4D85
@B5F9?EC@B13D935G9D8C;9<<#@5B6?B=542ICDE45>D" (?D5D81D96D85CDE45>D81C>?@B5F9?EC
@B13D935CG9D81C;9<< ?BD855<1@C54D9=59C 5<1@C54D9=5?6 31>?33EB9>B51<CICD5=C96
D9=5CD1=@C 1B5 >?D 31@DEB54 1D 5>?E78 7B1>E<1B9DI  D85> G5 1CC97> D85 =9>9=E= F1<E5C '5=53,@139>73?E<425EC54G9D859D85B@B13D935?B45B@ ? ?BG9D81D9=5G9>4?G  $#$
!?BD855H@5B9=5>DC@B5C5>D549>D89C1BD93<5 G5EC5441D16B?=+5F51<'1D8?EBC5 1'3"B1G #9<<4979D1<=1D8@B?4E3DD81D3?F5BC7B145.,=1D83EBB93E<E= -859D5=C6B?=
D851CC5CC=5>DC6B?=D89C41D11B5D17754G9D8?>5?B=?B5C;9<<C -8541D1G5EC5431=5
6B?= DG? '94G5CD5B> C38??< 49CDB93DC 1>4 ?>5 ,?ED8G5CD5B> C38??< 49CDB93D  )>5 ?6 D85C5
C38??<49CDB93DC9CG9D89>1<1B75. , 39DIG85B5?F5B81<6?6CDE45>DC1B53<1CC969541C<13; 
1B?E>4 ?6CDE45>DC1B53<1CC969541C#9C@1>93 1>41696D8?661=9<95C<9F5E>45BD85@?F5BDI
<9>5 C53?>449CDB93D9CG9D89>1C=1<<D?G>G85B51B?E>4?6CDE45>DC1B53<1CC969541C<13; 1B?E>4 ?6CDE45>DC1B53<1CC969541C/89D5 1>41B?E>4 ?661=9<95C<9F5E>45B
D85@?F5BDI<9>5 D89B49CG9D89>1<1B75BD?G>G85B5:ECD?F5B81<6?6CDE45>DC1B53<1CC96954
1C#9C@1>93 :ECDE>45B81<6?6CDE45>DC1B53<1CC969541C/89D5 1>412?EDQ?661=9<95C<9F5
E>45BD85@?F5BDI<9>5 <<EC5D85("5>D5B ,,)?==?>?B5,D1>41B4C 
HDB13D5441D1C@1>C25DG55>E7ECD 1>4'1I  -85B51B5 E>9AE5
9D5=C 1B5D17754G9D8=E<D9@<5C;9<<C -859D5=C9>3<E45=E<D9@<538?935 69<<9>D852<1>; 
1>45>D5B9>75AE1D9?>9D5=C )F5B1<< D8541D1C5D814E>9AE5C;9<<CG8938G5B5<9>;54D?D85
9D5=C6B?=D8541D1C5D )ED?6D85C5C;9<<C G5B53<1CC969541C3?==?>1D<51CD CDE45>DC
G9D81D<51CD@B13D935CM'195B5D1<  1>4D85B5=19>9>7G5B53<1CC969541CB1B5 -8541D1C5D81C 41D1@?9>DC 1D1@?9>DCB5@B5C5>DCDE45>DCPB5C@?>C5C1>4
D859B>?B=1<9J54C3?B5C96D85B5C@?>C59C3?BB53D  969>3?BB53D  ?6D8541D1@?9>DC
3?>D19>541@1BD91<<I3?BB53DC3?B5 G8938G5B5DB51D541C 6?BD85@EB@?C5C?6D89C1>1<IC9C !?BD855H@5B9=5>DC G5C@<9D?EB41D1C5D9>D?DB19>9>71>4D5CD9>7C5DC /5B1>4?=<IC5<53D54
12?ED   ?6 D85 CDE45>DC  CDE45>DC     41D1 @?9>DC   C;9<<C 6?B D85 D5CD9>7 C5D <51F9>7 ?6D85CDE45>DC CDE45>DC  41D1@?9>DC C;9<<C6?BDB19>9>7  '!"$#%$#!?B F1<941D9?>  G5 B1> C5F5B1< 5H@5B9=5>DC EC9>7 D85 @B?@?C54 1@@B?1385C 6B?= D89C
1BD93<5  1C G5<< 1C ?D85B 5H9CD9>7 1<7?B9D8=C  !?B 3?=@1B9C?> B51C?>C  G5 9=@<5=5>D54 D85
?B979>1<*!*1F<9;5D1<  G9D814:ECD=5>DCD?81>4<5B1B5C;9<<C1C45C3B92549>'195B
5D 1<     /5 31<< D89C D85 1C5<9>5 =?45<  /5 1<C? 9=@<5=5>D54 ?D85B 1<7?B9D8=C D?
25>38=1B;1719>CD'5=531>4'5=53,@139>7+ *!"1<I1B4D"?<49>   1>4
DG? 1<7?B9D8=C D81D G5B5 9>C@9B54 2I =?45<C 6B?= &%- *1F<9; 5D 1<     /5 @B?F945
9>6?B=1D9?>?>D8?C59>D85?=@1B9C?>'?45<CCE2 C53D9?> !?B1<<=?45<C G51<<?G546?B
=E<D9 C;9<<9D5=C2IEC9>71CE==1D9?>613D?B13B?CC=E<D9@<5C;9<<C<9>;54D?1>9D5= $> 1> 566?BD D? CDE4I D85 49665B5>35C 1>4 D85 56653D9F5>5CC 25DG55> 5138 =?45<  G5
31<3E<1D54 D85 . 1>4 +',  F1<941D9?> =5DB93C  <C?  G5 @B5C5>D F1<941D9?> B5CE<DC 6?B49665B5>D7B?E@C?641D1@?9>DCG9D89>D85D5CD9>741D1C5DN1<<41D1O=51>CG5F1<941D541719>CD1<<41D1@?9>DCN>?> 4561E<DO=51>CD81DG5?><IEC5441D1@?9>DC6?BG8938D859D5=G1C

29

<9>;54D?1D<51CD?>53?==?>C;9<<N?><I>?> 4561E<DO=51>CG5?><IEC5441D1@?9>DCG8?C5
9D5=CG5B5<9>;54G9D85H3<EC9F5<I3?==?>C;9<<CN4561E<DO=51>CG5?><IEC5441D1@?9>DC
G8?C59D5=CG5B5<9>;54D?1D<51CD?>5B1B5C;9<<1>4N?><I4561E<DC;9<<CO=51>C41D1@?9>DC
G9D89D5=CC?<5<ID17754D?B1B5C;9<<C  !!#"!
/5DB19>541=?45<D81D<51B>54D8B55@1B1=5D5BC6?B5138?6D853?==?>C;9<<C1>4
D8B55@1B1=5D5BC6?BD85B1B5C;9<<C -85F1<941D9?>B5CE<DC1B5@B5C5>D549>-12<525<?G
-12<5 *$"& 1>4%-&"1"

"1%'-/4-),10"0%*),%
%+%#/"#1)#%/$%/
**$"1" 
  
   
,-,$%&"2*1
 
  
   ,*4,-,$%&"2*1

  
   
$%&"2*1  
  ,*4$%&"2*1      
 !!#"!
-?CDE4ID8549665B5>3525DG55>=5D8?4CEC54D?7?F5B>D859>3B51C59>4531I G59=@<5=5>D541>4D5CD541F1B91D9?>?6'5=53D81DEC54D85@B13D935 ?B45B1@@B?138 1CG5<<
1C 1 F1B91D9?> D81D EC54 D85 D9=5 G9>4?G =5D8?4  !?B D85 4531I 613D?BC6 1>46 G5 DB954
C5F5B1<3?=29>1D9?>C?6F1<E5C6B?= 0 /89<5?D85B3?=29>1D9?>C71F5C9=9<1BB5CE<DC D85
25CDG5B5?2D19>54G9D86  1>46   /5@B5C5>DD89C=?45<PCB5CE<DC9>-12<5 /5
?2C5BF51C97>96931>D9=@B?F5=5>DG85>3?=@1B54G9D8D851C5<9>5=?45< I9>3?B@?B1D9>7
D85 >?D9?> ?6 =5=?BI 4531I  '5=53 13895F54 1> . ?6 12?ED  ?> 1<< D5CD9>7
41D1@?9>DC G85B51CD8521C5<9>5B513854?><I1>.?612?ED  ,97>96931>D9=@B?F5=5>DC
G5B5 ?2C5BF54 9> 1<< ?D85B 31D57?B95C ?6 41D1@?9>DC  5H35@D 6?B 31D57?B95C 9>F?<F9>7 4561E<D
C;9<<C 6?BG8938D85DG?=?45<C13895F54C9=9<1B@5B6?B=1>35 <9;5<I2531EC5B1B5C;9<<C4?>?D
3?>D19>5>?E7841D1@?9>DC9>D8541D1C5D6?B?EB=?45<D?CE2CD1>D91<<I<51B>6B?= /51<C?B1>5H@5B9=5>DCG9D81D9=5 G9>4?G?6441IC 4 4 1>44C55
-12<5 -8525CDB5CE<DCG5B5?2C5BF546?B14D9=5G9>4?GG9D81>.5AE1<9>7 13B?CC1<<41D1@?9>DC6B?=D85D5CD9>7C5D G89389CC<978D<I<?G5BD81>D85.?6D85@B13D935?B45B=?45< -854G9>4?G=?45<?2D19>541>.?6 6?B1<<41D1 D854G9>4?G1>
.?6  1>4D854G9>4?G=?45<1>.?6  -89C45=?>CDB1D54D81D6?BD89C
41D1C5D D85=?45<D81DEC5CD85@B13D935 ?B45B1@@B?138@5B6?B=CC<978D<I25DD5B !?BD85D9=5
G9>4?GF1B91D9?>G5?2C5BF54D81DD85C=1<<5BD85G9>4?G D8525DD5BD85B5CE<DC -12<5 %."+!+"%."&'.*'")&+*"0*
"1%'-/4 $ $ $
**$"1"
  
  
  
,-,$%&"2*1
  
  
  
,*4,-,$%&"2*1
  
    $%&"2*1
  
  
  
,*4$%&"2*1      
  
-? CDE4I G85D85B =?45<9>7 1 3?=29>1D9?> ?6 2?D8 4531I 1>4 C@139>7 3?E<4 6EBD85B
9=@B?F5 D85 @B5493D9?>C  G5 B1> 5H@5B9=5>DC G9D8 D85 '5=53 ,@139>7 =?45<  /5
5H@5B9=5>D54G9D849665B5>DF1<E5C6?BD858I@5B@1B1=5D5BCG8938B5@B5C5>DCD85<?G5B1>4
E@@5B 2?E>4C ?6 D85 4531I 613D?B  1>4 6?B @B13D935 ?B45B G5 ?2D19>54 F5BI C9=9<1B B5CE<DC
3?=@1B54D?D85>?> C@139>7@B13D935?B45B'5=53=?45<C I1@@<I9>7@B13D935?B45B1>4

30

@1B1=5D5BC#   #    E    E    ' E    C8?G>9>-12<5
<56D3?<E=>G5?2D19>541C<978D<I25DD5BB5CE<DD81>'5=53G9D8?EDC@139>7 G9D81>?F5B1<<.?6  !?BD89C1@@B?1381>4D9=5G9>4?G?6441ID85.G1C  G89389C
C<978D<I<?G5BD81>D85.G85>EC9>7@B13D935 ?B45B@ ?  &1B75D9=5G9>4?GC@5B6?B=54
=?B5@??B<ICD9<< )F5B1<< G9D8D9=5G9>4?GC '5=53,@139>771F5C<978D<I@??B5BB5CE<DC
D81>D85'5=53=?45< -89C69>49>7=1I254E5D?35BD19>@B?@5BD95C?6D8541D1C5DG5EC5 '1>I41D1@?9>DC1B5>?DC@13541@1BD=?B5D81>165GC53?>4C9>D9=5 G8938G?E<431EC5
9>3?B@?B1D9>7D8556653DC?6C@139>7D8B?E78D9=5G9>4?GCD?81F5>57<9792<556653DC?>D85
=?45<PC31<3E<1D9?>?6CDE45>D;>?G<5475 /85>EC9>7@B13D935?B45BD?B5@B5C5>D4531I D85
56653D ?6 C@139>7 C55=C D? 25 >57<9792<5  $D 9C @?CC92<5 D81D 96 1 49665B5>D 41D1C5D G9D8 =?B5
G945<IC@1354@B13D935C9CEC54 D8556653D?6C@139>7?>'5=53G9D8@B13D935?B45B=978D
25=?B525>569391< $>,53D9?>G5G9<<49C3ECCD859>D5B@B5D1D9?>?6D85C5B5CE<DC6EBD85B -12<5 %("& ."+!()%*6  6  6  E A B 6  E A B
"1%'-/4. -  $$**$"1"     
  
,-,$%&"2*1
     
  
,*4,-,$%&"2*1
     
  
$%&"2*1
     
  
,*4$%&"2*1      
    !!
/5 3?=@1B5 ?EB =?45<C G9D8 + *! "1<I1B4D  "?<49>     1C G5<< 1C DG?1<7?B9D8=CD81D9>3?B@?B1D53?=@?>5>DC6B?=&%-*1F<9;5D1<   G8938G531<<<71>4
<7 -85'6E>3D9?>?65138=?45<9C9>@EDD549>D?D85C97=?946E>3D9?> D?75D1@B?2129<9DI
F1<E525DG55> 1>4 <73?>D19>C1B535>3I3?=@?>5>DD81D31@DEB5CD855<1@C54D9=5+
25DG55>3EBB5>D1>4@B5F9?EC@B13D935?6D85CDE45>D"G9D8C;9<<#B19C54D?14531I613D?B $D
1<C?D1;5CD85>1DEB1<<?71B9D8=?6D85>E=25B?6CE335CC5C;  1>4>E=25B?6619<EB5C7  <73?>D19>C13?=@?>5>DD81DEC5C1G5978D54@B?@?BD9?>?6@B5F9?EC@B13D935C1<?>7G9D8
1@1B1=5D5BD81DB5@B5C5>DCD855H@?>5>D91<B1D5?64531I 1>478?CD@1B1=5D5BC <7:K8 9  54 ; 7L E  F> C ? F;  H C @ K7  H C = <  L<7:K8 9  54 ; 7L E   G> C ? 



 





 

I


531EC5=1>I?6D85C5=?45<CEC54DG?619<54 CE335CC6E<78?CD@B13D935C ?B
D8B55619<5478?CD@B13D935C G51<C?9=@<5=5>D541>4B1>5H@5B9=5>DCG9D8'5=531>4
'5=53,@139>7EC9>7D89C3?=29>1D9?>?6DG??BD8B5578?CD@B13D935C !?B'5=53 D85
@B5C5>35?678?CD@B13D935C8141>57<9792<59>6<E5>35?>B5CE<DC -85B5CE<DC?6+ *!G9D8
49665B5>D>E=25BC?678?CDCCE335CC5C1>4619<EB5C1B579F5>9>-12<5 -85C569>49>7CC8?G
D81D'5=53C97>96931>D<I?ED@5B6?B=54+ *! B571B4<5CC?6D85>E=25B?678?CD@B13D935C -85>E=25B?678?CD@B13D935C494>?DC55=D?81F51=1:?B9>6<E5>35?>D85+ *!B5CE<DC 
<7@5B6?B=5425DD5BD81>+ *! G9D81>.?6 6?B1<<D5CD41D1@?9>DC <7 G9D8
2?D8DG?1>4D8B5578?CD@B13D935C @5B6?B=5425DD5BD81><7 G9D81>?F5B1<<.?6 6?B DG? 78?CD @B13D935C 1>4  6?B D85 =?45< G9D8 D8B55 78?CD @B13D935C  -85C5 B5CE<DC 
C8?G>9>-12<5 1B5CD9<<G?BC5D81>'5=531>4'5=53,@139>7 -12<5  6  ')")&+*,**&"$ !'*+()+"* 
"1%'-/4
'(-01 0 &
'(-01 0 &
'(-01**$"1"         
,-,$%&"2*1         
,*4,-,$%&"2*1         $%&"2*1      
,*4$%&"2*1
         31

-12<5  $ & $  &
"1%'-/4*'  '(-01*'  0 &
'(-01**$"1"
  
     
,-,$%&"2*1
  
     
,*4,-,$%&"2*1
  
  
  $%&"2*1
       
,*4$%&"2*1
    
  

)F5B1<< '5=531>4'5=53,@139>7?ED@5B6?B=541<<?D85B=?45<C9=@<5=5>D54
9>D89CCDE4I 9>3<E49>7*! + *! <7 1>4<7 /569>4D81DD85@B13D935 ?B45BF1B91D9?>
?6 '5=53 ,@139>7 1>4 '5=53 @B?F9454 D85 25CD @B5493D9?>C  G9D8 1 =9>9=1< 89785B
@5B6?B=1>35 C55> 9> '5=53 ,@139>7  ?D8 G5B5 6?<<?G54 2I D85 D9=5 G9>4?G '5=53
F1B91D9?>G9D81C<978D<I=?B5C97>96931>D49665B5>35 /89<5'5=53,@139>7G9D8D9=5 G9>4?G
G1C ?ED@5B6?B=54 2I '5=53 G9D8 D9=5 G9>4?G  9D G1C CD9<< =?B5 56653D9F5 D81> 1>I ?D85B
D5CD54 =?45<C 9> D89C 5H@5B9=5>D  -85 @B13D935 ?B45B =?45< G1C 12<5 D? 5CD9=1D5 CDE45>D;>?G<5475 =E38 =?B5 133EB1D5<I D81> 1C5<9>5 *! ?B + *!  449D9?>1<<I  G9D89> D85
'5=53F1B91>DC @B13D935 ?B45B=?45<CG5B5=?B556653D9F5D81>D9=5 G9>4?G=?45<C 1>4
78?CD@B13D935C8141>57<9792<556653D?>@5B6?B=1>35@B5493D9?>C  #%## 

%# #$>D89CG?B;G5CDE4954D853?7>9D9F5C395>353?>35@DC?6=5=?BI4531I1>4D85C@139>7
56653D9>D853?>D5HD?6F1B91>DC?6D85&?79CD93%>?G<5475-B139>76B1=5G?B; /53B51D541
>5G1<7?B9D8=31<<54'5=53G89385H@1>4C?>+ *! 1F1B91D9?>?6*!D81D9>3?B@?B1D5C
4531I 5C@9D5D8551B<I5=@81C9C?>=E<D9 C;9<<9D5=C259>71CDB5>7D8?6*!*1F<9;5D1<   D?D8525CD?6?EB;>?G<5475 D85B59C>?@B5F9?ECG?B;?>+ *!?B?D85BD9=5 9>F?<F54
5HD5>C9?>CD81D<??;541D41D13?>D19>9>7=E<D9 C;9<<9D5=C 1<D8?E783?=@?>5>DC9>&%-31>
81>4<5=E<D9 C;9<<9D5=C*1F<9;5D1<   44B5CC9>7D89C<9=9D1D9?>9>3B51C5CD851<7?B9D8=PC
B5<5F1>35D?B51< G?B<454E31D9?>1<CICD5=CG85B59D5=C1B51CC?391D54G9D8=E<D9@<5C;9<<C /56EBD85B5H@1>454'5=53D?31@DEB5D85C@139>756653D9>?EB=?45<'5=53,@139>7 /51<C?CDE495449665B5>DG1IC?6=?45<9>74531I D8B?E78D85?B45B?6@B13D935C1>42I
9>D5BF1<C ?6 D9=5 5<1@C54 25DG55> @B13D935C D9=5 G9>4?GC  -? D85 25CD ?6 ?EB ;>?G<5475
@B5F9?EC 5HD5>C9?>C ?6 *! 1>4 &%- 3?=@?>5>DC =?CD<I 6?3EC54 ?> 1 @B13D935 ?B45B
1@@B?138  /5 D5CD54 G85D85B 49665B5>D F1<E5C ?6 D85 4531I 613D?B <54 D? 9=@B?F54 =?45<@B5493D9?>C  9> 1<< F1B91D9?>C  )EB >5G 1<7?B9D8=C G5B5 3?=@1B54 1719>CD DG? 3?=@1B9C?>
1<7?B9D8=C21C54?>&%-3?=@?>5>DC <71>4<7 1CG5<<1C1719>CD*!1>4+ *! -85B5CE<DC?6D89CCDE4IC8?G54D81D'5=531>4'5=53,@139>7?ED@5B6?B=541<<?D85B3?=@1B9C?>=?45<C *B13D935 ?B45B'5=53F1B91D9?>CC8?G5425DD5BB5CE<DCD81>D9=5
G9>4?GF1B91D9?>C /59>F5CD971D5449665B5>DD9=5G9>4?GC9J5C 6B?=4D?4G9>4?GC 1>4
6?E>4D81DC=1<<5BD9=5 G9>4?GC13895F5425DD5BB5CE<DC -85CDE4I1<C?6?E>4D81D=?45<9>7
4531I G9D8 D85 C@139>7 56653D 494 >?D C55= D? @B?F945 1> 14F1>D175 ?F5B C?<5<I =?45<9>7
4531I !?B@B13D935 ?B45B '5=53,@139>7@5B6?B=54C<978D<I25DD5BD81>'5=53 3B?CC
D9=5G9>4?GC '5=53?ED@5B6?B=54'5=53,@139>72I1C=1<<1=?E>D #?G5F5B D85C569>49>7C=1I254E5D?D85B5<1D9F5<I=1CC54>1DEB5?6D853EBB5>D41D1C5D -85B56?B5  9D =1I 25 F1<E12<5 6?B 6EDEB5 G?B; D? 3?=@1B5 D85C5 =?45<C G9D89> 41D1C5DC
3?>D19>9>7 =?B5 C@1354 9D5=C  D? 45D5B=9>5 G85D85B D85 D9=5 G9>4?G 1@@B?138 3?E<4 25
25>569391<?F5B@B13D935 ?B45B9>D89CC9DE1D9?> -89CG?E<41<C?C8?GG85D85B9>3?B@?B1D9>7D85
C@139>756653D1<?>7G9D84531I31>81F5129775B9=@13D?>@B5493D9?>6?BCE3841D1C5DC 
>?D85B 1B51 ?6 6EDEB5 G?B; 9>F?<F5C <??;9>7 9>D? 8?G G5<< D85 1@@B?1385C @B5C5>D54
@5B6?B= 1D @B5493D9>7 B5D5>D9?> <?>7 D5B=  !9>1<<I  6EDEB5 G?B; 9> D89C 1B51 =1I 25>569D 6B?=
7?9>725I?>4C9=@<I1CC5CC9>7@B5493D9F57??4>5CCD?1CC5CC9>7D85@B13D931<9=@<931D9?>C?6
G85>9>CDBE3D?BC1B5D?<41CDE45>D81C=1CD5B541C;9<< G85>9>613DD85I81F56?B7?DD5>9D )F5B1<<  D85 613D D81D '5=53 1>4 '5=53 ,@139>7 ?ED@5B6?B=54 D85 ?D85B =?45<C
8978<978DCD859=@?BD1>35?631@DEB9>73?7>9D9F5C395>35@B9>39@<5CCE381C=5=?BI4531I1>4
C@139>7G85>=?45<9>7CDE45>D;>?G<54751>4@B5493D9>76EDEB5@5B6?B=1>35 -851>1<IC9C

32

3?>4E3D541<C?C8?G31C5CD8549665B5>359>=?45<@5B6?B=1>3525DG55>9>3B51C9>74531I2I
59D85B ?B45B ?B D8B?E78 D9=5  -85 B5CE<DC C8?G D81D D85 @B?@?C54 =?45<C 1B5 CE9D12<5
;>?G<5475DB139>71@@B?1385C6?BB51< G?B<4141@D9F5<51B>9>7CICD5=CG9D8=E<D9 C;9<<9D5=C 
G85B5D85B51<@?CC929<9DI?6CDE45>DC6?B75DD9>7C;9<<C31>C97>96931>D<I9=@13DD85B5CE<DC "#
6=447<$=>7<3/C=C@2/-*73
' !=23:7<5'BC23<B 3/@<7<5/<2
=@53BB7<54=@#>B7;/::G'1632C:7<57AB@70CB32$@/1B713=4'97::A!42'))(-1+52*6,) 6,
16)41&6-21&/21*)4)1')21(7'&6-21&/&6&-1-1+
/@,7D    D 
=@03BB(<23@A=<& <=E:3253(@/17<5!=23:7<5B631?C7A7B7=<=4$@=132C@/:
<=E:3253$5)42()/-1+&1($5)4(&36)(16)4&'6-21    
5:7<5B=< $/D:79@$
#>B7;7H7<5>@/1B713A1632C:7<5@3?C7@3A?C/<B7B/B7D3B@/197<5=47<27D72C/:7B3;>3@4=@;/<13 !"'-)1')2*)&41-1+  
/:G/@2B=:27<
&313<B>3@4=@;/<134/1B=@A/</:GA7A!42'6,1621*(7'&6-21&/
&6&-1-1+
K
6=A63443@</<" /<'
=<B3FBE/@3BB3<B7D3<=E:3253(@/17<5 !42' 6,"16)41&6-21&/21*)4)1')21129/)(+)-5'28)4;&6&-1-1+=<5-3193443@</<"(=EB=1=<AB@C1B;=@3/11C@/B3ABC23<B;=23:A
=;>/@7<5/<2=>B7;7H7<59<=E:3253B@/17<5/<2>3@4=@;/<134/1B=@/</:GA7A1646-*16)//(7'D=: >> K /<6BB>2=7  
6/8/6!C/<5-=<H/:3H@3<3A$!=H3@!@CA7:=DA9!
0<B35@/B7<5
<=E:3253(@/17<5/<2B3;&3A>=<A3(63=@G(/:3=4(E=@/;3E=@9A!42'))(-1+52*
%24.5,2321!)4521&/-<&6-213342&',)5-1)&41-1+18-4210)165>>K 
6/8/6! 7<2A3G&*!=H3@!
=E33>7A<=E:3253(@/17<5!42'))(-1+52*6,)
6,16)41&6-21&/21*)4)1')21(7'&6-21&/&6&-1-1+

7<2A3G&*'6@=G3@$/A6:3@!=H3@!
;>@=D7<5ABC23<BAL:=<5B3@;
9<=E:3253@3B3<B7=<B6@=C56>3@A=</:7H32@3D73E!5;',2/2+-'&/"'-)1')   K !/73@/93@&''B/:H3@'6/::3<53AB=>>:G7<5$3@4=@;/<13/1B=@</:GA7A7<F7AB7<5 3/@<7<5'GAB3;A!42' 6,16)41&6-21&/21*)4)1')2120376)45-1(7'&6-21
!=H3@!$/A6:3@3>32/" 7<2A3G&*C:
$@3271B7<5B63#>B7;/:'>/17<5=4
'BC2G!C:B7A1/:3=<B3FB!=23:=4!3;=@G<(8&1')5-1 )74&/1*240&6-21!42')55-1+
";56)05 >>
#32/'A/7
'BC23<B!=23:7<5!3B6=2<B35@/B7<5<=E:3253(@/17<5/<2&(E7B6
31/G4431B<%>> K 
$/D:79$<23@A=<)A7<5/!=23:B==;>CB3B63#>B7;/:'1632C:3=4$@/1B713
2741&/2*:3)4-0)16&/!5;',2/2+;33/-)(   #  ,  $/D:79$5:7<5B=< /@@3::+7::7/;A 
 =57AB719<=E:3253B@/17<51=<AB@/7<32
4@/;3E=@94=@:3/@<3@;=23:7<5#4&15&'6-21521)&41-1+#)',12/2+-)5  K $/D:79$3<=327<53@&$3@4=@;/<13/1B=@A</:GA7AK"3E:B3@</B7D3B=
<=E:3253(@/17<5!42'16>/21*)4)1')2146-*-'-&/16)//-+)1')-1(7'&6-21>>  
$/D:79$@<23@A=<&$@/1B713/<2=@53BB7<54431BA=<*=1/0C:/@G!3;=@G<1B7D/B7=</A32!=23:=4B63'>/17<54431B2+1-6-8)"'-)1') 
$3:I<39&/G3A7/<9<=E:3253B@/17<5:=57AB71;=23:A/<203G=<2/<=D3@D73E=4:3/@<3@  
;=23:7<5B316<7?C3A$5)42()/-1+&1($5)4(&36)(16)4&'6-21 $7316/AA3<C/<5/<5C:7''/6/;7!C70/A '=6:719AB37<

33><=E:3253(@/17<5(8&1')5-1 )74&/1*240&6-21!42')55-1+";56)05   '16;C193@&!7B163::(!
(@/<A43@@/0:3'BC23<B$3@4=@;/<13!=23:7<54=@<B3::753<B(CB=@7<5'GAB3;A!42' 6,16>/21*2120376)45-1(7'&6-21
'1@C55A&/93@&'!1 /@3<!
FB3<27<533><=E:3253(@/17<5<43@@7<5
<B3@>@3B/0:3<=E:3253/<2$@3271B7<5$=AB'GAB3;$3@4=@;/<13!42'))(-1+52*6,) 6,16)41&6-21&/21*)4)1')2120376)45-1(7'&6-21
'63<' 7C%63<C/<5.C/<5+-7<-'C-+/<5' 3/@<7<5$@=13AA1=<A7AB3<B<=E:3253(@/17<5> 7+756 = +/:A6!!:C19C<H3:;/<</AB@H3;0A97(@CA;/@9!!GC<5$7BB!
.6=C&!316/<7A;AC<23@:G7<5B63A>/17<534431B7<:3/@<7<51=;>/@7A=<=4B6@33
1=;>CB/B7=</:;=23:A2741&/2*:3)4-0)16&/!5;',2/2+;)1)4&/ 

+7A363/@B!$/A6:3@*C:+7FB32&=6@3@
7AB@70CB32$@/1B7137<*3@0/:
&31/::(/A9A&3D73E/<2%C/<B7B/B7D3'G<B63A7A!5;',2/2+-'&/7//)6-1
    #     33

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Developing a Video-based e-Learning
System Incorporating a Fill-in-the-blank
Question-type Concept Map
Takumi HASEGAWAa & Tessai HAYAMAa*a
Nagaoka University of Technology, JAPAN*t-hayama@kjs.nagaokaut.ac.jp
Abstract: E-learning, which can be used anywhere and at any time, is very convenientand has been introduced to improve learning efficiency. However, securing acompletion rate has been a major challenge. Recently, the learning forms of e-learning
require learners to be introspective, deliberate, and logical and have proven to beincompatible with many learners with low completion rates. In this study, we propose
an e-learning system that incorporates a fill-in-the-blank question-type concept map todeepen learners’ understanding of learning contents while watching learning videos.
The proposed system promotes active learning reflectively and logically by allowing
learners to answer blank question labels on concept maps from video content andlabels associated with the question labels. We conducted an experiment with 14
subjects to evaluate our proposed system by comparing it with a conventional videobased e-learning system and confirmed that the proposed system promotes active
learning while watching lecture videos and is effective in improving the comprehensionof learning content for learners who are unfamiliar with video-based learning.
Keywords: e-Learning Support, Visual Thinking Tool, Video-based Learning

1. Introduction
In online video-based learning, a learner typically does not have to be in direct contact withthe class at the same time, which has significant advantages. It supports learning anywhere
and anytime via a mobile phone or a computer and enables access to learning that has neverbeen available before. Recently, although online learning courses have been increasingly
marketed to adults and students, the high dropout rate from such courses remains a concern
to educational institutions. Much research has shown a high dropout percentage rate oflearners participating in online courses compared with learners in a face-to-face classroom;
e.g. more than 95% of learners dropped out from MOOC (Alraimi, Zo, & Ciganek (2015)) andapproximately 78% of learners dropped out from Open University (Simpson (2013).
According to a survey report (Tominaga & Koga (2014)), the reasons for learners
dropping out of online learning courses are classified into two major problem categories:
learners' procrastination and incompatible learning style to e-learning. The first problem
category entails that some learners cannot effectively manage the time and learning progressof e-learning by themselves, which makes it difficult to complete e-learning courses. To solve
the first problem category, some e-learning institutions have introduced online tutoring andmentoring to provide individual support to such learners (Huang, Lin, & Huang (2012)). Others
have identified learners who are likely to drop out by analyzing their learning activities fromsystem logs (Mduma, Kalegele, & Machuve (2019)). These approaches are effective ways tomaintain the retention rate of e-learning courses (Grau, & Minguillon (2013)). Meanwhile, thesecond problem category is that multimedia content provided by e-learning systems generates
cognitive load effects on learners compared with typical face-to-face learning in the classroom(DeLeeuw & Mayer (2008)). The cognitive load effects in e-learning reduce the learning
performance of learners, increasing the tendency of learners to drop out. To solve the second
problem category, some researchers have designed e-learning systems with functions tomatch the learning style of e-learning (Kizilcec, Papadopulolos, & Sritanyaratana (2014);

34

Mayer(2005); Guo, Philip, Kim, & Rubin (2014)). Despite creating a better approach than thenormal e-learning style, many recent e-learning systems still adopt a simple form with a video
in the center of the interface. Moreover, some researchers have investigated the learning styleof e-learning and demonstrated that both reflective and active learning styles are relevant toenhance the e-learning progress and that sensory learners are probably adaptive to thelearning style of e-learning(Huang, Lin, & Huang(2012); Dalmolin, Mackeivicz, Pochapski,
Pilatti, & Santos (2018)). Therefore, it is required for a learner to perform learning withmetacognitive skills and a step-by-step thinking method during e-learning. However, it isdifficult for many learners to use their skills because current video-based learning affords
passive learning, and it is challenging for learners to leave the concepts of learning content
while watching the video. Furthermore, learners who have dropout tendencies cannot
effectively understand the learning content from the beginning of an e-learning course(Simpson (2013)). If the e-learning system has support to match a learner's thinking methodin e-learning and learning style, which e-learning requires, the dropout rates in each e-learning
course will be reduced.
Therefore, in this study, we propose an online video learning interface that provides afill-in-the-blank question-type concept map. A concept map is an effective way to provide
complex concepts with graph representations and frequently used in educational institutions(Novak (1990)). However, it is almost impossible for them to create an appropriate concept
map while watching a lecture video(Hayama & Sato(2020)). Thus, we employed a fill-in-theblank question-type concept map in which some labels that are important for understanding
learning contents are punched out from a completed concept map, encouraging learners tothink about them while watching the video. To achieve this, we also developed a method forextracting labels for fill-in-the-blank questions from the concept map. We conducted anexperiment with 14 subjects to evaluate our proposed system by comparing it with aconventional system with a concept map and confirmed that the proposed system promotes
active learning while watching a lecture video and is effective in improving the comprehensionof learning contents for learners who are unfamiliar with video-based learning.

2. Related Studies
A concept map is a useful tool for supporting learners' deep understanding, evaluating theirunderstanding, and so on (Novak (1990); Schmid & Telaro (2015)) as it can visually represent
the relationships between various concepts. Some researchers have used concept maps to
support learning by reflecting on the learning content of a class, complementing or deepening
the understanding of the learning content after the class. For example, Cimolino & Kay (2002)
confirmed the effectiveness of concept maps for teachers to assess students' understandingof concepts in a subdivided learning area by having students create and analyze concept
maps to recognize conceptual understandings and misunderstandings of the learning content.
As it is difficult for all learners to create proper concept maps, some researchers havedeveloped a method using Kit-Build concept maps, where experts prepare components withlabels and links in advance, and learners assemble them. For example, Hirashima, Yamasaki,
Fukuda, & Funaoi (2011) demonstrated that Kit-Built concept maps can be used to diagnosea learner's comprehension of learning content and to compare the concept maps of learners
with those of experts, respectively, for learning assessment. Hayashi (2019) presented more
examples in which a Kit-Built concept map was highly useful in actual classroom teaching andcooperative learning. Conversely, some researchers have developed systems incorporating
fill-in-the-blank concept maps, where some labels are masked from a concept map created by
an expert. Chang, Sung, & Chen (2001) developed a system using fill-in-the-blank concept
maps. They confirmed that the system was more effective than conventional concept maps interms of students' understanding of learning content through the task of creating a concept
map of learning content after the class.
Nevertheless, with the recent spread of online learning, technologies using concept
maps while watching lecture videos have been developed. For example, Liu, Kim, & Wang(2018) developed a system incorporating concept maps as navigation while watching lecture
videos. In their study, they proposed a concept map creation method by aggregating multiple

35

concept maps created by general users and confirmed that the concept maps created by their
method contained contents similar to those created by experts. However, their method only
allows learners to pore over concept maps while watching a lecture video. To promote a more
active learning style, learners need to interact when creating concept maps while watchinglecture videos. In this study, we propose a learning support system incorporating a concept
map with fill-in-the-blank questions, which makes a learner answer questions about important
points of the learning contents while watching a lecture video. To achieve this, we develop amethod for identifying the fill-in-blank labels on the concept map to understand the learning
contents. To the best of our knowledge, this is the first study on the above approach.

3. Developed System
We developed a video-based e-learning system using a concept map with fill-in-the-blankquestions. As depicted in Figure 1, the developed system consists of a lecture video-watchingarea, a concept map display area, and an answer input area for fill-in-the-blank questions.

Figure 1. Interface of the Developed System.
In the video-watching area, lecture videos, including lecture slides and lecture
speeches, are provided. These lecture videos can be played and stopped using buttons, andthe playback position can be specified using the playback bar. In the concept map display
area, the locations of multiple labels on the concept map created by the lecturer or otherexperts are automatically punched out and displayed. As shown in the concept map display
area, the yellow blank labels have been automatically punched out of the concept map. Eachlabel is automatically associated with the playback position of the lecture video that may
contain the content of the label. To make the user notice the label(s) associated with the
playback position of the lecture video, the label(s) is/are colored orange, and a link to thecorresponding video playback position is also set.
In the answer input area for the fill-in-the-blank questions, a label candidate list isprovided, as well as an execute button for matching the answers. In the label candidate list, alist of punched-out labels is displayed. In the concept map display area, for each label selectedwith the mouse, the item selected in the candidate list is inserted into the blank label. When auser presses the execute button to match the answers, the system checks whether the item
inserted into the blank label in the concept map is correct or not, and if the wrong item isinserted into the blank label, the label turns red to notify the user that it is a misunderstanding.
While watching a lecture video, the user checks the part of the concept map related to
the learning contents being watched, considers the learning contents corresponding to theblank label, and inserts the item into the label by selecting it from the candidate label list. The

36

user then confirms whether his/her answers are correct, and if there is/are wrong answer(s),the user reconfirms the corresponding learning contents in the video. Thus, the systemenables the user to always be aware of important points of lecture contents to pay attention toand to watch the lecture video while confirming information related to the important points.
Label extraction for fill-in-the-blank questions on concept maps:
We created heuristic rules to automatically extract labels for fill-in-the-blank questions froma concept map. To extract fill-in-the-blank question labels, labels related to important parts oflearning content are identified on the basis of the number of links to each label and the kindsof link labels, and blank labels should be recognized from the surrounding labels. As link
labels for concept maps, simplified by trial and error, the labels of “Method,'' “Description,''“Function,'' “Example,'' “Contents,'' “Advantage/Disadvantage,'' and other user’s definitionphrases were used. The label extraction rules for fill-in-the-blank questions are listed below.
(a) If a label outputs a link with a ``Description'' label, the label is left blank because theanswer to the label can be inferred from the output link label that describes it.
(b) If a label outputs a link with a label that appears only once in the concept map, thelinked label is left blank because the content of the linked label is inferred from contents
associated with the unique user's defined link label.
(c) If a label outputs multiple labels linked with a link label of ``Method," ``Description,"``Function," or ``Contents," the odd-numbered label(s) from the left side of the linked
labels is/are left blank because the characteristics of the contents of the linking label are
inferred from the linked labels. However, if a blank label has no output link, the blank to
the label is canceled.
(d) If a label outputs link(s) with a ``Function'' label, the linked label(s), which have morethan one output link(s), is/are left blank because the content of each label can beinferred from the contents of labels linking/linked to the label.
(e) If a label does not output a link and is linked from multiple labels, the label is left blankbecause the content of the label is inferred from the contents of multiple labels linked to
the label.
(f) If a label outputs links with ``Advantage'' and ``Disadvantage'' labels, the labels to which
each link of their labels is output are left blank because the contents of the label can beinferred from the link labels with its advantage and disadvantage.
(g) If a label outputs only a link(s) with an ``Example'' label(s), the label is left blank because
the content of the label is inferred from the contents of the example(s) of the label.

4. Evaluation
4.1 Settings
We conducted an experiment to investigate the effectiveness of our proposed system in termsof comprehension of the learning content and interaction with the system. In this regard, thedeveloped system and a video-based system that incorporates a concept map as it is werecompared based on the comprehension test results and the operation history of the system.
Two lecture videos (approximately 50 min each) on the subject “Information Networks”
were selected for the experiment. Fourteen undergraduate and graduate students majoring in
information engineering, but who had never taken the subject, were recruited as subjects. For
cross-validation, the subjects were divided into two groups (8 and 6 subjects, due to theabsence of two subjects). The time limit for each lecture was set to 1 h, and each subject
completed the lectures to his or her satisfaction. A comprehension test was then conducted
for each lecture, consisting of 29 and 26 correct/incorrect questions on the learning contentsof each lecture. Given the difference in the number of questions and the difficulty level of thelectures, the subjects’ deviation scores for each lecture were compared between the twosystems for learning comprehension. The system operation history for each subject during theexperiment was recorded with timestamps in a file, including the playback positions of thelearning videos and the answers to the fill-in-blank questions on the concept maps. The

37

concept maps for Lectures 1 and 2 used in this experiment were similarly composed of 77 and67 labels, 35 and 29 blanked labels, and 83 and 76 links, respectively. The concept maps andcomprehension tests were created by the lecturer.

4.2 Results
The results of the comprehension test are shown in Figures 2 and 3, respectively.
65.00
Deviation value

60.00
55.00
50.00
45.00

40.00
35.00
The proposed systemThe comparison system

Figure 2. Average deviation values of learners’ Figure 3. Deviation values ofcomprehension scores using the developed comprehension scores for each subject withsystem/comparison system.
lower performance when using thecomparison system.
We calculated the deviation values of the subjects' comprehension scores for eachlecture and compared them separately for the proposed and comparison systems. As shownin Figure 2, there was little difference between the proposed and comparison systems.
Conversely, the deviation values of the comprehension scores were examined for the sevensubjects with lower performance when using the comparison system, which requires aconventional learning style for a learner. As shown in Figure 3, the results showed that thefive subjects improved their performance when using the proposed system, with subject (C)exhibiting a significant difference in performance when using both systems and subject (E)exhibiting lower performance; thus, this indicates that subjects with low performance in theconventional system improved their understanding of the learning contents in video-basedlearning by using the proposed system.
Next, we confirmed that all of the system functions were used while watching thelearning videos. All subjects provided correct answers to the fill-in-blank questions on thelabels for each lecture. The number of trials for matching the correct answers to the fill-in-theblank questions was 4.3 and 4.8 for the 77 and 67 questions in Lectures 1 and 2, respectively,
indicating that the subjects answered some of the questions before jointly checking theircorrectness. The number of times the subjects intentionally moved the playback position of
the lecture videos during learning was 29.4 and 12.4 for the proposed and comparison
systems, respectively, indicating that the subjects moved the playback position of the lecture
videos more frequently with the proposed system than with the comparison system, with astatistically significant difference.
The post-questionnaire on the proposed system showed good results, with all items
being rated at 3.4 or higher out of 5. For the learning environment of the proposed system,“Satisfaction with his/her learning,” “Level of understanding of the learning contents,”“Sufficiency of the learning time,” and “Usability of the proposed system” were highly rated,indicating that the proposed system is a favorable learning environment to introduce. For thefunctions of the proposed system, “Effectiveness of the system function ‘concept maps with
fill-in-the-blank questions’ in understanding the learning contents” was rated very high at 4.36
of 5. From the free description items, it can be inferred that the proposed system made it easierfor the learners to watch and learn lecture videos by making them aware of what they need to
pay attention to while watching the videos. In addition, we confirmed that the function of “thelabel extraction for the fill-in-the-blank questions on the concept map” was properly applied to38the concept maps of Lectures 1 and 2 because the labels for the fill-in-the-blank questions
were related to the questions of the comprehension tests.
Overall, the proposed system promotes active learning by having learners answer fillin-the-blank questions on a concept map while watching a lecture video and can provide
learning effects to improve the understanding of learning contents for learners who do not fit
the learning style of conventional video-based learning.

Acknowledgements
This work was supported by JSPS KAKENHI Grant Numbers 19K12264 and 23K11364.

References
Alraimi, K. M., Zo, H., & Ciganek, A. P. (2015) Understanding the MOOCs continuance: The role ofopenness and reputation. Computers & Education, 80, pp.28-38.
Chang, K. E., Sung, Y. T. & Chen, S. F. (2001) Learning through computer-based concept mappingwith scaffolding aid, Journal of Computer Assisted Learning, vol. 17, no. 1, pp. 21-33.
Cimolino, L., & Kay, J. (2002) Verified Concept Mapping for Eliciting Conceptual Understanding,
Proceedings of ICCE2002, pp. 1561-1563.
Dalmolin, A. C., Mackeivicz, G. A. O., Pochapski, M.T., Pilatti, G.L., & Santos, F.A. (2018) Learning
styles preferences and e-learning experience of undergraduate dental students. Revista deOdontologia da UNESP, 47(3), pp.175-182.
DeLeeuw, K., & Mayer, R. E. (2008) A comparison of these measures of cognitive load: Evidence forseparable measures of intrinsic, extraneous, and germane load. Journal of Educational
Psychology, 100: pp.223-234.
Grau, J., & Minguillon, J. (2013) When procrastination leads to dropping out: analysing students at risk.
eLC Research Paper Series, 6, pp.63-74.
Guo, Philip J., Kim, J. & Rubin, R. (2014) How video production affects student engagement: anempirical study of MOOC videos. In Proceedings of the first ACM Learning@ scale conference,
pp.41?50.
Hayashi, Y., (2019) Open Information Structure Approach to Learning Support and Management with
Kit-build Concept Map, The 4th International Conference on Education and Management炷COEMA
2019炸, PB - Atlantis Press, pp.82-88.
Hayama T. & Sato S. (2020) Supporting Online Video e-Learning with Semi-automatic Concept-Map
Generation. In: Zaphiris P., Ioannou A. (eds) Learning and Collaboration Technologies. Designing,
Developing and Deploying Learning Experiences. HCII 2020. Lecture Notes in Computer Science,
vol 12205, pp.64-76.
Hirashima, T., Yamasaki, K., Fukuda, H., & Funaoi, H., (2011) Kit-Build Concept Map for Automatic
Diagnosis, Proc. of AIED2011, Lecture Notes in Computer Science, Vol. 6738/2011, pp.466-468.
Huang. E., Lin. S., & Huang. T. (2012) What type of learning style leads to online participation in themixed mode e-learning environment? A study of software usage instruction. Computers &Education, 58(1), pp.338-349.
Kizilcec, R. F., Papadopulolos, K., & Sritanyaratana, L. (2014) Showing face in video instruction: Effects
on information retention, visual attention, and affect. In Proceedings of the Annual SIGCHI
Conference on Human Factors in Computing Systems, pp.2095-2102.
Liu, C., Kim, J., & Wang, H. (2018) ConceptScape: Collaborative Concept Mapping for Video Learning.
In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Paper
N.387, pp.1-12.
Mayer, R. E.(2005) Principles of multimedia learning based on social cues: Personalization, voice, andimage principles. In The Cambridge handbook of multimedia learning, R. E. Mayer, Ed. Cambridge
University Press, pp. 201-212.
Mduma, N., Kalegele, K., & Machuve, D. (2019) A Survey of Machine Learning Approaches andTechniques for Student Dropout Prediction. Data Science Journal, 8(14), pp.1-10.
Novak, J.D. (1990) Concept Mapping: A Useful Tool for Science Education, Journal of Research in
Science Teaching, 27(10), pp.937-949.
Schmid, R., & Telaro, G., (2015)&oncept Mapping as an Instructional Strategy for High School Biology,
The Journal of Education Research, pp. 78-85.
Simpson, O. (2013) Student retention in distance education: Are we failing our students - Open
Learning: The Journal of Open, Distance and e-Learning, 28(2), pp.105-119.
Tominaga, A., & Kogo, C. (2014) Developments and issues in practical studies of e-learning. The
Annual Report of Educational Psychology in Japan, 53, pp.156-165 (in Japanese).

39

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring Students' Adoption of ChatGPT
as a Mentor for Undergraduate Computing
Projects: PLS-SEM Analysis
Swapna GOTTIPATI*, Kyong Jin SHIM, Venky SHANKARARAMAN
School of Computing and Information Systems, Singapore Management University,
Singapore
Swapnag,kjshim,venks@smu.edu.sg
Abstract: As computing projects increasingly become a core component ofundergraduate courses, effective mentorship is crucial for supporting students' learningand development. Our study examines the adoption of ChatGPT as a mentor forundergraduate computing projects. It explores the impact of ChatGPT mentorship,
specifically, skills development, and mentor responsiveness, i.e., ChatGPT’s
responsiveness to students' needs and requests. We utilize PLS-SEM to investigate
the interrelationships between different factors and develop a model that captures theircontribution to the effectiveness of ChatGPT as a mentor. The findings suggest thatmentor responsiveness and technical/design support are key factors for the adoptionof AI tools like ChatGPT. The study provides practical implications for educators
seeking to incorporate AI as a mentor to support students doing computing projects
and contributes to the broader understanding of the use of AI in education.
Keywords: Artificial Intelligence, ChatGPT, mentorship, computing projects, PLS-SEM
analysis

1. Introduction
Recently, chatbots and virtual assistants gained popularity in education, providing
personalized support to students (Mhlanga, 2023). ChatGPT, an AI language model by
OpenAI, has been used in education (Sok et al., 2023) but not extensively as a mentor forcomputing projects. This study investigates the impact of ChatGPT mentorship, skills
development, and responsiveness on its adoption as a mentor for undergraduate computing
projects. Mentorship is crucial in student learning, especially in computing (Kuhail et al., 2023;
Gottipati et al., 2023). However, traditional human mentors face limitations in availability andexpertise (Eller et al., 2014). Chatbots like ChatGPT offer on-demand support, trained on vast
data for tailored responses, making them attractive mentors. Still, their effectiveness incomputing projects is unclear. Previous research focused on chatbots in education but not as
mentors in computing (Kuhail et al., 2023). Variables like ChatGPT mentorship, skills
development, and responsiveness in the context of computing projects need further
exploration.
This paper examines the relationship between ChatGPT mentorship and its adoption incomputing projects. Using the Partial Least Squares Structural Equation Modeling (PLS-SEM)
statistical analysis method, we investigate the connections between various factors related to
ChatGPT's use as a mentor (Hair et al., 2021. It explores how factors like ChatGPT's
effectiveness as a mentor, its usage in mentoring, and student skills development areinterconnected. Ultimately, a comprehensive model is developed to understand how thesefactors contribute to ChatGPT's effectiveness as a mentor in computing projects. This study's
implications are significant. It highlights chatbots' potential as mentors for computing projects,
guides the design of effective mentorship programs, and contributes to the understanding of
AI in education's benefits and limitations.

40

2. Related Work
Artificial Intelligence in Education (AIED) can enrich teaching and learning in higher education
by personalizing student learning, automating assessment tasks, and even providing aroundthe-clock access to learning resources In education, chatbots can be integrated into e-learning
platforms that use Natural Language Processing (NLP) to interact with students by interpreting
their questions and retrieving relevant information from the knowledge base module to assist
students with revision (Clarizia et al., 2018). Interfacing with such conversational chatbots canprovide students with the benefits of one-on-one mentoring, Active and frequent interaction
between the student and the instructor impacts the student’s academic performance (Kuhail
et al., 2023). Using chatbot software applications, institutions can scale one-on-one mentoring
so that students have continuous support and can receive feedback 24/7 Benefits of chatbots
for undergraduate students’ academic performance are demonstrated by Essel et al. (2022).
In our work, we adopt the approaches from recent studies and apply them to analyze the useof ChatGPT to mentor students while doing their computing projects.

3. Hypothesis Development
In this section, we present the hypothesis of the study by identifying the factors of mentorship.
Technical mentorship: ChatGPT provides technical guidance to the student, covering dataengineering, analysis, software architecture, coding standards, and best practices in softwaredevelopment (Castro et al., 2018). We assess the quality of advice provided by ChatGPT forcompleting technical tasks as a measure of technical mentorship.
Design mentorship: This construct involves ChatGPT's guidance in design-related areas(Castro et al., 2018). It includes advising on software modules, analytical techniques, data
sources, and data processing methods to achieve project objectives (Iacob & Faily, 2020). We
evaluate ChatGPT's effectiveness in helping students complete design-related tasks as ameasure of design mentorship.
Mentor responsiveness: This construct focuses on ChatGPT's prompt and effective responseto the student's needs and requests. Effective communication and understanding are crucial
for successful problem-solving (Yin & Qiu, 2021). To assess mentor responsiveness, wemeasure the speed and quality of ChatGPT's responses to the student's questions.
Skills development: This construct focuses on the specific skills and knowledge gained by thestudent through ChatGPT's mentorship. A mentor enhances the student's professional andpersonal development by transferring technical and soft skills. We assess skills development
by measuring the student's perception of acquired skills after the mentorship.
Future adoption: This construct relates to students' likelihood of using AI tools in futurecomputing projects. Positive experiences increase the likelihood of future use, whilechallenges may discourage it. To assess future adoption, we measure student satisfaction
and their intention to use ChatGPT in subsequent tasks.
We formulate below statistical hypothesis and test it with SMART-PLS 4 (Khoi & Van, 2018).
H1: ChatGPT technical mentorship has a positive impact on student technical skills development
H2: ChatGPT design mentorship has a positive impact on student soft skills development
H3: ChatGPT responsiveness has a positive impact on the future adoption of ChatGPT as mentor
H4: Technical skills development has a positive impact on the future adoption of ChatGPT as mentor
H5: Soft skills development has a positive impact on the future adoption of ChatGPT as mentor

41

4. Methodology
4.1 ParticipantsParticipants in this study were undergraduate students from a "Text Mining and Language
Processing" course at a university. The course lasted for 14 weeks, with a 12-week projectcomponent starting from week 3 and ending in week 14. The students worked in teams,
applying skills in designing a text analytics application using Python and Java libraries. They
were also required to demonstrate test cases during project presentations and complete atechnical report. A total of 64 students were recruited for the survey questionnaire, with 58
fully completed surveys used for the PLS-SEM study.

4.2 Data Collection and Analysis
The survey instrument comprises several sections, each targeting different aspects of
ChatGPT's mentorship. The technical mentorship section focuses on code-related solutions,while the design mentorship section examines project design approaches. The
responsiveness section gauges student satisfaction with ChatGPT's guidance, and the skills
development section assesses its impact on both technical and soft skills. Additionally, thesurvey explores students' future intentions regarding ChatGPT's use as a mentor. Finally, an
open-ended question aids the qualitative analysis of the research question. We use thestatistical software package SmartPLS 4 (Khoi & Van, 2018) for data analysis. The
significance of relationships among variables is tested using regression coefficients, Tstatistics, and P-values. The reliability and validity of the scale are tested by Cronbach's Alpha,Average Variance Extracted (AVE), and Composite Reliability (rho_c). A Cronbach's alphacoefficient greater than 0.6 ensures scale reliability (Hajjar, 2018). Rho_c should be better
than 0.6 and AVE must be greater than 0.5 (Hair et al., 2021).

5. PLS-SEM Model Results and Analysis
5.1 Model Construction
Table 1 presents the variables used in the study and their descriptions.
Table 1. Construction of the variables.
Construct Description
Variable
TUSE_1ChatGPT provided code without logical errors for my project.
TUSE_2ChatGPT provided code without missing steps for my project.
TUSE_3ChatGPT provided clear solutions to my questions.
DUSE_1ChatGPT supported me to find additional resources to support my project.
DUSE_2ChatGPT supported me to generate new ideas or approaches for my project.
MSUP_1ChatGPT responded to my questions and concerns during the project.
MSUP_2ChatGPT provided sufficient support and guidance for my project.
MSUP_3
The feedback and suggestions provided by ChatGPT were helpful for my project.
TSKILL_1 ChatGPT increased my confidence in coding.
TSKILL_2 ChatGPT increased my confidence in technical report writing skills.
TSKILL_3 ChatGPT increased my levels of problem-solving skills.
TSKILL_4 ChatGPT made me less dependent on faculty/peers for coding help.
SSKILL_1 ChatGPT increased my levels of participation in team discussions.
SSKILL_2 ChatGPT increased my levels of quality in work submission.
SSKILL_3 ChatGPT increased my levels of time management in work submission.
FUT_1
ChatGPT is helpful as a mentor for my computing project.
FUT_2
I would use ChatGPT as a mentor for a future computing project.

42

5.2 Model Validity Checks, Results, and Analysis
5.2.1 Convergent validity
Table 2 shows that composite reliability varies from 0.762 to 0.894, which is above thepreferred value of 0.5, demonstrating that the model is internally consistent. From Table 2, it
can be observed that all the factors are reliable (Cronbach's alpha > 0.64, AVE > 0.5) andhave composite reliability (rho_c, rho_a) greater than 0.65. Therefore, all the factors aresupported, and we can proceed with the model analysis in the next steps.
Table 2: Consistency and reliability results
Variable
Cronbach's
Composite
alphareliability
DUSE
0.674
0.869
FUT
0.644
0.67
MFB
0.69
0.716
SSKILL
0.666
0.96
TSKILL
0.707
0.727
TUSE
0.822
0.825

Compositereliability
0.848
0.847
0.829
0.84
0.812
0.894

AVE

Findings

0.738
0.735
0.62
0.728
0.526
0.738

Supported
Supported
Supported
Supported
Supported
Supported

According to Chin, 2009, the variables that have path coefficients less than or equal to 0.5
should be excluded. Therefore, we removed SSkill_3 (path coefficient of 0.410) from themodel. The final measurement model is depicted in Figure 1 (convergent validity is confirmed).

Figure 1. Measurement Model with variables whose path coefficient >0.5
5.2.2 Discriminant Validity
The AVE can be used to verify the discriminant validity (Fornell & Larcker, 1981). If the squared
correlation coefficients among the latent variables are less than the AVE values of therespective latent variables, then the discriminant validity is confirmed. Table 3 presents thedata, and we observe that the correlation coefficients are less than the AVE indicating thatvariables are distinct and unrelated to each other, providing evidence for discriminant validity.

43

Table 3. Correlation coefficients among the variables.
Variable
DUSE
FUT
MFB
SSKILLTSKILL
TUSE

DUSE
1
0.576
0.536
0.357
0.133
0.301

FUT

MFB

SSKILLTSKILL

TUSE

1
0.605
0.239
0.162
0.273

1
0.091
-0.01
0.189

1
0.558
0.123

1
0.395

1

AVE
0.738
0.735
0.62
0.728
0.526
0.738

By comparing correlation coefficients among the latent variables squared with the AVE
indicators, we confirm the discriminant validity of the model.
5.2.3 Discussion and Analysis of Results
Table 4 displays the results of the research hypotheses after a bootstrap test is performed.
We conclude that H1, H2, and H3 are valid hypotheses, while H4 and H5 are not.
Table 4: Hypotheses testing results
Hypothesis

Constructs

PathStandard
deviation

T-stats P-value

(H1)
(H2)
(H3)
(H4)
(H5)

TUSE -> TSKILL
DUSE -> SSKILL
MFB -> FUTTSKILL -> FUTSSKILL -> FUT

0.395
0.357
0.593
0.094
0.132

0.11
0.112
0.096
0.165
0.143

3.573
3.172
6.172
0.571
0.927

0.000
0.002
0.000
0.568
0.354

Validated

Yes
Yes
Yes
No
No

The results indicate that both design mentorship and technical mentorship have positive
impacts on the development of soft skills and technical skills, respectively. Additionally, mentor
responsiveness has a significant positive impact on the effectiveness of ChatGPT adoption asa mentor for future computing projects.
5.2.4 Qualitative Analysis
The open-ended question reveals insights into why H4 and H5 hypotheses are not validated.
Student feedback on ChatGPT shows two key observations. First, many students find
ChatGPT helpful for the technical and design aspects of their projects. Some even compare it
favorably to existing resources, citing more accurate answers. ChatGPT aids withbrainstorming and analysis options, positively impacting work quality. Second, some students
express caution about using ChatGPT. They mention unreliability in coding and faulty code
returns. Students emphasize the need to cross-reference ChatGPT's responses to ensure
reliability. Despite skill improvement, code errors affect technical mentorship quality. The
comments demonstrate that ChatGPT has helped students to progress in their work andimprove their work quality. However, caution needs to be exercised while using ChatGPT as
some students have pointed out its limitations, particularly in areas such as coding, where it
can be highly inaccurate and return faulty code. Future research will explore other factors suchas ease of use, trust, and ethics to better understand ChatGPT's impact on various computing
activities. Expanding the sample size to include multiple computing courses would offer a morecomprehensive understanding of ChatGPT adoption in diverse contexts. Techniques likefocus group interviews and longitudinal studies could further enhance data collection.

6. Conclusion
Our research studies ChatGPT's impact as a mentor in undergraduate computing projects.
Mentor responsiveness emerges as a crucial factor influencing the future adoption of AI tools.
Providing responsive mentorship with technical and design support is key to AI tool adoption.

44

Our study offers valuable insights into factors influencing ChatGPT's adoption as a mentor
and provides practical implications for educators integrating AI into computing projects.

References
Castro, F., Tedesco, P., Alves, H., Quintino, J. P., Steffen, J., Oliveira, F., . . . da Silva, F. Q. B. (2018).
Developing a Corporate Chatbot for a Customer Engagement Program: A Roadmap. ICIC 2018
Chin, W. W. (2009). How to write up and report PLS analyses. In Handbook of partial least squares:
Concepts, methods and applications (pp. 655-690). Berlin, Heidelberg: Springer Berlin Heidelberg.
Clarizia, F., Colace, F., Lombardi, M., Pascale, F., & Santaniello, D. (2018). Chatbot: An Education
Support System for Student. Cyberspace Safety and Security, 10th CCS 11161, 291–302.
Eller, L. S., Lev, E. L., & Feurer, A. (2014). Key components of an effective mentoring relationship: A
qualitative study. Nurse education today, 34(5), 815-820.
Essel, H. B., Vlachopoulos, D., Tachie-Menson, A., Johnson, E. E., & Baah, P. K. (2022). The impactof a virtual teaching assistant (chatbot) on students' learning in Ghanaian higher education.
International Journal of Educational Technology in Higher Education, 19(1), 1-19.
Fornell, C., & Larcker, D. F. (1981). Structural equation models with unobservable variables andmeasurement error: Algebra and statistics.
Gottipati, S., Shim, K. J., & Shankararaman, V. (2023). AI for Connectivism Learning-Undergraduate
Students’ Experiences of ChatGPT in Advanced Programming Courses.
Hair Jr, J. F., Hult, G. T. M., Ringle, C. M., Sarstedt, M., Danks, N. P., Ray, S., ... & Ray, S. (2021). An
introduction to structural equation modeling. Partial Least Squares Structural Equation Modeling(PLS-SEM) Using R: A Workbook, 1-29.
Hajjar, S. T. (2018). Statistical analysis: internal-consistency reliability and construct validity.
International Journal of Quantitative and Qualitative Research Methods, 6(1), 27-38.
Iacob, C., & Faily, S. (2020, February). The impact of undergraduate mentorship on student satisfactionand engagement, teamwork performance, and team dysfunction in a software engineering groupproject. In 51st ACM Technical Symposium on Computer Science Education (pp. 128-134).
Karandish, D. (2021): 7 Benefits of AI in Education The Journal, Transforming Education throughTechnology,https://thejournal.com/articles/2021/06/23/7-benefits-of-ai-in-education.aspx (29 April
2023).
Khoi, B. H., & Van Tuan, N. (2018). Using SmartPLS 3.0 to analyse internet service quality in Vietnam.
In Econometrics for Financial Applications (pp. 430-439). Springer International Publishing.
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A
systematic review. Education and Information Technologies, 28(1), 973-1018.
Mhlanga, D. (2023). Open AI in education, the responsible and ethical use of ChatGPT towards lifelong
learning. Education, the Responsible and Ethical Use of ChatGPT Towards Lifelong Learning.
Sok, S., & Heng, K. (2023). ChatGPT for education and research: A review of benefits and risks.
Available at SSRN 4378735..
Yin, J., & Qiu, X. (2021). AI technology and online purchase intention: Structural equation model basedon perceived value. Sustainability, 13(10), 5671.

45               ! 
! " #  $ ! !          
    !" #!$         
   !$ %                                                    !"#          $            $    %    $"  &            "    $      $ '  $                 &                        $     $"      


% "  &( " $)&  &'   
*                                $       "$ $   
  +            "        $          , $                                          -                !"# $.
 !/012#         !(--# %  3  !/015#    "      $!&("#
       &             , '
    $ ' &("$ $   "  $ $      6$    !6"#&                                      "   $     
       !7 /018#  
    !9  /01:# 3 ; '<" '  '   =                       "  &("   .   , * $!.,*# 3                  " $              &                                        46          $         3  $                
3     '            +    >    3 $ '            (' )*       *, $      ?   
          ,  
     $   
!  '  #   ∈       ∈     3$       
  !7#  $   
  $  $        ' 
    @                    $   3              + %      &      3        
 



    *      '   !"#$      '                             $    $       A  $ " $    $        !   #  $+             $"                   
    
      A   !/01B#  "     $                              
   &   " . !/012# $        "$                      $                      3           $           $    '                              &                                     $    
"       "     & 
             "& ")7 '  "A  !/01B#                            $                       
"$   "    >?  !/01C#    "         D                            $     $     $       47

A    $                        !. $  > $/01CDE? >%  /0/0# )   
              #     +#, -'&    $ $ ' F$     " $   $    !  #+ 
          $
                       "  $        +      $     &  
  $   $       '       !A?G#              "'$3  !/015#      &(" " A' 
  !A?G#                &("         '      $  '$   "$        G!* /01C#  $ A&("
. /   0  !     $    ' )      $           *   ,  A  !*,A#     .     ,  * $!.,*# $.  7 >  !/00C# "$                 
           !       '  $  # *                  *)H% $7
 !/01C#             & .,*        
      +               $       .          #'9>!/01C#  
               &   $'     &(".,*A&(" "A           &     E  !/0/0# 
               "1' 2  % 34  
3                  +                                               
    D    $ "$  $   '  
       
 +    48

&      $  & '                     $
    .             '          &       $    
    !  "#  $       %& "      $ &      $       '        (--          "+      %                 &              
&   7                    ! '  # (      )   "                     
    &        +
       !  
 

#   ! *
&$ & &(" A&("  
  &  &  
! #   !
#  
1 .,*! + % $ "
* 'A&("   !$                 $#               !$  
     #     !  #+"  "$ $            $  &    $             !     $  "#     "  $         
 '  +-    '   
 
 
+ $  & A&("  & " "
$"    $ A&(" (-    $      )
, 1   & E  !/0/0#              "  

49

 $ "$$ A&("
!! + % %%   $ "  "  $       $ 
$(-- "        &         
.,* $      "$  
        $ $ +         & ' '         "   $ 
    , '  1 !  # &(" .,*   +           3    ; <  *     
   "     "
        9>!/01C#!! ,( 3      +            "                                          $    =                   $                    
       
3                  $  !*77#              
!*@7#

50

5' 6 
" '   E%++II I

I - .(3          *   (--   $!             E             !     '  3    E(@  !E(  @ #    ?)"A!? ) " A $#     !7  /012# &       ;  <  $           $    ;<  $   D     G'       
"     +"        (--                    "   ;<"      ;<"    .,*                               D       ;< 
 1        ; <  !.,*#     ; <              !?(#  - +
3          
0 ' 2H5          /0        ' 
  B7      * $ "   7
  !7 /010# "  $   
'$   *   7 *        1/0! (778' H85B50      8/1:        /55BB    
  1/H7!,  /01C# )  78)  1 1HB/' ) * '25//011:H0   /H87)  A==7    $  6 $  &        /C             7  G             $  )       /     10/0  '   "  -! +   32   + 2  $                     &            $     $ $         $) ' 3      $       $  ) 
"9/:  '3    $ 
      %    $   
     )   
    ;  9' =        "            ,     $              $ 

51     $   $ * $    '6.""+    !3  1CC0# " 
   '

-- (    (3  * ! >6/018#   ' $ "'!      # 3   100   100 , (--E(@ $      $/ 20 =    .$"  3  ,     '      7.@       *      6 $  
'   E.@ "    /00

<' ,  
(   "1H    
"1 /(    +
A
G 
2 
E(@
?(
- 
.,*
- 
"
E(@
"
E(@








*77
7'==7
0 B2H
0 B28
0 ::/
0 :51

*@7
7'855
0 C1B
0 C1:
0 B88
0 BHC

"/ /( 0  +
A
G 
;,
- 
= 
E(@
"
E(@
.,*
- 







*77
7'>&5
0 :11
0 5C1
0 5B/

*@7
7'>5=
0 :/5
0 :01
0 5B5

"H /( 1 )(+
A
G 
2 
E(@
;,
- 
"
E(@
.,*
- 







*77
7'>7>
7'>75
0 5B8
0 5H0

*@7
7'>>=
7'>><
0 :21
0 5BH

,       " A  $                        "    !E(@#$  $   $        ! #      6                   $=  *  6 $           &     $  
        3      
 ,             &$  +  $  $/0 )               
7       $   ,  $       $$ 52 77     %   *              .,* ?(!0 05    *@7# ) 6 $!0 0C    *@7# 7 $  

?' @ 
A               
        &    $                   $         $    $   &               "                 " $                $  $ 
     &   
   $*    $      , '
 )?*A!) /01B#                                            "  !A   /01H# @      $                      *       $                  $          $            >' !* 

3 )E      "   $J)"
7(G)"E  - J.AJ7(/11,/ 
*)   % )) >E  !/01C# *  G?=  
  $ ,+.23'  $%    (4 %  ,  (+.
$//BK/HH
7 A L  6   E 7 6  6 , ) % >6 N!/018#  ?               (--    K          '  $ %    2-  4 %      , (  .   ( 5 $$'  $6,. 5'71:/8K1:H8 +II  I10 H112I 1I1811:C
7 6 . , 6 N >9J J !/01C# *)H%+A ) ?   ,      =$  )      .     )  '  $%   &(%   (  4 %      ,  (  +  .
$  6,+.  237  /CKHB+II '  II1C02 05B:H
7 J EOP R 7 >6 N !/012# E        & , ( 6> A 6!G #'  $%!     ( %    ($  4.5  28  ( ((  %    9  :(  28 !9  H:    /05:K/0:2#  JA?(  +II     I H:I 12 7 ? " !/010# =  $    +"&-*   S ' 0( '($ (.   T >?  G 7 !/01C# 3$      & A
7     7  ,  ?$   *  A     >  (  -   !G # '  $  %       (   %        (    $  ,+.  23    ;(     :(  8
23 &   G A  )$!&GA)#

53, A %  - >   !/00C# *       
$   .( $  0    3!H#/8HK/55
E "    )  J A"   !/0/0# 3       U,+.= (%,  (+.
$!H#H1K28
E  *   %    -   >  ?   *  )  !/0/0#  7 '           '  $ %    9  04.   >++    (  4 %      &($  +   ?
+.
$/HH0K/HHC  . >6J !/018# *+*  '  $%!    (4 %   5
$/    6 45/7
A  " )   & 7  7 E ) > J !/01H#                     $ 0     (  %    '  $H111KH11C
A ) NN  A 7 V, >9J J !/01B#     $        2@ ,,,    (4 %   +.
$6 4+.711B/K11B:
A  ) *  * $) A 6 >A A !/01B#         U'  $%,(     (4 %   ,  (+.
$,  (+.
$ ' . $) > $E !/01C# *           (
4 %   ,  (+.
$ ,+.23HB8KHBC.  . & 7 % >   ( !/00C# .     $K      '  $%2234 %   0  %  ( (( $  ,  #1 ( $5$4 #  &($/    0%% .(( $2H1K2HB.7 6 J % J E ) )A E? J >)  J !/012#
    0    ( %   '  $6 '7202K21H)  6   6   7   E   G   %   A   >  A    -  !/01B#  )    
   '  $ %      A )        % 5' %
1 ( $,  (0(   25K52
9   *   )   -   .    -   @    J   J   ?   E  *  -      W   >.  & !/01:# *  $  & & E$ @ 9 ?' ) 6 % 3( , ) 9  >( E !G # 0    ( %     $
!9

H0#

7 

*& +II     I XI I/01:IIH2/8H28:C102H18B82. 9  J J   >    %  !/01C#     "    A +  ,   A    "   '  $%!!000 4 %   0  %  ( (( $ :20K:2:
3   .  J    !1CC0#  6       +  3               
'  $% ,,,B@!10#1220K1250
3  %   N % 6 >G  7 !/015# 6 +6$ '  
&("         $ '  $%3    (
4 %   ,  (+.
$6,+.72HCK288

54

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in Education.
Asia-Pacific Society for Computers in Education

Immediate Feedback in Computational
Thinking: Generating hints using a
Knowledge Graph
Nitesh Kumar JHAa, Plaban Kumar BHOWMIKb & Kaushal Kumar BHAGATa*aAdvanced Technology and Development Centre, Indian Institute of Technology Kharagpur, India
bG.S Sanyal School of Telecommunications Indian Institute of Technology Kharagpur, India*kkntnu@hotmail.com
Abstract: In this study, we present an Online Inquiry-based Learning Platform for
Computational Thinking (CT-ONLINQ) to develop the CT skills of students. The
platform provides immediate feedback with hints to support students during
problem-solving activities and encourages them to explore multiple solutions for a
problem. The hints are generated using a Knowledge Graph that stores information
about the solution details of a problem. A six-week study was conducted on 79 highschool students to determine the effectiveness of the platform with hints on students'
CT skills. The results showed that the students improved their CT skills significantly
after the intervention. Findings suggest that the CT-ONLINQ platform consisting of
IBL-based CT activities with immediate feedback could help school students improvetheir CT skills.
Keywords: Computational Thinking, Immediate feedback, Knowledge Graph,
Inquiry-based learning

1. Introduction
Computational Thinking (CT) is a cognitive ability that allows individuals to develop
computational solutions for a variety of problems (Wing, 2006). Inquiry-Based Learning (IBL)
is an approach to learning that guides students through various phases, starting with exploringa problem, followed by collecting data to develop models, generating findings, and rigorously
evaluating those findings to gain new understanding (Pedaste et al., 2015). In terms of thelearning process, IBL in science and CT education are relatively similar due to the cyclic
process of repeated revisions and refinement (Hoppe and Werneburg, 2019).
The current CT research focuses on teaching coding concepts to students (Jacob etal., 2020), but researchers argue that problem-solving, creativity, and algorithmic thinking
should be taught, as these transferable skills hold greater value (García Peñalvo et al., 2016).
In addition, studies on IBL in CT concentrate on creating logical artifacts and testing them toimprove CT abilities (Hoppe and Werneburg, 2019). Moreover, researchers argue thatanalysis and evaluation of artefacts is critical to improving learning outcomes (Prayogi and
Yuanita, 2018). To overcome the above issues, we have created CT-ONLINQ, an online
platform for Computational Thinking (CT) that follows IBL-based CT activities (Jha et al.,2023). The platform provides hints and prompts during problem-solving activities, which are
stored in a Knowledge Graph (KG). In KG, concepts are represented as nodes, andrelationships between concepts are represented as edges. Different forms of immediate
feedback have been developed by researchers to support coding and CT activities (Basu etal., 2017). However, there are lack of studies that provides hints stored in KG during
IBL-based CT activities. The current study discusses the design of KG and evaluates theeffect of the platform CT-ONLINQ on CT skills of high school students. The current study
addresses the following research question (RQ):x RQ: Did the online platform CT-ONLINQ with immediate feedback improved CT skills ofhigh school students?

55

2. The CT-ONLINQ platform
CT-ONLINQ is an online educational platform that supports IBL-based CT activities (Jha et al.,2023). A brief description of the platform architecture and example interface tab is shown in
Jha et al. (2023). The platform uses a KG to store hints and answers during IBL-based CT
activities. Figure 1 shows a basic schema structure of the KG with properties for eachproblem. The KG schema is divided into three subschemas: Answer comparison, Goal, Hints
and Answers, as described below.
zAnswer comparison: It contains problem’s question and the sequence of goals related to
CT step.
z
Goal: It contains step name, task statement, the best solution, hints and answers
sequence related to the task, and a no matched answer section that contains hints when
input does not match with any correct or incorrect answer.
z
Hints and Answers: It contains type of answer (correct/incorrect) that matches the input,
an alternate answer to compare with the input, a prompt for the hint, and three types ofproblem-specific hints: basic (with little elaboration), intermediate (with more elaboration),and alt hint (for alternate correct answer). The alternate correct answer (only available forthe correct answer type) is a solution but not the best solution for the task.

Figure 1. Schema of the Knowledge Graph
The platform parses the graph to provide relevant hints based on the number of attempts
made during problem-solving activities. For example, Consider the problem statement ‘Writean algorithm to find the sum of numbers 1+2+…+60’. For an incorrect answer indecomposition step- ‘Find a way to add number from 1 to 60’ the hints are: basic(attempts<4)can you do better? Add number paiwise to find total sum; intermediate (attempts<6)- can you
do better? Find pairs having sum 61; Advanced- (1+60)+(2+59)+…(30+31).

3. Methodology
The experiment involved 79 high school students, aged 13-15, from a public high school in
India. All the students were given immediate feedback with hints during problem-solving
activities. We adapted two different sets of 12 questions (max score 108, pre and post-test)from Bebras Thinking Challenge (Bebras-Ireland, 2020). Initially, the students signed up to the
CT-ONLINQ platform, completed an example activity (see Jha et al., 2023), and completed
bebras questions (pre-test with Cronbach’s alpha 0.74). Next, the students completed six

56

problem-solving activities on the platform and completed post-test with Cronbach’s alpha
0.77.

4. Result and Discussion
The average pre-test score was 42.11 with a standard deviation (SD) of 10.69, while thepost-test score was 55.43 with a SD of 14.07. We conducted a paired samples t-test tocompare the pre and post-test scores to assess the improvement in CT skills. Findings
showed that there was a significant difference in CT skills (t=6.697, p=.000, d=1.06), with anaverage difference of 13.316 (SD=1.988). The result supports the study that useful feedback
and hints improved student understanding in a CT-based learning environment (Basu et al.,2017).

5. Conclusion and future work
The present study investigated the effect of CT-ONLINQ with immediate feedback on the CT
skills of high school students. A total of 79 students from high school participated in the current
study. The platform implemented the IBL-based CT workflow, allowing students to develop
algorithmic solutions, analyze the algorithms, and compare the algorithms. The platform
provided immediate feedback at each CT step. Hints and answers were stored in the KG. The
findings confirmed that the intervention significantly improved the CT skills of students working
on the platform.
With regards to future work, the current study examined the effect of IBL-based CT
activities on the online platform but did not compare them with offline IBL-based CT activities.
Future studies can compare the effects of offline versus online IBL-based CT activities on theenhancement of CT skills. The study provides a new way to teach CT skills to high school
students using IBL-based CT activities without any coding practice.

References
Basu, S., Biswas, G., & Kinnebrew, J. S. Learner modeling for adaptive scaffolding in a computational
thinking-based science learning environment. User Modeling and User-Adapted Interaction 27, no. 1(2017): 5-53.
Bebras-Ireland (2020). Online reference included in article [Internet document] URL
https://bebras.techweek.ie/ Accessed 1/04/2023.
García Peñalvo, F. J., Reimann, D., Tuul, M., Rees, A., & Jormanainen, I. (2016). An overview of themost relevant literature on coding and computational thinking with emphasis on the relevant issues
for teachers. Belgium. doi:10.5281/zenodo.165123
Hoppe, H. U., & Werneburg, S. (2019). Computational thinking—More than a variant of scientific inquiry.
Computational thinking education, 13-30.
Jacob, S., Nguyen, H., Garcia, L., Richardson, D., & Warschauer, M. (2020, March). Teachingcomputational thinking to multilingual students through inquiry-based learning. In 2020 Research on
Equity and Sustained Participation in Engineering, Computing, and Technology (RESPECT) (Vol. 1,
pp. 1-8). IEEE.
Jha, N.K., Bhowmik, P.K., & Bhagat, K. K. (2023, July). Usability Evaluation of an Online Inquiry-based
Learning Platform for Computational Thinking (CT-ONLINQ). In 2023 International Conference on
Advanced
Learning
Technologies(ICALT)(pp.
182-186).
IEEE.
https://doi.org/10.1109/ICALT58122.2023.00059.
Pedaste, M., Mäeots, M., Siiman, L. A., De Jong, T., Van Riesen, S. A., Kamp, E. T., & Tsourlidaki, E.
(2015). Phases of inquiry-based learning: Definitions and the inquiry cycle. Educational researchreview, 14, 47-61.
Prayogi, S., & Yuanita, L. (2018). Critical-inquiry-based-learning: Model of learning to promote criticalthinking ability of pre-service teachers. In Journal of Physics: Conference Series (Vol. 947, No. 1, p.
012013). IOP Publishing.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35.

57&%  $"!%"& !&$!&"!"!$!"!" #'&$%!
'&"!%"&)"$" #'&$%!'&"!   
           
2)!--23)343%.&!-!'%,%-3()+)//)-%2
A65B5K5B5=A 98I-., . '52 B@2 <3 05.A/<A@ 5.@ /20<:2 6;0?2.@6;49F =<=B9.? 6; A52 36291 <3
21B0.A6<;2;02A52>B.96AF<3A52=?<:=A@B@21/F05.A/<A@0.;4?2.A9F6;39B2;02A52
>B.96AF <3 A52 42;2?.A21 ?2@=<;@2@ D5605 6; AB?; 0<;A?6/BA2@ A< 12@6?./92 <BA0<:2@
?29.A21A<A52<C2?.9992.?;6;42E=2?62;02<D2C2?2C2;.@8;<D;0?6A2?6.3<?2336062;A=?<:=A@.?2=?2C.92;AA52?26@.12.?A5<3:2.@B?./92.;10<;0?2A296;4B6@A603.0A<?@
A5.A4B612=?<:=A2;46;22?6;4'56@=.=2?=?2@2;A@.0<:=<@6A2@0<?23<?5.A#'=?<:=A@D56050.;/2:.12.==960./92A<<A52?3<B;1.A6<;.942;2?.A6C205.A/<A@
'5?<B45.0<:=BA.A6<;.996;4B6@A60.;.9F@6@<38;<D;2336062;A=?<:=A@B@216;92.?;6;4
2:2?42;A 96;4B6@A60 3.0A<?@ =<6;A A< A52 ?29.A6<;@56= <3 96;4B6@A60 32.AB?2@ .;1 A520<;3612;02 <3 5.A#' ?2@=<;@2@ A< D299@A?B0AB?21 =?<:=A@ A5.A B@2 A52 @.61
96;4B6@A60 32.AB?2@ '52 96;4B6@A60 32.AB?2@ .?2 A52 .C2?.42 0<99<@A?B0AB?.9 @A?2;4A5
0<99<@A?B0AB?.9?.A6<16C2?@6AF@=2063606AF.;1.0.12:609.;4B.42B@2'52@232.AB?2@
12=60AA52>B.96AF<3=?<:=A@A5.A=2?A.6;A<A524?.::.A60.9@A?B0AB?2@=2063606AF.;1
?292C.;02 A< A52 A.@8 .A 5.;1 .;1 .0.12:60 9.;4B.42 B@2 B?A52? A52@2 32.AB?2@
0<;@A6ABA2 . 0<:=<@6A2 @0<?2 3<? =?<:=A@ 6;A?<1B021 6; A56@ @AB1F A5.A ?2=?2@2;A
96;4B6@A602336062;0F.;1@B/@2>B2;A9F0<??29.A2@A<=2?=92E6AF<?02?A.6;AF2@A6:.A2@<3
A5242;2?.A21?2@=<;@2@<35.A#'"21*,!-5.A#'#!#0<:=BA.A6<;.996;4B6@A60@=?<:=A2;46;22?6;4  
CBJ9FG5H=CB5@5;9BHGGI7<5G7<5H6CHG5F9F5D=8@M9A9F;=B;5G5DFCA=G=B;H97<BC@C;M:CF
98I75H=CB5B8@95FB=B; 2=H<H<9<9@DC:58J5B798@5B;I5;9AC89@G 7<5H6CHG75B:57=@=H5H9
D9FGCB5@=N98@95FB=B;@@9B 9H5@  G<C? 9H5@  %C@CH9G7I   DFCJ=89
=BGH5BH:998657?$F59GG9F  )7 5FH<M 9H5@   5B89B;5;9GHI89BHG=B5B
=BH9F57H=J95B89B;5;=B;A5BB9F(=B <5B;    %CK9J9F H<99::97H=J9B9GGC:H<9G9
7<5H6CHG GD97=:=75@@M;9B9F5H=J9& 65G987<5H6CHG <95J=@MF9@=9GCBH<9EI5@=HMC:DFCADHG
H<5HH<9MIG9HC9@=7=HF9GDCBG9G:FCAH<9&@5F;9@5B;I5;9AC89@G(()G!5B; 9H5@   
/<97<5@@9B;9@=9G=B9B;=B99F=B;DFCADHGH<5H75B;9B9F5H9577IF5H95B8
7CADF9<9BG=J9F9GDCBG9G:FCA(()G 2<=@9A5BM9L=GH=B;DFCADHG5F99::97H=J9 H<9MC:H9B
F9@MCB8CA5=B GD97=:=7?BCK@98;9 K<=7<@=A=HGH<9=F;9B9F5@=N56=@=HM57FCGG8=::9F9BH
98I75H=CB5@G9HH=B;G )CF9CJ9F H<9F9=G5@57?C:IB89FGH5B8=B;C:H<9@=B;I=GH=7:95HIF9GH<5H
7CBHF=6IH9HCH<99::97H=J9B9GGC:DFCADHG K<=7<A5?9G=H8=::=7I@HHC89J9@CD9::97H=J9
DFCADHGGMGH9A5H=75@@M /<97IFF9BHGH5H9C:DFCADH9B;=B99F=B;:CF98I75H=CB5@7<5H6CHG=G@5F;9@M
7<5F57H9F=N986M5HF=5@ 5B8 9FFCF5DDFC57< K<9F989J9@CD9FGC:H9BF9@MCB58 <C7GHF5H9;=9GHC;9B9F5H9DFCADHG /<=G5DDFC57<=GBCHCB@MH=A9 7CBGIA=B;5B8F9GCIF79=BH9BG=J96IH5@GC@=A=HGH<99::97H=J9B9GGC:H<97<5H6CHG=B:57=@=H5H=B;@95FB=B; 58

/C588F9GGH<9G97<5@@9B;9G H<=GD5D9FDF9G9BHG57CADIH5H=CB5@@=B;I=GH=75B5@MG=G
C:DFCADHGK<=7<75B69@9J9F5;98=B9B<5B7=B;H<9@95FB=B;9LD9F=9B79 M@9J9F5;=B;
7CADIH5H=CB5@A9H<C8G H<=GKCF?5HH9ADHGHCEI5BH=:MDFCADH9::=7=9B7M    ! 

$9B9F5H=J9@5B;I5;9AC89@G<5J9F9JC@IH=CB=N98H<9:=9@8C:B5HIF5@@5B;I5;9DFC79GG=B;
*(, 9B56@=B;H<97F95H=CBC:7<5H6CHGH<5H75B;9B9F5H9<IA5B @=?9F9GDCBG9GHCIG9F
=BDIHG ACB;H<9G9AC89@G $,/$9B9F5H=J9,F9 HF5=B98/F5BG:CFA9FGH5B8GCIH5GCB9
C:H<9ACGHK=89@MIG985B89::97H=J9AC89@G:CF;9B9F5H=B;H9LH /<9$,/AC89@=G65G98CBH<9HF5BG:CFA9F5F7<=H97HIF9 5899DB9IF5@B9HKCF?H<5H
=G75D56@9C:DFC79GG=B;=BDIHG9EI9B79GC:J5F=56@9@9B;H<5B8;9B9F5H=B;CIHDIH
G9EI9B79GC:J5F=56@9@9B;H< /<9AC89@=GHF5=B98CB5@5F;97CFDIGC:H9LH85H5 5B8=HG
D5F5A9H9FG5F9CDH=A=N98HCA=B=A=N9H<9DF98=7H=CB9FFCFC:H<9B9LHHC?9B=BH<9G9EI9B79
;=J9BH<9DF9J=CIGHC?9BG +B9C:H<9ACGHK9@@ ?BCKB=AD@9A9BH5H=CBGC:H<9$,/AC89@=G<5H$,/ 57CBJ9FG5H=CB5@5;9BHH<5H75B;9B9F5H9<IA5B @=?9F9GDCBG9GHCIG9F=BDIHG <5H$,/=G65G98CBH<9$,/  $,/   5B8BCK$,/ 5F7<=H97HIF9 K<=7<=G7IFF9BH@M
CB9C:H<9@5F;9GH5B8ACGHDCK9F:I@AC89@G5J5=@56@9:CF;9B9F5H=B;H9LH &BH<9:=9@8C:98I75H=CB  <5H$,/5B8CH<9F;9B9F5H=J9&HCC@G<5J9BIA9FCIG
5DD@=75H=CBG +B9C:H<9ACGHDFCA=G=B;5DD@=75H=CBG=G=BH<95F95C:D9FGCB5@=N98@95FB=B; <5H6CHG75B69IG98HC89@=J9F7IGHCA=N98@95FB=B;9LD9F=9B79G 585DH=B;HCH<9B998G5B8
DF9:9F9B79GC:=B8=J=8I5@@95FB9FG /<9M75B5@GCDFCJ=89=BGH5BH:998657?5B8GIDDCFH <9@D=B;@95FB9FGHCCJ9F7CA9C6GH57@9G5B8GH5MACH=J5H98 BCH<9F5DD@=75H=CBC:;9B9F5H=J9&HCC@G=B98I75H=CB=G=BH<97F95H=CBC:
98I75H=CB5@7CBH9BH  <5H6CHG75B;9B9F5H9H9LH 65G987CBH9BHGI7<5GGIAA5F=9G GHI8M
;I=89G 5B8DF57H=79EI9GH=CBG K<=7<75B69IG98HCGIDD@9A9BHHF58=H=CB5@98I75H=CB5@
A5H9F=5@G 88=H=CB5@@M H<9M75B69IG98HC7F95H9=BH9F57H=J9@95FB=B;9LD9F=9B79G GI7<5G
G=AI@5H=CBG5B8;5A9G H<5H9B;5;9@95FB9FG=B5:IB5B8=AA9FG=J9K5M <5H$,/ 5GK9@@5GCH<9F;9B9F5H=J9&HCC@G <5GBCK697CA9DF9J5@9BH=B98I75H=CB
5B857FCGGA5BM8CA5=BG5B8=B8IGHF=9G /<9G9HCC@G<5J9H<9DCH9BH=5@HCF9JC@IH=CB=N9H<9:=9@8C:98I75H=CB6M9B56@=B;D9FGCB5@=N98@95FB=B; 7F95H=B;=BH9F57H=J97CBH9BH 5B8
DFCJ=8=B;=BGH5BH:998657?5B8GIDDCFH  #!"%  # !#&%! "::97H=J9DFCADH9B;=B99F=B;=G7F=H=75@:CF;9B9F5H=B;577IF5H95B87CADF9<9BG=J9
F9GDCBG9G:FCA(()G=BJ5F=CIG8CA5=BG ACGH9GD97=5@@M=B98I75H=CB5@G9HH=B;G ,FCADH
9B;=B99F=B;=BJC@J9G7F5:H=B;K9@@ 89G=;B98DFCADHGH<5H7@95F@M5B87CB7=G9@M7CAAIB=75H9
H<9=BH9B898@95FB=B;C6>97H=J9G5B889G=F98CIH7CA9G %CK9J9F DFCADH9B;=B99F=B;=GBCH
5GHF5=;<H:CFK5F8DFC79GG 5B8C:H9BF9EI=F9G=H9F5H=CBGC:HF=5@5B89FFCF :998657? 5B8
657? 5B8 :CFH<7CBJ9FG5H=CBGK=H< <5H$,/5G588=H=CB5@=B:CFA5H=CB=GF9EI=F98HC57<=9J9
H<989G=F98=B:CFA5H=CB=B;9B9F5H98F9GDCBG9G M=BJ9GH=B;=BDFCADH9B;=B99F=B;5B8
7F95H=B;9::97H=J9DFCADHG 98I75H=CB5@=BGH=HIH=CBG75B<5FB9GGH<9DCK9FC: <5H$,/:CF
A5BM5DD@=75H=CBG=B7@I8=B;HC9B<5B79GHI89BH@95FB=B;5B85IHCA5H9H=A9 7CBGIA=B;
H5G?G GI7<5G;F58=B;5B8:998657? HCB5A95:9K %CK9J9F =HG=ADCFH5BHHCBCH9H<5H
DFCADH9B;=B99F=B;F9EI=F9G75F9:I@D@5BB=B;5B8CB;C=B;F9:=B9A9BHHC9BGIF9H<5HH<9
DFCADHG5F99::97H=J95B85@=;B98K=H<H<9=BH9B898@95FB=B;CIH7CA9G GGI7< 7CB7F9H9
A95GIF9GC:K<5H7CBGH=HIH99::=7=9BHDFCADHGG<CI@869=BJ9GH=;5H985B8DIH=BD@579 +B;C=B;KCF?F9@5H98HCDFCADH9B;=B99F=B;C:@5F;9@5B;I5;9AC89@G(()G<5J9:CIB8
H<5H5BIA69FC:GHF5H9;=9G5::97HH<9EI5@=HMC:H<9;9B9F5H98CIH7CA9 9 ; F9D9H=H=CB 
DFCJ=8=B;9L5AD@9G IG=B;7C89 @=?9GHFI7HIF9G CF7<CCG=B;H<9F=;<H=BGHFI7H=CBG&&#% "#!" #$"'&#& 5945A:=F9G7I ,9F9=F5 9H5@   :C7IG=B;CBH<9KCF8CF89FC:DFCADHG35C(I   
IG=B;GHFI7HIF98DFCADHD5HH9FBGH<5H9LD@=7=H@M7CBH5=B9@9A9BHG@=?9=BH9BH 7CBH9LH 
ACH=J5H=CB2<=H9 9H5@   HCB5A95:9K %CK9J9F H<9F9=G@=HH@9KCF?CBDFC6=B;=BHC
H<9@=B;I=GH=7:95HIF9GH<5H7<5F57H9F=N9K9@@ GHFI7HIF985B89::=7=9BHDFCADHGH<5H;9B9F5H9
7CB:=89BHF9GDCBG9G:FCA(()G5B8CH<9F;9B9F5H=J9&AC89@G @=?9 <5H$,/  $#&$%! /<979BHF5@5=AC:H<9GHI8M=GHCG99?HC89J9@CD57CADF9<9BG=J9IB89FGH5B8=B;C:H<9@=B;I=GH=7:57HCFGH<5H7CBHF=6IH9HCH<99::97H=J9B9GGC:DFCADHG:CF(()GK<=7<75B69B9:=H
98I75H=CB5@G9HH=B;G /<9F9G95F7<EI9GH=CB;I=8=B;H<=GGHI8M=G2<5H5F9H<9@=B;I=GH=7
:95HIF9GC:9::=7=9BH <5H$,/DFCADHGM5BGK9F=B;H<=GEI9GH=CB H<9GHI8M5=AGHC
5FH=7I@5H9K<5HGD97=:=7@=B;I=GH=7A5F?9FGG<CI@869DF9G9BH=B:CFAI@5H=B;9::=7=9BHDFCADHG
K<=7<KCI@8=BHIFB;9B9F5H9ACF97CB:=89BHF9GDCBG9G:FCA(()G5B8A=B=A=N9H<99LH9BH
C:<5@@I7=B5H=CBG=BH<9;9B9F5H98F9GDCBG9G  
 #!"%!%! /<97CFDIGIG98=BH<=GGHI8MK5G7C@@97H98:FCADI6@=7@M5J5=@56@99::97H=J9 <5H$,/
DFCADHG /<9GCIF79G=B7@I899::97H=J9DFCADHG:CFH957<9FG:FCA
<HHDG KKK @95FBDFCADH CF; 7<5H ;DH DFCADHG :CF H957<9FG 5B8IG98=B7IFF=7I@IA
A5B5;9A9BH 7@5GGFCCAA5B5;9A9BH DFC:9GG=CB5@89J9@CDA9BH 9LD@5=B=B;8=::=7I@H
7CB79DHG5B8DFCJ=8=B;9L5AD@9G 5GK9@@5GCH<9FIG975G9G 88=H=CB5@9::97H=J9DFCADHG
K9F97C@@97H98:FCA<HHDG ;=H<I6 7CA : 5K9GCA9 7<5H;DH DFCADHGK<=7<=G5;=H<I6
F9DCG=HCFMC:9::97H=J9 <5H$,/DFCADHGK=H<DI@@F9EI9GHG  ?:CF?G 5B8=B7@I89G
  8CKB@C58G:FCA=HG<I;;=B;:579G=H95GC:)5F7<  HCH5@C: 9::97H=J9
<5H$,/DFCADHG,%!-  2$  K9F9;5H<9F98HC7CADF=G9H<=G
GHI8MPG7CFDIG   #!"% )'&%! #CFH<=GGHI8M DFCADH9::=7=9B7M=GCD9F5H=CB5@=N985GH<956=@=HMC:5DFCADHHC9@=7=H7C<9F9BH
5B87CB:=89BHF9GDCBG9GIGI5@@M7<5F57H9F=N986M<5J=B;@CKD9FD@9L=HMG7CF9G /<=GA9HF=7
=GIG9:I@K<9BH<9;C5@=GHC9J5@I5H9H<99::97H=J9B9GGC:DFCADHG=B;9B9F5H=B;<=;< EI5@=HM
F9GDCBG9G:FCA5B&;9B9F5H=J9AC89@ /<99::=7=9B7MCFEI5@=HMC:DFCADHGK5GD9F:CFA98
IG=B;H<9D=D9@=B9=B:=;IF9 F5B8CAG5AD@9C: DFCADHG:FCAH<97CFDIGK5GIG98=B
75@7I@5H=B;H<9D9FD@9L=HMG7CF9GC:;9B9F5H98F9GDCBG9G ,9FD@9L=HMG7CF9G5F9IG98HC
89H9FA=B9H<9EI5@=HMC:;9B9F5H98F9GDCBG9G;=J9BH<9DFCADHGH<5H5F9DFCJ=898HC5
;9B9F5H=J9&AC89@ /<9@CK9FH<9D9FD@9L=HMG7CF9 H<9ACF9@=?9@M=H=G;9B9F5H986M
<5H$,/)5@=B?5 9H5@  CF5B&;9B9F5H=J9AC89@ /<=GA9HF=7<5G699BIG98=B
KCF?G@=?989H9FA=B=B;=:G<CFHH9LHG5F9;9B9F5H986M5<IA5BCF6M <5H$,/3958CB 9H
5@  )=HFCJ=Q   0G=B;CD9B&PG$,/,& H<9F9G95F7<9F;9B9F5H98F9GDCBG9G
HCH<9G5AD@97CFDIGC: K9@@ KF=HH9BDFCADHGH<9B5DDFCL=A5H98H<9D9FD@9L=HMG7CF9GC:
H<9;9B9F5H985BGK9FGIG=B;%I;;=B;#579PG;DHAC89@G /<9F9GI@H=B;D9FD@9L=HMG7CF9GC:
H<9;9B9F5H98F9GI@HGF5B;98:FCA HC K=H<5A95BC: G8  /<9
F9@5H=J9@M@CK5DDFCL=A5H9D9FD@9L=HMG7CF9A95BGI;;9GHGH<5HH<9DFCADHGH<5HK9F9
F5B8CA@MG5AD@98:FCAH<97C@@97H987CFDCF5IG98=BH<9GHI8MK9F956@9HC9@=7=H;9B9F5H98
F9GDCBG9GH<5HK9F9F9@5H=J9@MACF979FH5=B /<=G7CI@8=AD@MH<5HH<9DFCADHG5F97@95F5B8
9::97H=J9 5GH<9$,/AC89@K5G56@9HC;9B9F5H97C<9F9BH5B8K9@@ :CFA98H9LH=BF9GDCBG9
HCH<9AK=H<F9@5H=J9@M<=;<79FH5=BHM5B8 K=H<DCGH <C7EI5@=H5H=J9=BGD97H=CB K9F9C6G9FJ98
HC69ACF9577IF5H9 &&#% '!"&%&%(%" &#&#$" #&%60)'41%  ,=D9@=B9:CF"J5@I5H=B;H<9"::=7=9B7MC:,FCADHG

  &$% ((%#%!
*5HIF5@@5B;I5;9DFC79GG=B;=GD9F:CFA98CBH<99BH=F97CFDIGC: DFCADHGG99:=;IF9 (=B;I=GH=7=B8=79GK9F99LHF57H98:FCAH<99BH=F97CFDIGC: DFCADHGIG=B;H<9.I=H9C:B5@MH=7(=B;I=GH=7B5@MG=G/CC@G.(/KKK @=B;I=GH=75B5@MG=GHCC@G CF;  /<9.(/*(,HCC@G9LHF57H5K=895FF5MC:@=B;I=GH=7=B8=79GH<5HA95GIF9GMBH5L F95856=@=HM @9L=75@
8=J9FG=HM @9L=75@GCD<=GH=75H=CB 7C<9G=CB ;F5AA5F GD9@@=B; 5B8CH<9FGHFI7HIF5@5B8
A97<5B=7G7CBGHFI7HG HCH5@C:@=B;I=GH=7A95GIF9GK9F99LHF57H98:FCAH<97CFDIGC:
9::=7=9BHDFCADHG /CDF9D5F9H<985H5G9H:CFDF=B7=D5@7CADCB9BH5B5@MG=G,  H<9
9LHF57H98=B8=79GK9F9:IFH<9FDF9DFC79GG98 #=FGH =B8=79GK=H<N9FCHCCB@MJ5F=5B79
K9F9F9ACJ98 5GH<9M8CBCHDFCJ=89G=;B=:=75BH=B:CFA5H=CB:CFH<95B5@MG=G *9LH <=;<@M
7CFF9@5H98=B8=79G K9F99@=A=B5H98HC5JC=8AI@H=7C@@=B95F=HM=GGI9GH<5HA5M5F=G9
8IF=B;,  #=B5@@M 5BM=B8=79GK=H<A=GG=B;CFBI@@J5@I9GK9F9F9ACJ98 /<9F9GI@H=B;
85H5G9H7CBH5=B98 @=B;I=GH=7=B8=79GK<=7<K9F9IG98:CFH<9, 5B5@MG=G ,F=CFHCH<9, 5B5@MG=G H<985H5G9HK5GGH5B85F8=N986MG75@=B;957<=B89LHC<5J95A95BC:N9FC
5B85GH5B85F889J=5H=CBC:CB9)'41%  *(,5B8DF9DFC79GG=B;D=D9@=B9 # "!"!  %)$$

:H9F9LHF57H=B;H<9@=B;I=GH=7=B8=79G K9D9F:CFA985DF=B7=D5@7CADCB9BH5B5@MG=G, HC
=89BH=:MIB89F@M=B;:57HCFGH<5H9LD@5=B98H<9J5F=56=@=HMC:H<9@=B;I=GH=7=B8=79G 29IG98H<9
.7=?=H @95FB@=6F5FM=B,MH<CBHCD9F:CFA, CBH<985H5G9H /<9IG9C:, =BH<=GGHI8M
K5G7<CG9B HCDFCJ=89 5ACF97CADF9<9BG=J9IB89FGH5B8=B;C:H<9@=B;I=GH=7D5HH9FBG
DF9G9BH =B98I75H=CB5@DFCADHG 5B8:CF;9B9F5H=B;ACF9H5F;9H98F97CAA9B85H=CBG:CF
DFCADH9B;=B99F=B;=B98I75H=CB5@G9HH=B;G)'41%  , D=D9@=B9/C9BGIF9H<5HH<985H5G9HK5G5DDFCDF=5H9:CF,  H<9'5=G9F )9M9F +@?=B')+
A95GIF9C:G5AD@=B;589EI57M K5G7CB8I7H98 /<9')+A95GIF9GH<9DFCDCFH=CBC:
J5F=5B795ACB;H<9C6G9FJ98J5F=56@9GH<5H75B699LD@5=B986MH<9IB89F@M=B;:57HCFG &B
H<=GGHI8M H<9CJ9F5@@')+J5@I9K5G  5B8H<9=B8=J=8I5@:95HIF9G5@@<585')+56CJ9 =B8=75H=B;H<5HH<985H5G9HK5GGI=H56@9:CF, 5B5@MG=G )CF9CJ9F 5CB:9FFCB=H9GH
C:G5AD@9589EI57MK5G5@GC7CB8I7H98HC89H9FA=B9=:H<9G5AD@9G=N9K5GGI::=7=9BH:CF,  /<9H9GH=B8=75H98H<5HH<9G5AD@9G=N9K5GGI::=7=9BH:CF, 5B5@MG=GD  
7FI7=5@GH9D=B, =GHC89H9FA=B9H<9BIA69FC:DF=B7=D5@7CADCB9BHG, GHCF9H5=B /<9
F9G95F7<9FIG98 HKCA9H<C8G:CFG9@97H=B;H<9BIA69FC:7CADCB9BHG,5F5@@9@B5@MG=G
, 5B8+DH=A5@ CCF8=B5H9G +  ,5B8+ 5F9HKCA9H<C8G:CFG9@97H=B;H<9BIA69F
C:7CADCB9BHG=B, H<5H<5J958J5BH5;9GCJ9FH<99@6CKA9H<C85B89=;9BJ5@I9G
A9H<C8 CH<A9H<C8G7CBG=89F H<9J5F=56=@=HMC:9=;9BJ5@I9G5B8H<97CFF9@5H=CBG5ACB;

61

J5F=56@9G F9GI@H=B;=BACF9577IF5H95B8F9@=56@99GH=A5H9GC:H<9BIA69FC:7CADCB9BHGHC
F9H5=B /<9, 5B5@MG=GDFC8I798:CIFDF=B7=D5@7CADCB9BHGIG=B;7CADIH5H=CB5@A9H<C8G
D5F5@@9@5B5@MG=G5B8CDH=A5@7CCF8=B5H9G G99:=;IF9 9LD@5=B=B; C:H<9HCH5@
J5F=5B79 /<9:=FGH7CADCB9BH577CIBH98:CFH<9<=;<9GHDFCDCFH=CBC:H<9J5F=5B79 
H<9G97CB87CADCB9BH577CIBH98:CFC:H<9J5F=5B79 K<=@9H<9@5GHHKC7CADCB9BHG
577CIBH98:CFH<9@CK9GHDFCDCFH=CB957< /<97CADCB9BHGK9F9H<9BB5A985B8
=BH9FDF9H9865G98CBH<9@=B;I=GH=75HHF=6IH9GC:H<9:95HIF9GH<5H<58H<9<=;<9GH@C58=B;G
@C58=B;GH<F9G<C@8   )'41%  CADIH5H=CB5@.C@IH=CBGHC#=B8=B;H<9+DH=A5@*IA69FC: CADCB9BHG   &%%')$$! /<9F9G95F7<9F7CB8I7H985EI5@=H5H=J9F9J=9KC:DFCADHGH<5HG7CF98<=;<5B8@CK=B957<
C:H<99A9F;9BH7CADCB9BHG /<9DF=A5FMC6>97H=J9C:H<=GGH9DK5GHC5GG=GH=B=BH9FDF9H=B;
5B8B5A=B;H<99A9F;9BH7CADCB9BHG5B8HCG7FIH=B=N9H<9GD97=:=7DFCADHG H<9F96M
7CB:=FA=B;H<97CBGH=HIH=J9*(,=B8=79GH<5H@C5898CBHC957<7CADCB9BH &BH<=GDCGH<C7
EI5@=H5H=J95B5@MG=GH<9F9GI@HGC:H<9, 5F9GHI8=98=B5ACF989H5=@985B8BI5B798K5M
HC;5=B569HH9FIB89FGH5B8=B;C:K<5H957<7CADCB9BHF9DF9G9BHG=BH9FAGC:H<9CF=;=B5@
J5F=56@9G    &$%!"!  %$  %#"#%%! ! $&%$
/<9, 5B5@MG=GF9J95@985HCH5@C:7CADCB9BHGH<5H577CIBH98:CFIDHC C:H<9
7IAI@5H98J5F=5B79=BH<9@=B;I=GH=7DFCD9FH=9GC:9::=7=9BH <5H$,/DFCADHGG99H56@9 
/56@9 !#3.1.!$)-'2!-$!1)!-#%2
*(+*)").
6;4B6@A60@@<06.A6<;.;1
<;@A?B0A6<;&A?2;4A5

<99<@A?B0A6<;%.A6<6C2?@6AF
&=2063606AF0.12:60.;4B.42(@2*)-.&./.&0")!& "-,&$&)',&'"-1&.%
'*!&)$  C2?.4292::.0<;@A?B0A6<;0<:/6;.A6<;
3?2>B2;0F.0.12:60;2D@360A6<;
:.4.G6;2
 C2?.42.==?<E6:.A20<99<@A?B0A6<;.9@A?2;4A5.0.12:60;2D@360A6<;
:.4.G6;2 <99<@A?B0A6<;?.A6<.0.12:60;2D@
360A6<;:.4.G6;2
 &2;A2;0292;4A5
 'B;6A92;4A5
 <B;A<3C2?/=5?.@2@ ?2>B2;0F<3.0.12:60D<?1@ <:/6;.A6<;<3.0.12:60D<?1@

,*+*,.&*)
*#,&) "








   .,/.-%-3 )-'4)23)#22.#)!3).-!-$.-2314#3).-31%-'3(
/<9:=FGH7CADCB9BH K<=7<577CIBH98:CFH<9@5F;9GH5ACIBHC:J5F=5B79  =G7CADF=G98
C:H<95J9F5;9@9AA57CBGHFI7H=CB7CA6=B5H=CB:F9EI9B7M( ,5B8H<95J9F5;9
5DDFCL=A5H97C@@CGHFI7H=CB5@GHF9B;H<  /<9( ,A95GIF9G<CK:F9EI9BH@MD5FH=7I@5F
7CA6=B5H=CBGC:KCF8G5F9IG98HC;9H<9F5B8<CK:F9EI9BH@MD5FH=7I@5F7CA6=B5H=CBGC:
KCF8G5F9IG98HC;9H<9F /<9 A95GIF9GH<9GHF9B;H<C:5GGC7=5H=CB69HK99B5;=J9B

62

KCF85B8H<9CH<9FKCF8GH<5HHMD=75@@M7C C77IFK=H<=H CH<A95GIF9G9A9F;98GD97=:=75@@M
:CFKCF8GCBGD97=:=7;9BF9G GI7<5G57589A=7 B9KG :=7H=CB 5B8A5;5N=B95B8<CKH<9M
5F97CA6=B98=B5DFCADH  C@@CGHFI7H=CBCF5D9FGCBPG7C@@CGHFI7H=CB5@DF9:9F9B79GF9J95@=BH9F9GH=B;9::97HGC:H<9;F5AA5H=75@7<C=797CBHF=6IH=B;HCDF5;A5H=7D<9BCA9B5 9 ; <98;=B;CFH9BH5H=J9B9GG-5IH=CB5<C  .H9:5BCK=HG7<$F=9G   ( ,5B8 
F9:9FHC5GGC7=5H=CBGHF9B;H<K<=7<75B69G99B5G5B=B8=75HCFC:H<9@9J9@C:@9L=75@
8=J9FG=HM DFCHCHMD=75@=HM 57EI=G=H=CB ,FCHCHMD=75@=HM5@@CKG:CFACF9B5HIF5@5B89::97H=J9
@5B;I5;9 /<9G9:95HIF9G<5J9699BG<CKBHC69DCG=H=J9@M7CFF9@5H98K=H<KF=H=B;EI5@=HM'M@9   /<IG DFCADHGH<5H9L<=6=H<=;<9F@9J9@GC:5GGC7=5H=CBGHF9B;H<( ,5B8 A5M69ACF99::97H=J9=BGH=AI@5H=B;H<CI;<H:I@5B8=BG=;<H:I@F9GDCBG9G:FCA
;9B9F5H=J9&6CHG    .,/.-%-3 .++.2314#3).-!3).)5%12)37
/<97C@@CGHFI7H=CBF5H=C -A95GIF9GH<989;F99C:5GGC7=5H=CB69HK99BHKCKCF8G=B957<7CFDIG &H=GH<9F5H=CC:H<9C6G9FJ98:F9EI9B7MC:H<97C@@C75H9HCH<99LD97H98:F9EI9B7MC:
H<97C@@C75H9 ;=J9BH<9:F9EI9B7MC:H<9KCF8  CADCB9BH:C7IG9GCBH<9:F9EI9B7MC:
79FH5=B7C@@C75H=CBGK=H<=B5DFCADH B5A9@MH<9F5H=CC:KCF8GH<5HH9B8HC7C C77IFK=H<
CH<9FKCF8G=BGD97=:=78CA5=BG GI7<5G57589A=7 B9KG A5;5N=B9 :=7H=CB 5B85@@
8CA5=BG /<9 -=G5A95GIF9C:H<9GHF9B;H<C:5GGC7=5H=CB69HK99BHKCKCF8G 5B8=H<5G
699B:CIB8HC695B9::97H=J9=B8=75HCFC:G9A5BH=7F9@5H98B9GG B=B7F95G9=B7C@@CGHFI7H=CBF5H=C<5G699B:CIB8HC5@GC89D=7H5B=B7F95G9=BKF=H=B;EI5@=HM <@A.3.?<@@92F  -!=J9FG=HM<5G:=J97CBGH=HIH=J9=B8=79G 957<C:K<=7<=G5 -:CF58=::9F9BH7CFDIG
57589A=7 B9KG A5;5N=B9 :=7H=CB 5B85@@ /<9 -:CF57589A=7KCF8GA95GIF9GH<9
GHF9B;H<C:5GGC7=5H=CB69HK99BH<9DFCADH5B857589A=7KCF8G K<=@9H<9 -:CFB9KG
KCF8GA95GIF9GH<9GHF9B;H<C:5GGC7=5H=CB69HK99BH<9DFCADH5B8KCF8G7CAACB@M:CIB8
=BB9KG5FH=7@9G 5B8GCCB DFCADHH<5H<5G58=J9FG9F5B;9C:7C@@C75H9G7C@@CGHFI7H=CBF5H=C:FCA8=::9F9BH7CFDCF5A5M69ACF99::97H=J9=B9@=7=H=B;5F9GDCBG9:FCA57<5H6CH 
9GD97=5@@M=:H<97<5H6CH=G89G=;B98HCF9GDCB8=B5K5MH<5H=GF9@9J5BHHCH<957589A=7
7CBH9LHK<=7<=G=BHF=BG=75@@M8=J9FG9=BH9FAGC:HCD=7G5B8?BCK@98;98CA5=BG    .,/.-%-3 /%#)&)#)37
CADCB9BH=G7CADF=G98C:H<F997CBGH=HIH=J9=B8=79GG9BH9B79@9B;H< IB=H@9B;H< 5B8
J9F6D<F5G9G /<=G7CADCB9BHA95GIF9GH<989;F99C:GD97=:=7=HM5B87@5F=HMC:DFCADHG .9BH9B79@9B;H<=GH<95J9F5;9BIA69FC:KCF8GD9FG9BH9B79=BH<9DFCADH K<=7<=G5
A95GIF9C:GMBH57H=77CAD@9L=HM 0B=H@9B;H<=GH<95J9F5;9BIA69FC:7@5IG9GD9FG9BH9B79 
K<=7<=B8=75H9G<CKAI7<=B:CFA5H=CB=GD57?98=BHC957<G9BH9B79 19F6D<F5G9GF9:9FHC
H<95J9F5;9BIA69FC:J9F6D<F5G9GD9FG9BH9B79 K<=7<F9:@97HGH<9@9J9@C:GD97=:=7=HM5B8
7@5F=HM=BH<9DFCADH5B85F9F9@5H98HCH<9KF=H=B;EI5@=HM <@A.3.?<@@92F  19F6G
D@5M57F=H=75@FC@9=B7CBJ9M=B;GD97=:=7A95B=B;5B857H=CB=B5G9BH9B79  <CCG=B;H<9
5DDFCDF=5H9J9F675B9B<5B79H<97@5F=HM5B8GD97=:=7=HMC:H<9A9GG5;969=B;7CBJ9M98 /<9F9:CF9 =B7@I8=B;J9F6D<F5G9G5G57CBGH=HI9BH=B89L=BH<97CADCG=H9G7CF975B<9@D
5GG9GGH<9GD97=:=7=HM5B87@5F=HMC:H<9@5B;I5;9IG98=BDFCADHG K<=7<75BI@H=A5H9@M
7CBHF=6IH9HCH<99::97H=J9B9GGC:H<9DFCADHG=B98I75H=CB5@G9HH=B;G #IFH<9F =H=G5@GC
BCH56@9H<5H5@@H<F997CBGH=HIH=J9=B8=79G5F9=B8=75HCFGC:GMBH57H=75@7CAD@9L=HMF92
?<@@92F  2<=@9H<9GMBH5LC:5DFCADH8C9GBCHB979GG5F=@M89H9FA=B9H<9EI5@=HMC:
H<9F9GDCBG9;9B9F5H986M@5B;I5;9AC89@G5B87<5H6CHG CH<G=AD@9F5B8ACF9
GCD<=GH=75H98GMBH5L75B@958HC<=;< EI5@=HMF9GDCBG9G 89D9B8=B;CBH<97CBH9LH5B8H<9
GD97=:=7H5G?5H<5B8 0@H=A5H9@M H<9GMBH5LC:5DFCADHG<CI@8697<CG9B65G98CBH<9=BH9B8985I8=9B795B8H<9GD97=:=7;C5@GC:H<9H5G?  <5H6CHG5F975D56@9C:IB89FGH5B8=B;
5B8;9B9F5H=B;H9LHK=H<5K=89F5B;9C:GMBH57H=7GHFI7HIF9G    .,/.-%-3 #!$%,)#!-'4!'%2%CADCB9BH 7589A=7(5B;I5;90G9 =G7<5F57H9F=N986M5G9HC:7CBGH=HI9BH=B8=79GH<5H
A95GIF9H<9:F9EI9B7M5B87CA6=B5H=CBC:57589A=7KCF8G=BDFCADHG /<9G9=B8=79G
=B7@I89A95GIF9GC:5J9F5;9@9AA5:F9EI9B7M 7CBGHFI7H=CB:F9EI9B7M 5B8@9AA5
7CBGHFI7H=CB7CA6=B5H=CB:F9EI9B7MC:57589A=7KCF8G /<9IG9C:57589A=7@5B;I5;9=B

63

DFCADHG=GD5FH=7I@5F@M=ADCFH5BH=B98I75H=CB5@G9HH=B;G 5G=HF9:@97HGH<9H97<B=75@H9FAGC:
57589A=7GI6>97H5F95G5B875B<9@D;9B9F5H=J9&6CHG5B8@5B;I5;9AC89@G9LHF57H
8CA5=B GD97=:=7F9GDCBG9G 7589A=7@5B;I5;9=GC:H9B7<5F57H9F=N986M=HGDF97=G=CB 
GD97=:=7=HM 5B8H97<B=75@H9FA=BC@C;M &B57589A=7KF=H=B; 5IH<CFG5F99LD97H98HCIG9
@5B;I5;9H<5H=G7@95F5B8IB5A6=;ICIG 5B8HC5JC=8J5;I9CF=ADF97=G99LDF9GG=CBG G5
F9GI@H 57589A=7@5B;I5;9H9B8GHC69ACF9GD97=5@=N985B8GD97=:=7H<5B9J9FM85M
@5B;I5;9 2<9B5DFCADH7CBH5=BG57589A=7@5B;I5;9 =HC:H9BG=;B5@GH<5HH<9HCD=7CF
GI6>97HA5HH9F=GACF9GD97=5@=N98CFH97<B=75@=BB5HIF9 /<9IG9C:GD97=:=7H9FA=BC@C;M5B8
>5F;CB=BH<9DFCADH75B<9@DHC89:=B9H<9G7CD95B87CBH9LHC:H<9EI9GH=CB A5?=B;H<9;9B9F5H98F9GDCBG9GACF9H5F;9H985B8H5=@CF98HCH<9;=J9BHCD=7 M=B7CFDCF5H=B;5<=;<9F
89;F99C:57589A=7@5B;I5;9IG9=BDFCADHG H<97<5H6CH75B DFC8I79ACF9GD97=5@=N985B8
H97<B=75@F9GDCBG9GH<5H5@=;BK=H<H<957589A=77CBH9LH /<=G75B699GD97=5@@M=ADCFH5BH=B
98I75H=CB5@G9HH=B;GK<9F9H<9HCD=7G8=G7IGG985F9C:H9B7CAD@9L5B8GD97=5@=N98&BGIAA5FM H<9F9GI@HGC:H<9, 5B5@MG=GG<CK98H<5HH<9@=B;I=GH=7:95HIF9GF9@5H98
HC( .  -! GD97=:=7=HM 5B857589A=7@5B;I5;9IG9 K9F9H<9ACGHDF9J5@9BH:95HIF9G
H<5H7<5F57H9F=N98H<97CFDIGC:9::=7=9BH98I75H=CB5@DFCADHGH<5HK9F9?BCKBHCDFC8I79<=;<EI5@=HMF9GDCBG9G:FCA <5H$,/  !"!$%!#  &$%#!"% )!#
0G=B;H<9F9GI@HGC:H<9,  57CADCG=H9G7CF9=G=BHFC8I798=BH<=GGHI8M5B889F=J98:FCA
H<9, @C58=B;G5B87CBGH=HI9BH=B8=79G ,  @C58=B;GF9DF9G9BHH<9F9@5H=J9=ADCFH5B79C:
957<7CADCB9BH=B9LD@5=B=B;H<9J5F=5B79=BH<985H5 K<=@9H<97CBGH=HI9BH=B8=79G
F9DF9G9BHH<9GD97=:=7@=B;I=GH=7:95HIF9GH<5H7CBHF=6IH9HC957< 7CADCB9BH MAI@H=D@M=B;
H<9@C58=B;GK=H<H<9BCFA5@=N98G7CF9GC:H<97CBGH=HI9BH=B8=79G:CF957<DFCADH 5
7CADCG=H9G7CF9H<5HH5?9G=BHC577CIBHH<9F9@5H=J9=ADCFH5B79C:957<7CADCB9BH5B8H<9
GD97=:=7@=B;I=GH=7:95HIF9GH<5H7CBHF=6IH9HC=HG9::=7=9B7M =G=BHFC8I798=BH<=GD5D9F
*(+*-&."- *,"  )$/$"-"   +" &#& &.2  !"(& 

2=H<H<=GDFCADH7CADCG=H9G7CF9 5ACF9EI5BH=H5H=J9A9HF=7=G=BHFC8I798=BH9FAGC:
<CKDFCADH9::=7=9B7M75B695DDFC57<98H<FCI;<@=B;I=GH=7:95HIF9G ,FCADH9B;=B99FGK=@@6956@9HCH5=@CFH<9=F@9L=75@7<C=79GK<9B7F5:H=B;9::=7=9BHDFCADHGIG=B;H<99A9F;9BH
7CADCB9BHG=BH<=GGHI8M5B8H<9(,".7CADCG=H9G7CF9 %CK9J9F H<9F9G95F7<9FKCI@8@=?9HC<=;<@=;<HH<5HH<9, 8CB9=BH<=GGHI8MCB@MF9GI@H98HC7CADCB9BHGH<5HF9DF9G9BH
IDHC C:H<9J5F=5B79=BH<9DFCADH7CFDIG #57HCFGH<5H75B577CIBH:CFH<9F9A5=B=B;
 B998HC69=BJ9GH=;5H98:IFH<9F /C9J5@I5H9H<9D9FD@9L=HMG7CF9GC:H<9;9B9F5H98
F9GDCBG9G:CFDFCADHGK=H<<=;<(,". H<9:C@@CK=B;D=D9@=B9K5G=AD@9A9BH98G99:=;IF9 )'41% (,".9J5@I5H=CBD=D9@=B9DFCADHK=H<<=;<(,".=GG<CKB69@CK
96!-37.43.!#3!2!31!5%+'4)$%6)++61)3%7.4,7+.#!3).-!-$7.46)++24''%23!
/+!#%3.5)2)3-%!1,7+.#!3).--2.,%#!2%26)++!+2.')5%7.43(%37/%.&/+!#%26)++
5)2)3 .46)++!+2.24''%23,%/+!#%2.&2),)+!137/%3(!3!1%#+.2%3.,7&)123+.#!3).-7
&)12324''%23).-1%04%23)2!,)-23!-"4+ %7.<+4!-$6!-33.5)2)3.-+7,42%4,2

64

/<9DFCADH=GGD97=:=7=B=HGF9EI9GH:CF5D@579HCJ=G=HB95FH<9@C75H=CBDFCJ=898 K=H<
H<958898GD97=:=75H=CBC:AIG9IAGCB@M /<9DFCADH=G7@95F=B=HG=BGHFI7H=CBG K=H<5
GHF5=;<H:CFK5F8F9EI9GH5B857@95F=B8=75H=CBC:H<9HMD9C:D@579H<9IG9F=G=BH9F9GH98=B /<9DFCADH5@GC9L<=6=HG57589A=7@5B;I5;9IG9 5G=H=GDF9G9BH98=B5:CFA5@5B8H97<B=75@
A5BB9F K=H<DF97=G9@5B;I5;95B8GD97=:=7H9FAGGI7<5G@C75H=CB D@579HCJ=G=H 5B8
AIG9IAG /<9G99@9A9BHGDFCJ=8957@95F5B8IB5A6=;ICIG89G7F=DH=CBC:K<5HH<9IG9F
=G@CC?=B;:CF K<=7<<9@DGH<97<5H6CH;9B9F5H95ACF99::97H=J95B8F9@9J5BHF9GDCBG9 +J9F5@@ H<9DFCADH=GK9@@ 7F5:H985B8A99HGH<9F9EI=F9A9BHG:CF5B9::97H=J998I75H=CB5@
7<5H6CHDFCADH /<9;9B9F5H98F9GDCBG9C:H<95:CF9A9BH=CB98DFCADH=GDF9G9BH9869@CK
K=H<5D9FD@9L=HMG7CF9C:  
41%!2!31!5%+'4)$%#!-24''%232.,%,42%4,2&.17.43.5)2)3)-23!-"4+ %7.<+4(%1%!1%2%5%1!+,42%4,2)-3(%!1%!3(!3!1%6.13(5)2)3)-')-#+4$)-'
3(%23!-"4+.$%1-42%4,3(%%1!42%4,!-$3(%42%4,.&--.#%-#%++.&3(%2%
,42%4,2(!5%!4-)04%#.++%#3).-.&!13!-$!13)&!#323(!32(.6#!2%3(%1)#(#4+341!+(%1)3!'%.&23!-"4+$$)3).-!++7)&7.41%)-3%1%23%$)-5)2)3)-',.1%,42%4,23(%1%!1%
2%5%1!+.3(%1./3).-2-%!1"73(!37.4,)'(36!-33.#(%#*.4324#(!23(%23!-"4+1#(!%.+.')#!+42%4,3(%./*!/)!+!#%42%4,!-$3(%!')!./()!42%4,
(%2%,42%4,2!1%!+++.#!3%$6)3()-!2(.13$)23!-#%.&%7.<+4!-$.&&%1!&!2#)-!3)-'
)-2)'(3)-3.3(%()23.17!-$#4+341%.&23!-"4+J5;I9DFCADHK5GKF=HH9BG<CKB69@CK6975IG9H<97CFDIGC:DFCADHGIG98=BH<=G
GHI8M=G7CADF=G98C:?BCKBK9@@ KF=HH9B5B89::97H=J9DFCADHG /<=G=@@ K=FH9BDFCADH;CH5
@CK(,".G7CF95B8C6G9FJ56@M@CKDF9G9B79C:H<97CBGH=HIH=J99@9A9BHG=GG<CKB69@CK
9#3!2!$!3!2#)%-3)23$%2)'-!(7"1)$1%#.,,%-$%12723%,!//1.!#(&.1!1%!+
6.1+$42%#!2%:/<9;9B9F5H98F9GDCBG99L79FDHG<CKB69@CKK5G89H5=@985B8=B:CFA5H=J9 6IHH<9
D9FD@9L=HMG7CF9K5G  = 9 G<CK=B;H<5H <5H$,/<5G@CK7CB:=89B79=B=HG;9B9F5H98
F9GDCBG9 2!$!3!2#)%-3)23#!-(%+/$%2)'-!(7"1)$1%#.,,%-$%12723%,!//1.!#(&.1!
1%!+6.1+$42%#!2%%32#.-2)$%1!2#%-!1).6(%1%6%!1%$%2)'-)-'!1%#.,,%-$%1
2723%,&.1!-.-+)-%,!1*%3/+!#%3(!32%++2!5!1)%37.&/1.$4#321!-')-'&1.,%+%#31.-)#23.
#+.3()-'.$%2)'-!(7"1)$1%#.,,%-$%12723%,&.13()242%#!2%6%#!-#.,")-%
#.++!".1!3)5%&)+3%1)-'!-$#.-3%-3"!2%$&)+3%1)-'.++!".1!3)5%&)+3%1)-'#!-"%42%$3.
)$%-3)&7'1.4/2.&2),)+!142%12"!2%$.-3(%)1/41#(!2%()23.171!3)-'2!-$.3(%1"%(!5).128
&HK5GC6G9FJ98H<5HH<9A5H<DFCADH 9J9B5G=H=GK9@@ KF=HH9B <58H<9@CK9GH(,".
5B8<585F9@5H=J9@M<=;<D9FD@9L=HM #=B5@@M 57CFF9@5H=CB5B5@MG=GK5GD9F:CFA98CBH<9GI6
7CFDIGC: F5B8CA@MG5AD@98DFCADHG /<9F9GI@HG=B8=75H95G=;B=:=75BHB9;5H=J9
7CFF9@5H=CB69HK99B(,".5B8,9FD@9L=HMG7CF9 K=H<57CFF9@5H=CB7C9::=7=9BHC: D /<=GF9GI@H=B8=75H9G5A95B=B;:I@F9@5H=CBG<=D69HK99B(,".5B8,9FD@9L=HMG7CF9 
88=H=CB5@@M H<9G75HH9FD@CHC:H<985H5G<CKG57@95F8CKBK5F8HF9B8 :IFH<9FGIDDCFH=B;H<9
B9;5H=J97CFF9@5H=CB69HK99B(,".5B8,9FD@9L=HMG7CF9 /<9(,".G7CF9G:CFH<9G5AD@9C: K9@@KF=HH9BDFCADHGF5B;98:FCA HC A95B G8    
0B89FGH5B8=B;<CKHC9B;=B99FDFCADHGH<5H5F97CADF9<9BG=J95B8577IF5H975B5=8=BH<9
89J9@CDA9BHC:ACF99::97H=J998I75H=CB5@A5H9F=5@G M@9J9F5;=B;H<9G9@=B;I=GH=7:95HIF9G
=B9B;=B99F=B;DFCADHG @95FB9FG5B8H957<9FG75B69ACF9=BH9BH=CB5@=B:CFAI@5H=B;
DFCADHGH<5H@9J9F5;97<5H6CHG @5F;9@5B;I5;9AC89@G 5B8CH<9F;9B9F5H=J9&HCC@G /<9:CIF7CADCB9BHGC:@5B;I5;9=BDFCADHGF9J95@H<5H9::=7=9BHDFCADHG5F97<5F57H9F=N986M
=BH9BH=CB5@;F5AA5H=75@GHFI7HIF9G( .5B8 -! GD97=:=7=HM 5B857589A=7@5B;I5;9

65

IG9 ,FCADHGK=H<<=;<J5@I9G:CF6CH<( .7CADCB9BH5B8 -!7CADCB9BH5F9
@=?9@MHC6969HH9F;F5AA5H=75@@MGHFI7HIF985B8ACF9B5HIF5@DFCADHG7CAD5F98HCH<CG9
K=H<@CKJ5@I9G:CF6CH<A95GIF9G ,FCADHGK=H<<=;<( .7CADCB9BH=B8=75H9H<5HH<9H5F;9HKCF8=BH<9DFCADH
<5G5GHFCB;H9B89B7MHCC77IFK=H<GD97=:=77C@@C75H9G K<=7<A95BGH<5HH<9DFCADH=GACF9
@=?9@MHC69;F5AA5H=75@@MK9@@ :CFA98GI7<H<5HH<9A9GG5;9=GA5897@95F9F6975IG9C:H<9;F5AA5H=75@GHFI7HIF9 ,FCADHGK=H<<=;< -!7CADCB9BH=B8=75H9H<5HH<9KCF8G=BH<9
DFCADHH9B8HCC77IFHC;9H<9F:F9EI9BH@M K<=7<GI;;9GHGH<5HH<9M5F9G9A5BH=75@@MF9@5H98
5B8@=?9@MHC:CFA57C<9F9BHG9BH9B79CF=895  CBJ9FG9@M DFCADHGK=H<@CK( .5B8
-!J5@I9GGI;;9GHH<5HH<9KCF8G=BH<9DFCADH5F9BCH7@CG9@M5GGC7=5H98K=H<957<CH<9F 
5B8A5MBCH:CFA57C<9F9BHCFB5HIF5@ GCIB8=B;G9BH9B79 &BH<=G75G9 H<9;9B9F5H98
DFCADHA5M69@9GGIG9:I@CF9::97H=J9:CF=HG=BH9B898DIFDCG9 +J9F5@@ <=;<9FJ5@I9G:CF
6CH<( .5B8 -!=B8=75H9H<5H57<5HDFCADH=GACF9@=?9@MHC69B5HIF5@ 7C<9F9BH 5B8
9::97H=J9:CF;9B9F5H=B;A95B=B;:I@5B89B;5;=B;7CBJ9FG5H=CBG6975IG9C:H<9;F5AA5H=75@
897=G=CBGK<9B:CFAI@5H=B;H<9DFCADH  <5H6CHGP56=@=HMHC;9B9F5H969HH9F5BGK9FGHCDFCADHGH<5H5F9G9A5BH=75@@M7C<9F9BH5G5F9GI@HC:H<9K9@@ :CFA98;F5AA5F=G8I9HCH<9
HF5=B=B;CBB5HIF5@@5B;I5;9D5HH9FBG 5@GC5;F5AA5H=75@@MK9@@ :CFA98?BCK@98;965G9CF
HF5=B=B;65G9 HC;9B9F5H9F9@5H98F9GDCBG9G=BH<9G5A97CBH9LH .D97=:=7=HM7CADCB9BHA95GIF9GH<989;F99C:7@5F=HMC:DFCADHGH<FCI;<H<F997CBGH=HIH=J9=B8=79GG9BH9B79
@9B;H< IB=H@9B;H< 5B8J9F6D<F5G9G "::=7=9BH7<5H$,/DFCADHGG<CI@869GD97=:=75B87@95F
HC;9B9F5H9<=;< EI5@=HMF9GDCBG9G  CADCB9BH<9@DG57<=9J9H<=G6MA95GIF=B;H<97@5F=HM
5B8GD97=:=7=HMC:H<9DFCADHGH<FCI;<H<9H<F997CBGH=HIH=J9=B8=79G MA95GIF=B;G9BH9B79
@9B;H< IB=H@9B;H< 5B8J9F6D<F5G9G H<97CADCB9BH9J5@I5H9G<CKAI7<=B:CFA5H=CB=G
D57?98=BHC957<G9BH9B79 &B588=H=CB IG=B;GD97=:=7J9F6G75B<9@D9B<5B79H<97@5F=HM5B8
GD97=:=7=HMC:H<9A9GG5;969=B;7CBJ9M98 5B8@CB;9F5B8ACF97CAD@9LDFCADHGA5M
DFCJ=89ACF97CBH9LH5B8GD97=:=7=HM %CK9J9F H<9EI5@=HMC:5F9GDCBG9I@H=A5H9@M89D9B8G
CBJ5F=CIG:57HCFG69MCB8>IGHH<9@9B;H< =B7@I8=B;H<9F9@9J5B795B87@5F=HMC:H<9DFCADH
5B8H<9577IF57MC:H<9=B:CFA5H=CBDFCJ=898 +J9F5@@  CADCB9BH7CBHF=6IH9GHC9::=7=9BH
7<5H$,/DFCADHG6MDFCJ=8=B;5A95BGHCA95GIF95B8=ADFCJ9H<9GD97=:=7=HM5B87@5F=HMC:
H<9DFCADHG6M=B7CFDCF5H=B;>IGH9BCI;<9@56CF5H=CB <9B79<5J=B;F9@5H=J9@M@CB;9F
DFCADHG (5GH@M 57589A=7@5B;I5;9IG9=GDF97=G9 GD97=:=7 5B8H97<B=75@ A5?=B;=HACF9
GD97=5@=N98H<5B9J9FM85M@5B;I5;9 /<9DF9G9B79C:57589A=7@5B;I5;9=B5DFCADH
=B8=75H9G5ACF9H97<B=75@CFGD97=5@=N98HCD=7 0G=B;57589A=7@5B;I5;9<9@DG89:=B9H<9
7CBH9LH5B8G7CD9C:H<9EI9GH=CB 9B56@=B;7<5H6CHGHC;9B9F5H9H5F;9H985B8H5=@CF98
F9GDCBG9G /<=GF9GI@HG=BACF9577IF5H95B8DF97=G95BGK9FG  CADCB9BH=G7FI7=5@=B
98I75H=CB5@G9HH=B;G K<9F9HCD=7G5F97CAD@9L5B8F9EI=F9GD97=5@=N98?BCK@98;9 
DFC8I7=B;ACF9GD97=5@=N985B8H97<B=75@F9GDCBG9GH<5H5@=;BK=H<57589A=77CBH9LH &B7CB7@IG=CB H<9@=B;I=GH=75HHF=6IH9GC:9::=7=9BHDFCADHGH<5HK9F9=89BH=:=98=BH<=G
GHI8M<5J9=ADCFH5BH=AD@=75H=CBG:CF@95FB=B; M@9J9F5;=B;H<9G95HHF=6IH9G @95FB9FG5B8
H957<9FG75B69ACF9=BH9BH=CB5@=BH<9=F@9L=75@7<C=79GK<9BH<9M@9J9F5;9;9B9F5H=J9&
5B8G=A=@5FHCC@G=BH<9H957<=B;5B8@95FB=B;DFC79GG  %%! $/<985H5G9HIG98=BH<=GGHI8MK5G@=A=H98HCDI6@=7@M5J5=@56@97CFDCF5C:?BCKB9::=7=9BH
DFCADHGH<5HH957<9FG F9G95F7<9FG 5B8DF57H=H=CB9FGIG98 /<9F9K5GBC7CADF9<9BG=J9
J5@=85H=CBC:H<957HI5@EI5@=HMC:H<9DFCADHG5G=89:FCAH<9D9FD@9L=HM9GH=A5H9J5@=85H=CBC:
=HG;9B9F5H98F9GDCBG9G #IFH<9FH<97CFDCF5IG98=BH<=GGHI8M8CBCHF9DF9G9BHH<9:I@@
F5B;9C:DCGG=6@9DFCADHG5B8HCD=7GH<5H75B699B7CIBH9F98=BF95@ KCF@85DD@=75H=CBG5B8
89D@CMA9BHGC:;9B9F5H=J9&=B98I75H=CB  

66

/<9F95F9G9J9F5@5F95GC::IHIF9KCF?H<5H75B6I=@8CBH<9:=B8=B;GC:H<=GGHI8M +B9
DCH9BH=5@5J9BI9=GHC89J9@CD5HCC@CF:F5A9KCF?H<5H75B5IHCA5H=75@@M=89BH=:M<=;< EI5@=HM
DFCADHG65G98CBH<9@=B;I=GH=7:95HIF9G=89BH=:=98=BH<=GGHI8M .I7<5HCC@7CI@869IG986M
98I75HCFG5B8GHI89BHGHC7F95H9ACF99::97H=J9DFCADHGH<5H75B@958HCACF9577IF5H95B8
7CADF9<9BG=J9F9GDCBG9G:FCA(()G 2<=@9H<9G7CD9C:H<=GGHI8M=GDFCADH9B;=B99F=B;
:CF7<5H6CHG :IHIF98=F97H=CBG=B98I75H=CB75BG75@9H<969B9:=HGC:H<97CADCG=H9G7CF9
(,".H<5H=G=BHFC8I798=BH<=GGHI8M (,".75B69IG98HC9J5@I5H9H<9@=B;I=GH=79::=7=9B7M
C:DFCADHGIG98=B98I75H=CB5@7<5H6CHGCFHIHCF=B;GMGH9AGH<5H5F96I=@HCBHCDC:
:CIB85H=CB5@AC89@G <9@D=B;HC=89BH=:MK<=7<DFCADHG5F9ACGH9::97H=J9HC:57=@=H5H9CDH=A5@
=B:CFA5H=CBF9HF=9J5@ 

992;.@0.9B  0!.:.?.&?<@@9F&'?.B@.; .AB&   <1296;46;16C61B.9
16332?2;02@.:<;4D?6A2?@B@6;4?2.12?/2;05;!'$2/ 4*+/4'2/$4+0/$-%0/('2'/%'0/'&5%$4+0/$/&/'7-'$2/+/)4'%*/0-0)+'3==  '0.12:F@5<8 %.:.@.:F&;256A5.22?A56&%   .?05@F@A2:.A60@B?C2F<3
0<4;6A6C205.A/<A@6;=2?@<;.96G2192.?;6;43?.:2D<?8;  +84*/4'2/$4+0/$-0/('2'/%'0/"+2'-'330..5/+%$4+0/3+)/$-20%'33+/)$/&'4702,+/)"+ ==
<9<A2@0B   ""B11F.5.A/<A3<?=2?@<;.96G2192.?;6;4D6A5 ""@;0==  
?.2@@2?  22=2?92.?;6;4D6A5.1C.;02@6;16@0<B?@2@062;02.;1A205;<9<4F0-+%9/3+)*43(20.4*''*$6+02$-$/&2$+/%+'/%'3    
0.?A5F&%<@0<2%992;682;@ 0!.:.?.&  BA<:.A21
D?6A6;42C.9B.A6<;<2@@=2996;4.;14?.::.?3221/.08@B==<?A5645>B.96AFD?6A6;4.;1
?2C6@6<;33'33+/)"2+4+/)  
6; #5.;4  ;5.;06;4=<@A@20<;1.?FD?6A2?@KD?6A6;4@8699@D6A5.
05.A/<A052/$-0(&5%$4+0/$- '%*/0-0)90%+'49   
.;4 208225:.;;<992?&B@0528  <DA<#?<:=A"==<?AB;6A62@.;15.992;42@<3-2?<.;12D&5<A2.?;6;43<?B:.;;A2?.0A6<;6;?2.A6C2
==960.A6<;@<32;2?.A6C2 <129@$2#+612'12+/4$2#+6  
-.:36?2@0B#2?26?.*<;4%,.?A:.;;,.;4$  =?69*5F<5;;F0.;KA=?<:=A5<D;<;2E=2?A@A?F.;13.69A<12@64; =?<:=A@;20%''&+/)30(4*'  
0/('2'/%'0/5.$/$%4023+/0.154+/)934'.3==  ,.<B .E.?A<9<9.@A.6? <<?2&2/.@A6.;%62129#<;AB@&A2;2A<?=  .;A.@A60.99F
"?12?21#?<:=A@.;1*52?2A<6;1'52:"C2?0<:6;42D&5<A#?<:=A"?12?&2;@6A6C6AF
.?+6C=?2=?6;A.?+6C  
*56A2B$.F@&&.;1/<?; "92.69/2?A9;.@5.?&=2;02?&:6A5
&05:61A  =?<:=A=.AA2?;0.A.9<4A<2;5.;02=?<:=A2;46;22?6;4D6A5
05.A4=A$2#+612'12+/4$2#+6    
.96;8.#2?2NI;6 6?0B7PH8".;BN  ";A5221B0.A6<;.96:=.0A<3
5.A#'@?A63606.9;A299642;02?2.1FA<</A.6;.B;6C2?@6AF124?22$2#+612'12+/4
$2#+6   ,2.1<;*;F.;4"" 6G<B?6#2.05'2@A?<D#  '5212.A5<3A52@5<?A
3<?:=5F@60@2@@.F6;A520<:6;4?2C<9BA6<;*93+%3&5%$4+0/    
6A?<C6O&;1?2<92AA6F<B/"  5.A4=A<?5B:.;12A20A.;12E=9.6;2E=9.6;6;4
1206@6<;@<3:.056;292.?;6;4:<1293<?12A20A6;4@5<?A05.A4=A42;2?.A21A2EA.?+6C=?2=?6;A.?+6C    %.BA6<;.5<#B05@%  %202;A05.;426;@A.A6C2=?<4?2@@6C2@.0<99<@A?B0A6<;.9
6;C2@A64.A6<;<3?6A6@5;496@56;  .;1  /)-+3*$/)5$)'+/)5+34+%3   
&A23.;<D6A@05?62@&'  <99<@A?B0A6<;@;C2@A64.A6;4A526;A2?.0A6<;<3D<?1@.;1
0<;@A?B0A6<;@;A2?;.A6<;.97<B?;.9<30<?=B@96;4B6@A60@  F92   2.@B?6;4@F;A.0A6012C29<=:2;A6; D?6A6;46;24?.6;216;1602@<3@F;A.0A60
0<:=92E6AF.;1B@.42/.@216;1602@<3@F;A.0A60@<=56@A60.A6<;
<@A.3.'?<@@92F&  )2?/.?4B:2;A0<;@A?B0A6<;0<:=92E6AF6;1602@.;1 D?6A6;4
>B.96AF3320A@<3D?6A6;4A.@8@.;1=?<:=A@<B?;.9<3&20<;1.;4B.42*?6A6;4   
F92?<@@92F&   2.@B?6;4@F;A.0A600<:=92E6AF6; D?6A6;4B@6;436;24?.6;21
09.B@.9.;1=5?.@.96;1602@ *'0&'2/$/)5$)'052/$-  
 

67

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Large Language Models (GPT) for
automating feedback on programming
assignments
Maciej PANKIEWICZa* & Ryan S. BAKERb
Institute of Information Technology, Warsaw University of Life Sciences, PolandbPenn Center for Learning Analytics, University of Pennsylvania, USA*maciej_pankiewicz@sggw.edu.pla

Abstract: Addressing the challenge of generating personalized feedback forprogramming assignments is demanding due to several factors, like the complexity ofcode syntax or different ways to correctly solve a task. In this experimental study, we
automated the process of feedback generation by employing OpenAI’s GPT-3.5 model
to generate personalized hints for students solving programming assignments on an
automated assessment platform. Students rated the usefulness of GPT-generated
hints positively. The experimental group (with GPT hints enabled) relied less on theplatform's regular feedback but performed better in terms of percentage of successful
submissions across consecutive attempts for tasks, where GPT hints were enabled.
For tasks where the GPT feedback was made unavailable, the experimental groupneeded significantly less time to solve assignments. Furthermore, when GPT hints
were unavailable, students in the experimental condition were initially less likely tosolve the assignment correctly. This suggests potential over-reliance on GPTgenerated feedback. However, students in the experimental condition were able tocorrect reasonably rapidly, reaching the same percentage correct after sevensubmission attempts. The availability of GPT hints did not significantly impact students'
affective state.
Keywords: Programming, automated assessment tools, automated feedback, LLM,
GPT

1. Introduction
Large Language Models (LLMs) are deep neural network models able to effectively process
and analyze complex linguistic structures (Carlini et al., 2021). LLMs have garnered significant
attention in the field of natural language processing (NLP) because of the ability to generate
human-like text. This feature makes LLMs a promising technology in educational settings(Finnie-Ansley et al., 2022; Dai et al., 2023; Pardos & Bhandari, 2023), where the provision of
personalized feedback is integral to scaffold learning effectively (Jackson & Graesser, 2007;
Hull & du Boulay, 2015).
One of the most prominent examples of LLMs is the OpenAI's Generative Pre-trained
Transformer (GPT) series (Radford et al. 2019; Brown et al. 2020), which leverages atransformer architecture and is able to capture long-range dependencies and positional
information within a text. It is available through the ChatGPT application and via an Application
Programming Interface (API), enabling integration with different existing applications,
including educational apps. Especially in the field of computer science education, positiveeffects of such embedment may emerge earlier than in other educational domains, becauseof the significant effort conducted to develop models trained on large sets of programmingcode examples (Finnie-Ansley et al., 2022).

68

The field of computer science education has already extensively leveraged softwaretools that automate the process of self-paced learning. Multiple automated assessment tools
have been designed to evaluate and provide feedback on student performance in various
educational tasks (Deeva et al. 2021). The proliferation of these tools in computer scienceeducation, in areas from introductory programming (Edwards & Murali, 2017; Brusilovsky etal., 2018) to databases (Stanger, 2018), has had benefits and has contributed to thedevelopment of self-paced learning environments in other domains (Paiva et al., 2022). Alongwith this development, there is a growing body of research on personalizing feedback toenhance student learning outcomes (Deeva et al. 2021). Nevertheless, effective
implementation of this approach continues to present significant challenges, given the difficultyof comprehensively assessing student work (Maier & Klotz, 2022).
In this study, we try to leverage the potential of GPT models for supporting students inproviding feedback on programming tasks by integrating GPT through an API with anapplication for automated assessment of programming code. The GPT-3.5 model is used toautomatically generate personalized feedback for university students taking an Objectoriented programming course. In this paper, we presented a controlled experiment comparinga system that adds GPT-generated hints to a system that only uses the original, humandeveloped hints. We compare the two conditions across multiple dimensions, including
immediate performance, usage, performance on later content (without GPT-generated hints),
time taken to complete assignments, and affect.

2. Methods
The study involved second-semester computer science students at the Warsaw University ofLife Sciences (Poland) enrolled in an Object-oriented programming course that was required
for their major. The course was conducted using the C# programming language. A total of 132
students consented to participate (out of 174 students taking the course) and were randomly
assigned to either the control (N=66) or the experimental group (N=66). A pre-test wasadministered at the beginning of the semester to establish a baseline understanding of thestudents' knowledge before engaging with the platform's content. The process of datacollection spans the initial part of the semester, specifically covering a nine-week period within
the fifteen-week academic term. This timeframe extends from the first week of March 2023
through to the first week of May 2023. 93 students submitted at least one solution during theperiod of the study (control: 46, experimental: 47).
The experiment utilized the RunCode online application – a platform for automatedexecution and testing of a programming code available at runcodeapp.com (Pankiewicz, 2020).
The application has been used by students at the University since 2017 within programmingcourses. All students that participated in this experiment were familiar with the application andhad actively used it during the Introduction to programming course taught in the previous (first)
semester. Students submitted programming code using an editor integrated within theapplication.
46 programming assignments covering basic object-oriented programming concepts(classes, objects, fields, methods, constructors, encapsulation, inheritance, and polymorphism)
have been made available on the platform for the purpose of practicing the course material.
These concepts were introduced during the first 6 weeks of the course. Each assignment wascomposed of a collection of more specific subtasks, such as: the creation of a class, theaddition of fields with appropriate access modifiers, the definition of constructors, etc. A
comprehensive suite of 809 unit tests was developed to thoroughly assess performance onthese subtasks, ensuring a detailed evaluation of their individual components. Multiple
submissions on a task were allowed. Usage of the platform was voluntary, and neither usage
nor results within the platform counted towards the final grade. 5923 code submissions werecollected during the period of study (control: 3077, experimental: 2846).
The unit tests were designed to validate code requirements specified by the assignment.
For instance, a test aimed at confirming the presence of the "User" class within the submitted
code might be denoted as TestIsClassDefined("User"), with an expected value of true. This69

approach ensured that different aspects of the assignment's requirements were thoroughly
examined and verified in the students' code submissions.
The application provided students with a score (0-100%) and feedback after each codesubmission. If the submitted code failed to compile, the regular feedback available on theplatform was presented without requiring any additional clicks and included details on compiler
errors. The platform also highlighted in the online code editor lines where errors occurred. If
the code compiled successfully, the feedback contained information on the executed unit tests,
including input values and expected output. Students needed to click on a specific test fromthe list to access detailed information about its execution. The tests were color-coded, withgreen indicating success and red signifying failure, to facilitate easy identification of the test
outcomes.
Both the control and experimental groups had access to the same set of tasks within theapplication. However, for 38 out of 46 tasks, students in the experimental group had an
additional feature enabled, which provided them with feedback generated by a large language
model through the GPT-3.5 API. For the remaining 8 tasks, students in the experimental grouponly had access to the regular feedback offered by the platform. These tasks were slightly
more challenging and were designed to encapsulate the concepts introduced in previously
solved tasks, as well as assess the comprehension of these concepts.
Within the experimental condition, additional feedback was provided by the GPT model,
going beyond the standard feedback provided by the application, which included informationon compiler errors, runtime errors and details on executed unit tests. The GPT-generated
feedback provided suggestions for code improvement, explanations of compiler errors, andhints for debugging. The feedback was designed to be informative and constructive, aiming to
guide students towards correct solutions and improve their understanding of programming
concepts, without revealing the correct code solution. This feedback was presented
immediately after code submission, with a prompt asking students to rate the usefulness ofthe hint on a 5-point Likert scale ranging from 'Not useful at all' to 'Extremely useful'. Feedback
was automatically generated in Polish, to match the assignments and the interface of thesystem, which also used this language.
Feedback was requested from GPT when the code submitted by a student exhibitedcompiler errors, runtime errors, or failed at least one unit test. The dynamically generated
prompt included the assignment text (in Polish) and the student's code, accompanied byadditional information based on the testing results. This information contained a compiler
message for non-compiling code, exception type and message for compiled code thatgenerated an exception during execution, or details regarding a failed unit test: the test name,input values, and expected outcome. To accommodate the OpenAI GPT API's token limitation(4,000 tokens), assignments were designed in a way that ensured that even the most complex
tasks and accompanying code could fit within the imposed constraint. The prompt text was
similar for the three scenarios mentioned, with variations stemming from the testing process
outcomes. In cases where the code did not compile, a specific prompt was generated:
I want you to act as a Stackoverflow post that helps me to solve a programming
assignment in C#. I want you to explain in Polish why this code does not compile.
Don't write solution in the explanation, but focus on meaningful hints. I want you
to also include the line where the compiler error occurred in the explanation. I
want you to also include a line number for each detected error in the explanation.
To help me better understand your response, highlight keywords, line numbers, classnames, variable names, messages, line numbers and error names with the <code> </code>
HTML markup in the explanation. Programming assignment: ### <ASSIGNMENT_TEXT> ###C# code: <STUDENT_CODE> ### Compiler errors: <COMPILER_ERRORS>
The following parameters were used for the request: 'model': 'text-davinci-003','temperature': 0, 'max_tokens': 500, 'n': 1, 'top_p': 1, 'frequency_penalty': 0,'presence_penalty': 0.

An illustration of the hint proposed by the GPT in the event of encountering code with asyntax error is presented as follows (translated from Polish to English with highlights
generated by the GPT): “A compiler error error CS1002 occurred in line 5 of the code withthe message ; expected . This means that there is a missing semicolon ; after the70

declaration of the string name field in line 5. To fix the error, a semicolon ; should be
added at the end of line 5. A ; is required at the end of every declaration in C#.”
Students self-reported their affective states while completing tasks on the platform byinteracting with a dynamic HTML element. The platform prompted students for their emotional
state following the receipt of submission results by asking them to select the option that best
described their current feelings. The response options included: Focused, Anxious, Bored,
Confused, Frustrated, and Other (in this order), accompanied by representative emoticons.
These affective states were chosen due to their relevance to the learning process and theirprevalence in past research (Karumbaiah et al., 2022). To mitigate potential frustration fromoverly frequent survey prompts, the platform randomly decided after each submission whether
to present the survey, with a probability of one in three.

3. Results
Due to violations of normality assumptions, non-parametric Mann-Whitney U tests were
utilized. Benjamini-Hochberg alpha correction (Benjamini & Hochberg, 1995) was applied tocontrol for multiple comparisons in analysis of differences in affective state frequencies among
students.

3.1 Pre-test results
To ensure that the control and experimental groups were comparable in terms of their initial
understanding of the relevant programming concepts at the beginning of the study, a pretest
consisting of 7 multiple-choice questions was administered to both the control andexperimental groups. These questions covered key topics on object-oriented programming,
including classes, constructors, encapsulation, inheritance, and polymorphism. No statisticallysignificant differences were identified between the control (Mdn=0) and the experimental(Mdn=1) groups (W=1791, p=0.249, Mann-Whitney U test).

3.2 Usefulness of the GPT hints
In the examination of the perceived usefulness of the GPT generated hints, a total of 1,442
responses were collected from N=46 participants within the experimental group throughout
the duration of the study. As the control group did not have access to the GPT hints, no
responses were obtained from this group. Figure 1 displays the histogram of the frequency ofusers according to the median rating they gave to received hints, with the x-axis representing
the median of the hint rating for a user.

Figure 1. Distribution of users according to the median rating for received hints.

71

As Figure 1 shows, the majority of users rated the GPT hints they received as useful –these results suggest that the hint feature was generally well-received by users.

3.3 Usage of the regular platform feedback
One possible impact of the GPT feedback is that learners may use the other feedback lessoften. To evaluate whether there were differences in the usage of the regular (non-GPT)
platform feedback between the experimental and control groups, we analyzed the percentage
of incorrect submissions for which students clicked on feedback to access details about tests
that failed. We did not examine clicks on feedback for tests that ended successfully, as theapplication did not generate GPT hints for these tests. Therefore, students who only madesuccessful submissions, and students who submitted only non-compiling code in theirsubmissions (in this case the platform always presents feedback, so users are not required toclick on it), were not included in this analysis.
A statistically significant difference was found between the experimental (Mdn=0.321,
N=45) and control (Mdn=0.710, N=45) groups (W=282, p<0.001, Mann-Whitney U test) for theset of tasks where experimental group had GPT feedback enabled (38 out of 46). The
experimental group used the regular feedback significantly less than the control group forthese tasks.
However, no significant difference was found between the experimental (Mdn=0.769,
N=23) and control (Mdn=0.667, N=25) groups (W=341, p=0.269, Mann-Whitney U test) for theset of tasks, where GPT feedback was not available for experimental group (8 out of 46).
When GPT feedback was unavailable, the experimental group utilized the platform's regularfeedback in a similar amount to the control group (Figure 2).

Figure 2. Comparison of the percentage of incorrect submissions with at least one click torequest (view) details on the test that failed (regular platform’s feedback) between the control
and experimental group (on the left: tasks with GPT feedback enabled for experimentalgroup; on the right: tasks with GPT feedback disabled for the experimental group).

3.4 Performance within the platform – GPT hints enabled
In order to examine the influence of the GPT hints on student performance within the platform,
we conducted an analysis of users' consecutive task attempts for tasks where GPT hints wereenabled for the experimental group (38 out of 46 tasks). The primary focus was on assessing
the percentage of successful submissions after each attempt (cumulative).
For the first attempt, the experimental group appeared to have a similar percentage ofsuccesses (48.3%) as the control group (48.0%). Values for the second and all consecutive
attempts decreased for both groups, with performance worsening across attempts. Following
the second attempt, the experimental group appeared to have a slightly higher percentage of72

successful submissions, a trend that persisted across subsequent attempts. For the purposeof clarity, only the first 15 attempts are depicted in the chart (Figure 3).

Figure 3. Comparison of the percentage of successful submissions at each attempt(cumulative) for the control and experimental group, for tasks where GPT hints were enabledin the experimental group.
We employed a linear mixed-effects model to test whether there was a significant
difference in slopes between the control and experimental groups concerning the number ofattempts. The model was fit using restricted maximum likelihood (REML) estimation with thenlme R package (Pinheiro et al., 2017). The fixed effects of the model consisted of the main
effects of group and attempt, as well as the interaction between group and attempt. Our
analysis revealed a non-significant interaction effect between group and attempt, t(13)=-0.172,
p=0.866. This indicates that the slopes of the control and experimental groups do notsignificantly differ. Nonetheless, the main effects of group and attempt were found to bestatistically significant, group: t(13)=5.38, p<0.001; attempt: t(13)=-7.38, p<0.001. This finding
suggests that, averaged across all attempts, the experimental group scored higher than thecontrol group on the dependent variable.

3.5 Performance within the platform – without GPT hints
To further evaluate the impact of the introduced platform feature on the experimental group,
we analyzed performance of both groups on the tasks where GPT-generated hints were notprovided for the experimental group (8 out of the total 46 tasks).
As in the prior section, we visualized the percentage of successful submissions following
each attempt. For the initial six attempts, the control group performed better than theexperimental group; however, starting from the seventh attempt correctness rates werecomparable (Figure 4).
A linear mixed-effects model fit using restricted maximum likelihood (REML) was
employed to assess whether the slopes of the control and experimental groups weresignificantly different with respect to the number of attempts. The model revealed a significant

73interaction effect between group and attempt, t(13)=7.13, p<0.001. This finding indicates thatthe slopes of the control and experimental groups were significantly different.

Figure 4. Comparison of the percentage of successful submissions at each attempt(cumulative) for the control and experimental group for tasks where GPT hints were disabledin the experimental group.

3.6 Time needed to successfully complete tasks
In order to assess the differences in the time required for students in the experimental andcontrol groups to successfully complete tasks on the platform, we analyzed 1,784 successful
student submissions during the study period (control: 905, experimental: 879).
We evaluated the time taken by students in both conditions to fully solve each task, while
excluding tasks completed by fewer than three students in each condition. Due to theplatform's lack of keystroke-level data monitoring, a time limit of 7,200 seconds (2 hours) wasestablished for tasks not solved within this duration. This decision was made because somestudents ceased working on tasks without logging out.
A significant majority (88%) of successful student attempts were completed within thistime frame. Due to violations of normality assumptions, we used rank-based regression – a
non-parametric alternative to traditional likelihood or least squares estimators (Kloke &McKean, 2012).
We analyzed the time performance on tasks where the GPT feedback was available forthe experimental group. For this set of tasks, the group was not a statistically significant
predictor of the time needed to solve tasks, t(89)=-0.24, p=0.811. However, the pretest score
was a marginally significant predictor, t(89)=-0.69, p=0.095.
We also conducted the analysis for tasks where the GPT feedback was not available forthe experimental group. The group was a statistically significant predictor of the time needed
to solve tasks, with the experimental group needing less time than the control group, t(48)=2.25, p=0.029. The pretest score was not significantly associated with time needed to solve
tasks, t(48)=-0.36, p=0.723. For the set of these tasks results indicate that students in theexperimental group took, on average, 375 seconds (6.25 min.) less time than the control group,
controlling for the pretest score.

74

3.7 Affective states
A total of 1,304 affect survey responses were collected from participants over the course of
the study (645 in the experimental condition, 659 control). In the analysis, we include students
who submitted at least 3 responses in this survey: in the control N=33, and in the experimentalgroup: N=35.
Both conditions predominantly reported the focused state, which constituted over half ofthe responses in each group. No significant differences between conditions were observed forthe reported frequency of any of the affective states (Table 1).
Table 1. Affective State Survey – Statistical Evaluation of Differences in Frequencies ofAffective States Reported by Students in Experimental and Control Group

FocusedFrustrated
AnxiousConfused
BoredExperimental(Mdn/Mean)
0.538/0.52
0/0.140/0.120/0.06
0/0.03Control(Mdn/Mean)
0.5/0.54
0/0.130/0.120/0.090/0.05W
statistic
537
611
550.5
571
559.5

p-value
0.620
0.653
0.718
0.935
0.782

Frustration did not significantly differ between the control (Mdn=0) and experimental(Mdn=0) groups (W=611, adjusted α=0.02, p=0.653, Mann-Whitney U test).
Similarly, no significant difference was found for boredom between the experimental(Mdn=0) and control (Mdn=0) groups (W=559.5, adjusted α=0.04, p=0.782, Mann-Whitney Utest).
The same pattern was observed for anxiety and confusion, with no significant difference
between the experimental (Mdn=0) and control (Mdn=0) groups for anxiety (W=550.5,
adjusted α=0.03, p=0.718, Mann-Whitney U test) or confusion (W=571, adjusted α=0.05,
p=0.935, Mann-Whitney U test).
Focused also exhibited no significant differences between conditions, with theexperimental group's median (Mdn=0.538) not statistically different from the control group(Mdn=0.5; W=537, adjusted α=0.01, p=0.620, Mann-Whitney U test).

4. Discussion and summary
In this study, we utilized the GPT-3.5 model to generate personalized hints for students
working on programming assignments within an automated assessment platform. Our findings
indicated that almost half of the students (46%) highly valued the usefulness of GPTgenerated hints, with a median rating of 4 or 5 on a 5-point Likert scale. Given that theassignments and generated hints were provided in Polish, these findings hold promise for the
future development and scalability of such systems across various languages, further
extending their applicability and impact in diverse educational contexts.
However, 19% of the students found the hints to be not useful, with median ratings of 1
or 2 on the same scale. Thus, there is still room for improvement and for making the hints
more useful. Considering the varied ratings of hint usefulness, we anticipate that further
optimization of the hint generation process could enhance the efficacy of GPT-based
automated feedback, ultimately leading to improved student performance. Future research
may be able to further optimize and enhance the hints, to increase the proportion of usefulhints and the degree to which each hint is useful. Potential strategies for achieving this include
more experimentation with prompt generation (prompt engineering), fine-tuning the GPTmodel (by providing examples on which the model can improve) or conducting correlation
mining to identify the properties associated with less useful hints.
Another finding is that the experimental group (with GPT hints enabled) relied less on
the platform's regular feedback for tasks where the GPT feedback was enabled. Despite of75

significantly lower usage of this kind of feedback, the experimental condition students
performed significantly better on the platform in terms of percentage of successful submissions
across consecutive attempts. There was not a significant difference in the time needed tosuccessfully solve each assignment with GPT feedback enabled. However, students in theexperimental group spent an additional average of 40 seconds after each attempt comparedto the control group. This extra time was used by the students to read and understand the hintthat was generated before they proceeded to rate it through a survey.
Furthermore, when GPT hints were made unavailable, students in the experimentalgroup needed significantly less time to successfully solve each assignment. Since hints
enabled for earlier tasks not only served to rectify the students' misconceptions but also
provided supplementary information, it is our conjecture that these hints played a pivotal rolein enhancing the students' learning process, apparently further facilitating their understanding.
This knowledge acquisition, in turn, may have expedited their task-solving abilities, leading tothe observed reduction in time required to complete tasks when the hints were no longer
available.
An unexpected outcome of this study is that when students in the experimental condition
switched over to more difficult tasks for which GPT hints were not enabled, these students
were significantly less likely to successfully solve the tasks in their first several attempts. Ourfindings indicate that users appeared to rely on the GPT-generated feedback during the study
and needed some time to adapt to a lack of this kind of feedback. This may indicate thatstudents were becoming overly dependent on the GPT-generated feedback, but if so, thisproblem seemed to correct itself fairly rapidly.
Although the GPT hints impacted student performance, the availability of these hints did
not appear to have a substantial impact on student affect.
This study presents several limitations that warrant further investigation. First, the study
duration spanned only 9 weeks of the 15-week semester term; future research should explore
the longer-term effects of GPT-generated hints on student performance and learning
outcomes. Second, the experiment focused solely on GPT-generated hints in the Polishlanguage, raising the need for additional experiments in different languages to establish thegeneralizability of the results. Third, the number of participants represents a limitation to therobustness and external validity of our findings; future work should examine this type ofintervention with a larger sample size. Finally, the study utilized a single programming
language (C#), while introductory computer science education often incorporates otherlanguages, such as Java and C++. Future research should expand the scope to include a
broader range of programming languages to more comprehensively assess the efficacy andapplicability of GPT-generated hints.
Addressing the challenge of generating personalized feedback for programming
assignments is a demanding task, given the myriad of potential code syntax errors and code
complexity across different assignments. The GPT model holds the potential to tackle thisissue more effectively and efficiently than human authoring of hints. In summary, the findingsof this study indicate that integrating GPT-generated feedback into computer programming
education may positively impact student performance, ultimately contributing to enhanced
learning outcomes.
Although this experiment employed the default GPT-3.5 model, it is plausible that finetuning the model could further improve the quality of generated hints, thereby facilitating more
effective programming education on a larger scale.

76

Acknowledgements
This paper was written with the assistance of ChatGPT, which was used to improve the writing
clarity and grammar of first drafts written by humans. All outputs were reviewed and modified
by two human authors prior to submission.

References
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerfulapproach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological),57(1), pp. 289–300.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020).
Language models are few-shot learners. Advances in neural information processing systems, 33,
pp. 1877-1901.
Brusilovsky, P., Malmi, L., Hosseini, R., Guerra, J., Sirkiä, T., & Pollari-Malmi, K. (2018). An integrated
practice system for learning programming in Python: design and evaluation. Research and practicein technology enhanced learning, 13(1), pp. 1-40.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Raffel, C. (2021).
Extracting Training Data from Large Language Models. In USENIX Security Symposium (Vol. 6).
Dai, W., Lin, J., Jin, F., Li, T., Tsai, Y., Gasevic, D., & Chen, G. (2023). Can Large Language Models
Provide Feedback to Students? A Case Study on ChatGPT. https://doi.org/10.35542/osf.io/hcgzj
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & De Weerdt, J. (2021). A review of automatedfeedback systems for learners: Classification framework, challenges and opportunities. Computers& Education, 162, 104094.
Edwards, S. H., & Murali, K. P. (2017). CodeWorkout: short programming exercises with built-in datacollection. In Proceedings of the 2017 ACM conference on innovation and technology in computerscience education, pp. 188-193.
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., and Prather, J. (2022). The robots arecoming: Exploring the implications of openai codex on introductory programming. In Australasian
Computing Education Conference, pp. 10–19
Hull, A., & du Boulay, B. (2015). Motivational and metacognitive feedback in SQL-Tutor. Computer
Science Education, 25(2), pp. 238-256.
Jackson, G. T., & Graesser, A. C. (2007). Content matters: An investigation of feedback categories
within an ITS. Frontiers in Artificial Intelligence and Applications, 158, 127.
Karumbaiah, S., Baker, R.S., Tao, Y., Liu, Z. (2022). How does Students' Affect in Virtual Learning
Relate to Their Outcomes? A Systematic Review Challenging the Positive-Negative Dichotomy. In
Proc. of the Int’l Learning Analytics and Knowledge Conference, pp. 24–33.
Kloke, J. D., & McKean, J. W. (2012). Rfit: Rank-based estimation for linear models. The R Journal,
4(2), pp. 57–64.
Maier, U., & Klotz, C. (2022). Personalized feedback in digital learning environments: Classification
framework and literature review. Computers and Education: Artificial Intelligence, vol 3.
Paiva, J. C., Leal, J. P., & Figueira, Á. (2022). Automated assessment in computer science education:
A state-of-the-art review. ACM Transactions on Computing Education (TOCE), 22(3), pp. 1-40.
Pankiewicz, M. (2020). Move in the Right Direction: Impacting Students’ Engagement With Gamification
in a Programming Course. In EdMedia+ Innovate Learning, pp. 1180-1185. Association for the
Advancement of Computing in Education (AACE).
Pardos, Z. A., & Bhandari, S. (2023). Learning gain differences between ChatGPT and human tutor
generated algebra hints. arXiv preprint arXiv:2302.06871.
Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D., Heisterkamp, S., Van Willigen, B., & Maintainer, R.
(2017). Package ‘nlme’. Linear and nonlinear mixed effects models, version, 3(1), 274.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models areunsupervised multitask learners. OpenAI blog, 1(8), 9.
Stanger, N. (2018). Semi-automated assessment of SQL schemas via database unit testing. In
Proceedings of the 26th International Conference on Computers in Education (ICCE 2018). AsiaPacific Society for Computers in Education (APSCE).

77" !   !" " "   #" !
#" !"$  #" ! #"           $   &"$%   ! &"$%   #!!
!!#"  
*-, #((%(!'!(*&,$+0(,('$. *+$,0)' &$ ', *!(* (&)-,$'"' $,-$ +0(,('$. *+$,0)'-,$(',$ '  ', *,$('%'+,$,-, !(*-,$('%(%$0 + *#
)'', *!(*''(.,$.  + *#'-,$('$',$ ' '+,$,-, !(*$ *% *,+'$ ' +0(,('$. *+$,0)'
I1=1E389 D19C59 GCD ;I?D? E 13 :@  '67AABC2G/7;A/B7;>@=D7<5B63>@3271B7=<?C/:7BG=4B63/CB=;/B71:/03:7<5
=4:3/@<7<5;/B3@7/:A/03:7<5:3/@<7<5;/B3@7/:A6/ABE=3F7AB7<57AAC3A3AB/0:7A67<5
1=;>:3B3:G /CB=;/B32 :/03:7<5 /<2 @32C17<5 ;/<C/: :/0=@ 4=@ /AA75<7<5 :/03:A B=
;/B3@7/:A/03:A=4B63;/B3@7/:A/@3CB7:7H324=@/</:GH7<5ABC23<BAP:3/@<7<5>/BB3@<A
B@/17<5 9<=E:3253 /<2 @31=;;3<27<5 3F3@17A3A B= ABC23<BA &7<13 7B 7A B==0C@23<A=;3 B= ;/<C/::G /AA75< A3D3@/: :/03:A B= ;/<G :3/@<7<5 ;/B3@7/:A /</CB=;/B71/:5=@7B6;0/A32:/03:7<5AGAB3;7A23A7@/0:3=E3D3@1:/AA7471/B7=<CA7<5
E=@23;03227<56/A=4B3<G73:232:=E3@/11C@/1G4=@;/B63;/B71A:3/@<7<5;/B3@7/:A
E7B6A6=@BB3FBA<B67A@3A3/@16E36/D31=<137D32/<27;>:3;3<B32/<7;>@=D32
/>>@=/16B=>@3271B/:/03:0G1/:1C:/B7<5B63A7;7:/@7BG=4<5@/;=4A3<B3<13ACA7<5
/11/@21=3447173<BAE3756B7<5B63;B=1@3/B3/D31B=@@3>@3A3<B/B7=</<2CA7<57BB=
>@3271B B63 :/03: =4 B63 3F3@17A3 )3 1=;>/@32 B63 /11C@/1G /<2  A1=@3 =4 B63>@3271B7=<@3AC:BA=4B63E3756B32<5@/;A7;7:/@7BG;=23:E7B6B6=A3=4B63AB/B3=4B63
/@BE=@23;03227<5;=23:)34=C<2B6/BB63<5@/;/>>@=/16E/AAC>3@7=@7<0=B6
/11C@/1G/<2A1=@3C@B63@;=@3E3>:=BB32B63D31B=@A=0B/7<324@=;3/16;=23:
7<BE=27;3<A7=</:1==@27</B3A/<2=0A3@D32B6/BB63<5@/;;=23:>@=2C132;=@3
4:3F70:3 >@3271B7=<A @35/@2:3AA =4 B63 D31B=@PA >=A7B7=< '63A3 @3AC:BA AC553AB B63
1:/AA7471/B7=<34431B7D3<3AAE7B6E3756B32A7;7:/@7BG=4<5@/;4=@;/B3@7/:AE7B6/A;/::
/;=C<B=4B3FB  CB=;/B71 :/03:7<5 E=@2 <5@/; /11/@2 1=3447173<B B&! E=@2
3;03227<5B=>710/A32:3/@<7<5 !"!
%125<9>7 <51B>9>7 =1D5B91<C 9C 1 G945<I EC54 =5D8?4 9> 54E31D9?>1< 4?=19>C CE38 1C
;>?G<5475DB139>7D?141@D8?GD?D5138D?5138CDE45>D.95$1C89=11ED?=1D93
5H5B39C5B53?==5>41D9?>CD?CE@@?BDCDE45>DCM<51B>9>7CDB1D57I,1;1=95D1< 1>4
1>1<IC9C21C54?>D85=1D5B91<D?@93/1>75D1<  /9D8D85C896DD?",54E31D9?>D?
@B5493D8?GG5<<D85CDE45>DCD13;<55H5B39C5C.95$1C89=1
D?B53?==5>4D85=?CD
1@@B?@B91D5 5H5B39C5 9> =1>I <51B>9>7 =1D5B91<C ,1;1=9 5D 1<  ?B D? ED9<9J5 D85= 6?B
<51B>9>7 @1DD5B> 1>1<IC9C /1>7 5D 1<  1 D?@93 21C54 3<1CC96931D9?> ?6 9>CDBE3D9?>1<=1D5B91<C 9C 3B9D931< D? ED9<9J5 D85= 3?>F5>95>D<I  !?G5F5B 1CC97>9>7 D85 3<1CC96931D9?> D?
@B?2<5=C=1>E1<<I9C181B4D1C;D81DB5AE9B5CD853??@5B1D9?>?65H@5BDC ">?D85BG?B4C9D9C
45C9B12<5D?<125<5H5B39C5C1ED?=1D931<<ID?B54E35D853?CD1>4D9=5?64?=19>5H@5BDCG8?
ECE1<<I@5B6?B=D85D1C; )B5F9?EC3<1CC96931D9?>CDE495C81F5D1;5>?>5B5<912<5=5D8?4G?B4
5=25449>7 @B5F9?ECCDE4I81CC8?G>D81DG?B45=25449>781C@B?4E354>?F5<B5CE<DC6?B
3<1CC96I9>7 >5GC CD?B95C 81B=1 5D 1<   !?G5F5B 1>?D85B @B5F9?EC B5C51B38 9>D?
=1D85=1D931<5H5B39C53<1CC96931D9?>B5CE<DC9><?G5B133EB13I6?B5H5B39C5C51383?>C9CD9>7
?61C=1<<>E=25B?6G?B4C,91>5D1< 78">D89CCDE4IG5@B?@?C51>1ED?=1D933<1CC96931D9?>1<7?B9D8=D81D31>81>4<55F5>
C8?BDC5>D5>35C9>=1D85=1D93C5H5B39C5C /56?3EC54?>D85?@D9=1<17B55=5>D?61C5D?6
=1D85=1D931< @B?2<5=C 1>4 31<3E<1D54 D85 C9=9<1B9DI 25DG55> D85 G5978D54 G?B4 > 7B1=
F1B91>35B5@B5C5>D1D9?>?61>I<125<545H5B39C51>4D81D?651385H5B39C5AE5BI ?B6EBD85B
E>45BCD1>49>7?6D85B5CE<DCG5EC5D +'.1>45B&11D5>!9>D?>
1>43?=@1B54
D85 @B?@?C54 29 7B1= F53D?B C@135 =?45< G9D8 D81D ?6 >?F5< G?B4 5=25449>7  /5 C5D D85
6?<<?G9>7B5C51B38AE5CD9?>1>4D13;<59D1>D85EC5?61G5978D54> 7B1=3<1CC9695B9=@B?F5D85<125<9>7?6#1@1>5C5
=1D85=1D93CAE9JJ5C3?=@1B54D?@B5F9?ECB5C51B38EC9>7CD1D5 ?6 D85 1BDG?B45=25449>7
=5D8?4C
!!"#$,85B59C1DB5>4D?G1B41>1<IJ9>7<51B>9>72581F9?B9>1>5GG1IEC9>7D85<125<C1CC97>54D?
D51389>7 =1D5B91<C  /9D8 B571B4 D? D85 EC5 ?6 651DEB5C 9> <51B>9>7 56653D9F5>5CC 1>1<IC9C 1
CDE4IB5@?BD54D81DD85@B?@?C54CICD5=1ED?=1D931<<I1CC97>54<125<CG9D8<51B>9>7=1D5B91<C
1>41>1<IJ54D851CC97>54<125<CD?49C3?F5BCDE45>DCM<51B>9>7@1DD5B>C/1>75D1< 
9F9>7=5D141D1D?5H5B39C5C6?B;>?G<5475DB139>79C1<C?18?DB5C51B38D?@93 (>5CDE4I
EC9>7=E<D9@<5B51<41D1C5DC3?>C9CD9>7?6D5>C?6D8?EC1>4C?6EC5BC1>49D5=CC8?G54D81D
B57B5CC9?> 3<1CC96931D9?> =?45<C 3?E<4 133EB1D5<I 1>4 B1@94<I 5CD9=1D5 CDE45>D ;>?G<5475
5F5>G85>CDE45>D41D19CC@1BC5<I?2C5BF54 ">1449D9?>D85CDE4IC8?G54D81DD85=?45<31>81>4<5=E<D9@<5;>?G<54755<5=5>DC1>4C9459>6?B=1D9?>CE381CD85>E=25B?6DB91<C?6
9D5=C1>4C;9<<<5F5<C.95$1C89=1 "D9C1<C?EC56E<D?31D57?B9J5=1>I5H5B39C5C6?B
B53?==5>49>71C@5396935H5B39C5D?5>81>35CDE45>DCME>45BCD1>49>7 (>5CDE4I45F5<?@54
1 B53?==5>41D9?> CICD5= G9D8 5H@<1>1D9?>C 21C54 ?> 1> 5H@<1>1D9?> 75>5B1D?B EC9>7
@1B1=5D5BC 6B?= 1 1I5C91> ;>?G<5475 DB139>7 =?45< ,1;1=9 5D 1<   ,85
B53?==5>41D9?>AE9JG9D85H@<1>1D9?>CG1CEC54=?B56B5AE5>D<I1>4G9D8=?B53?>D9>E?EC
EC5BC3?=@1B54D?1CICD5=G9D8?ED5H@<1>1D9?>C,1;1=95D1<  ,85B581C1<C?255>
B5C51B389>7 9>D? 5HDB13D9>7 <125<C 6B?= <51B>9>7 =1D5B91<C D? 6?B= ;>?G<5475 CDBE3DEB5
B5@B5C5>D1D9?>C D81D 31> 25 C8?G> D? <51B>5BC D? 9>3B51C5 D859B 1G1B5>5CC ?6 D85 CDE4I
@B?35CC<1>171>5D1<  ,85C5B5C51B385H1=@<5CC8?GD859=@?BD1>35?6@B?F949>7
<125<C D? =1D5B91<C 2531EC5 9D 9C 51C95B D? ?2D19> ?B ED9<9J5 45D19<54 9>6?B=1D9?> 12?ED D85
381B13D5B9CD93C?6D85=1D5B91<96D85I1B5<125<549>14F1>35 +DE495C ?> 1ED?=1D54 <125<9>7 ?B 3<1CC96I9>7 =5D8?4C 1<C? 6?3EC ?> B54E39>7 D85
2EB45> ?6 =1>E1< <125<9>7    CDE4I 81C 1DD5=@D54 D? 3<1CC96I EC9>7  49665B5>D <125<C D?
3<1CC96I I51BC?6=1D85=1D93C=1D5B91<C6B?=;9>45B71BD5>D8B?E788978C38??<+85>5D1< 
2 G8?C5 3<1CC96931D9?> 31D57?B95C 1B5 133?B49>7 D? D85 3?==?> 3?B5 CD1>41B4
45F5<?@542ID85->9D54+D1D5C*9DD5B
 ,85<125<?6D85CDE4I9C6?B$ 4E31D9?>9>
D85 ->9D54 +D1D5C G8938 9C >?D C?=5G81D 1@@B?@B91D5 6?B D85 #1@1>5C5 ?>5  (>5 CDE4I
@B?@?C54D85&1D8*,=?45<1=?45<3B51D542I@B5 DB19>9>7D8521C5*,=?45<5F<9>
5D 1<  ?> 1 <1B75 3?B@EC ?6 =1D85=1D93C 3?>D5>D B1>79>7 6B?= ;9>45B71BD5> CDE45>DC@B5 ;D?8978C38??<1>4E>9F5BC9DI7B14E1D5 <5F5<=1D85=1D931<3?>D5>D+85>5D1< 1 %125<9>71>43<1CC96I9>79>CDBE3D9?>1<=1D5B91<C1ED?=1D931<<I9CB5359F9>7=E381DD5>D9?>1>4
C5F5B1< B5C51B385BC 81F5 9>F5CD971D54 =5D8?4C D? C?<F5 D89C 9=@?BD1>D D1C;  (>5 @1@5B
@B?@?C541>1ED?=1D933<1CC96931D9?>=5D8?49>1=1D85=1D931<CE2:53D3<1CC96931D9?>C385=5
6?B ?B71>9J9>7 =1D85=1D931< <9D5B1DEB5 13895F9>7 1 3<1CC96931D9?> 133EB13I ?6   G8938 9C
F5BI3<?C5D?D853<1CC96931D9?>133EB13I?6DG?5HD5>C9F5@55B B5F95GC5BF935C "D1<C?5>12<54
1><12?BB54E3D9?>3?=@1B54D?=1>E1<3<1CC96931D9?>+38E2?DJ5D1<  ,89CCDE4I
C8?GCD859=@?BD1>35?6<125<9>7=1D5B91<C1ED?=1D931<<ID?B54E35D858E=1><12?B?6=1>E1<
<125<9>7 )B5F9?ECB5C51B3881C6?E>4D81DG?B45=25449>731>2556653D9F5<I1@@<954D?<125<
3<1CC96931D9?>D1C;C 81B=15D1< 81B=15D1< 6?E>4D81DG?B45=25449>7EC9>7D85
1CDD5HD=5D8?4G1C12<5D?3<1CC96I141D1C5D3?>D19>9>7 >5GCCD?B95C1>4 >5GC
D?@93CG9D8 133EB13IG89389C=?B5133EB1D5D81>1>I?D85BG?B45=25449>7=5D8?4 

79

!?G5F5BG85>9D3?=5CD?3<1CC96I9>75H5B39C5CG9D8C8?BDC5>D5>35CC5>D5>35F53D?B9J1D9?>
EC9>7G?B45=25449>781C255>6?E>4D?251<5CC56653D9F5=5D8?4 ,91>5D1< ,91>5D1<  1@@<954 G?B4 5=25449>7 1C 1 =5D8?4 D? 3<1CC96I C8?BD #1@1>5C5 5H5B39C5 D5HDC 1>4
6?E>49D13895F54 133EB13I 33EB13IG1CD85>6EBD85B9=@B?F542I3?=29>9>7D85
=5D8?4 9> 1> 5>C5=2<5 G9D8 1 ;5IG?B4C 5HDB13D9?> =5D8?4  ,89C CE775CDC D81D D85 G?B4
5=25449>7=5D8?4=978D>?D251C56653D9F56?BC8?BD5H5B39C5D5HDC ,85B56?B5 9> D89C CDE4I G5 @B?@?C5 1> 1ED?=1D93 3<1CC96931D9?> 1<7?B9D8= D81D 31>81>4<55F5>C8?BD#1@1>5C5C5>D5>35C9>=1D85=1D93C5H5B39C5C2I6?3EC9>7?>D85?@D9=1<
17B55=5>D ?6 1 C5D ?6 =1D85=1D931< @B?2<5=C 2I 31<3E<1D9>7 D85 C9=9<1B9DI 25DG55> D85
G5978D54 G?B4 > 7B1= F1B91>35 B5@B5C5>D1D9?> ?6 1>I <125<54 5H5B39C5 D5HD 1>4 D85 AE5BI
5H5B39C5D5HDC

 !/89<5 G?B4 5=25449>7 81C 255> 56653D9F5<I 1@@<954 D? =1>I D1C;C <125<9>7 =1D85=1D93C
5H5B39C5C 9> #1@1>5C5 D81D 3?>D19> 65G G?B4C 9C 381<<5>79>7 1>4 81C >?D @5B6?B=54 1C
1>D939@1D54  "> D89C @1@5B 9> 1449D9?> D? 3<1CC96931D9?> 2I G?B4 5=25449>7 G5 5H1=9>5 D85
3<1CC96931D9?>2I#1331B43?5669395>DG?B4> 7B1=D?C55969D31>@B?F9459=@B?F54@5B6?B=1>35
9>D85D1C; ?B D89C@EB@?C5G51CC97>541>1@@B?@B91D5E>9D<125<D?D855H5B39C5CDB19>54
D85=EC9>71>1<7?B9D8=1>4=145@B5493D9?>C
Word-embedding Prediction Method(Section 3.2)

%

!

"" 
""""

Data Preprocessing (Section 3.1)




!%!

 


Evaluation(Section 3.4)

!

#"!

!

#"!

 

" !

  

!# ! !$

!



N-gram Prediction Method (Section 3.3)  Which one is the betterclassification method?

$"-*  (F5BF95G?6D85=5D8?4,85 45D19<C ?6 D85 DG? 1@@B?1385C G9<< 25 5H@<19>54 9> D85 6?<<?G9>7 C53D9?>C 1C
9>4931D549>97EB5 '?D5D81DG55H@<19>D8545D19<C?6D85DG?1@@B?1385C9>D856?<<?G9>7
C53D9?>C   ,855H5B39C5D5HDCEC549>D855H@5B9=5>D1B56B?=C9H49665B5>D4979D1<=1D85=1D931<5H5B39C5
2??;C ,85C55H5B39C52??;C1B56?B 8978C38??<CDE45>DC9>133?B41>35G9D8D85D5HD2??;C
45C97>1D542ID85#1@1>5C5&9>9CDBI?64E31D9?>E<DEB5+@?BDC+395>351>4,538>?<?7I
&0,
1>41B5=1452ID85C1=53?=@1>ID81D@B?4E354D85D5HD2??;CG89381B5
1<< G5<< EC54 9> #1@1>5C5  /5 @B5@1B54 D5HD 69<5C 2I B5149>7 D5HD 41D1 6B?= @46 EC9>7 D85
)ID8?><92B1BI)46 D5HD)1<=5B E5D?@?CC92<5@B?2<5=C6B?=69<5CDBE3DEB51>4D5HD
5=25449>7 =5D8?4C )46 69<5C 1B5 =?B5 496693E<D D? ?2D19> 9> D859B 3?=@<5D5 D5HD 6?B= D81>
!,&% 6?B=1DD5469<5C1C@B5F9?EC CDE495C3?>69B=*1=1;B9C8>1>5D1< +=9D8/5EC541@13;17531<<54'179C1";541
 6?BD85=?B@8?<?7931<1>1<IC9C?6D5HD
41D1 '179C19C1@13;1756?BD85=?B@8?<?7931<1>1<IC9C?6#1@1>5C5C5>D5>35C ,89C@B?35CC
9CB5AE9B54D?5HDB13D9>49F94E1<G?B4C2531EC5D85B51B5>? 5H@<939DG?B4C5@1B1D?BCCE381C
2<1>; C@135C 9> #1@1>5C5  /85> '179C1 =?B@8?<?7931<<I 1>1<IJ5C >E=25B 4979DC 9D
B53?7>9J5C1C5D?64979DCD81D5138B53?7>9J541C1>?E> ?BDB51D9>749665B5>D>E=25BC1C
D85C1=5G5@B?35CC541C5D?64979DC9>D?1C9>7<5>E=25B1>4DB51D541<<>E=25BC1CD85
C1=5D89>7
1385H5B39C581CD85CE2:53D<125<CG8?C5@E2<9C85B1<<?31D54?>C?G5DB51D54D85=
1CD853?BB53D<125<C ,85B5G1C 1D?D1<?6 5H5B39C5C3?>C9CD9>7?6 CE2:53D<125<C1>4
5138CE2:53D<125<G1C1CC97>5425DG55> 1>4
5H5B39C5C 97EB5 C8?GC1>5H1=@<5?61>5H5B39C5@46 41D11>4D851CC97>54CE2:53D<125< ,85697EB51<C?C8?GC1@19B?65H5B39C5

80

D5HD41D1G9D8F5BIC9=9<1BC5>D5>35C1>4?D85B@19BC?6@B?2<5=41D1G9D8C9=9<1BC5>D5>35C
G5B56?E>49>D8541D1C5D,85B51B5 CE2:53DCG9D8513881F9>71>1F5B175?6  5H5B39C5C 1385H5B39C5
3?>D19>C& 1>4+ =?B@85=5CG9D81=9>9=E=?6  1>41=1H9=E=?6
?33EBB9>7 9> D85 41D1C5D D81D G1C 3?<<53D54  ,89C C8?GC D85 <9=9D54 1=?E>D ?6 9>6?B=1D9?>
3?>D19>549>D8541D1C5DG8938D853<1CC9695B31>EC5D?1ED?=1D931<<I1CC97><125<C ?=@1B54
D?>7<9C8=1D85=1D93CAE9JJ5CD85>E=25B?6=?B@85=5C 9CC=1<<5B      *%   $!    (  # *) "! %! &+' &,'#!
    ()#    %&*'             &+'              &,'                       

 

&*'
&,'

&+'                                   #     ##       

                               &+'                                 
 # #      ##  
&,'                                                     #  ##  
# #              #     &*'   

$"-*  H1=@<5?65H5B39C5C9>D8541D1C5D*978D>7<9C8DB1>C<1D9?> ,85CE2:53D<125<K>E=25B1>46?B=E<1L9C1CC97>54D?1>5H5B39C59>D85697EB5$     ?BF53D?B9J1D9?>G9D8G?B45=25449>7G5EC541=?45<31<<5461CD,5HD#?E<9>5D1<  
=?>7 D85 61CD,5HD =?45<C 6?B  <1>7E175C G5 EC54 D85 #1@1>5C5 =?45< 9> D89C
5H@5B9=5>D ,89C=?45<3?=29>5CD8B55=5D8?4CD?B5@B5C5>D1C9>7<5C5>D5>3541D19> 
49=5>C9?>C 381B13D5B  7B1= G5978D9>7 2I @?C9D9?> 1>4 /?B4 .53 8EB38  I
3?>C945B9>7 5138 G?B4 1C 1 C5D ?6 9DC 3?>CD9DE5>D CE2 G?B4C 9D ?665BC >?D12<I 9=@B?F54
@5B6?B=1>35 G85> 451<9>7 G9D8 <1>7E175C D81D 81F5 3?=@<5H =?B@8?<?7931< CDBE3DEB5C
9>3<E49>7 #1@1>5C5 $81> 5D 1<   ,? 5F1<E1D5 3<1CC96931D9?> @5B6?B=1>35 EC9>7 G?B4
5=25449>7G5EC5421C5<9>5=?45<CG893881F5 49665B5>D381B13D5BC 9F5 =?45<CG5EC54
1B51C6?<<?GC 0 ??CD85> E5CDB9>
1=?45<D81D3?=29>5C2??CD9>71>44539C9?>DB55C
1>481CC8?G>7??4B5CE<DC9>F1B9?EC>1DEB1<<1>7E175@B?35CC9>7D1C;CC?9DMC@B?@5B
D?EC59>D853?>D5HD?6D89C@1@5B
 *1>4?= ?B5CD B59=1>
 1 =?45< D81D EC5C =1>I 4539C9?> DB55C DB19>54 2I
B1>4?=<IC1=@<54DB19>9>741D1 ,89CG?B;CG5<<5F5>G9D81<1B75>E=25B?65H@<1>1D?BI
F1B912<5CC?9D31>451<G9D81 49=5>C9?>1<F53D?B
 +E@@?BD.53D?B&1389>5.1@>9;1>4%5B>5B 1=?45<6?B3?>CDBE3D9>71DG? ?B=?B5 3<1CC @1DD5B> 49C3B9=9>1D?B EC9>7 <9>51B 9>@ED 5<5=5>DC  ,89C 9C ?>5 ?6 D85 =?CD@?@E<1B=5D8?4CD?3<1CC96IC?=5D89>7C?9D9C1@@B?@B91D5D?EC51C121C5<9>5
 %?79CD93*57B5CC9?>?H 1CD1D9CD931<B57B5CC9?>=?45<G9D8F1B912<5CD81D6?<<?G
15B>?E<<949CDB92ED9?> CG5<<1C+.&9D9C?>5?6D85=?CD@?@E<1B=5D8?4CD?3<1CC96I
C?=5D89>7C?9D9C1@@B?@B91D5D?25DB51D541C121C5<9>5
 &E<D9<1I5B)5B35@DB?> 1B4>5B?B<9>7 1=?45<D81DEC5C65546?BG1B41BD969391<>5EB1<>5DG?B;C >5EB1<>5DG?B;9C1<C?DB51D541CD8521C5<9>56?BD85=1389>5<51B>9>7
D1C;CC?G5EC59D9>D89C5H@5B9=5>D

81



     

/53B51D54D85G?B4> 7B1=6B?=D85D?;5>9J545H5B39C5C5>D5>35C ,85@B5@1B5441D1C5D
9C49F94549>D?<125<41D11>4AE5BI AE5BI=51>C1>E><125<545H5B39C51>4D85C9=9<1B9DI
25DG55>D85C5D?6G?B4> 7B1=C9>D85<125<5441D11>4D85C5D?6G?B4> 7B1=C9>D85AE5BI
41D19C31<3E<1D54 /545C3B92548?GD?38??C5?>5CE2:53D6?B5138>?D I5D <125<545H5B39C5
9>97EB5  ">D85697EB5# C 9>7B55>2?H5CB5@B5C5>DD85> 7B1=C5D?6D85<125<545H5B39C5
D5HD1>4# C 9>2<E52?H5C4?D81D?61AE5BI /55F1<E1D54D85C9=9<1B9DI25DG55>#

1>4# 2I31<3E<1D9>7D85#1331B43?5669395>DC?6D8?C5 ,85>G5G5978D54 C1>46?E>4

1F1<E5! 3?BB5C@?>49>7D?<125< "6D85F1<E5! 9CD8589785CD?61<< ?6! D85>D85<125< G1C@B5493D54 
?B F53D?B9J1D9?> EC9>7 D85 G?B4 > 7B1= G5 EC54 1 6E>3D9?> 9> ?EB 1<7?B9D8=  C
@B5F9?EC CDE495C 9=@B?F5 133EB13I 2I G5978D9>7 6?B B51<9CD93 >?> 8?=?75>5?EC 41D1 C5DC
 B1?F135D1< $?21I1C89
G5=?496954D85G5978DC?6D8531<3E<1D54> 7B1=
C9=9<1B9DI D? 3B51D5 1 =?B5 133EB1D5 3<1CC9695B  /5 3B51D54 DG? 6E>3D9?>C 21C54 ?> DG?1CCE=@D9?>C 1C6?<<?GC• CCE=@D9?>  ,G? 5H5B39C5C D81D 81F5 D85 C1=5 <125< 1B5 C9=9<1B D? 5138 ?D85B ,85B56?B5G53B51D54  D?69>4D85=?CD@B?@5B<125<3?>C945B9>71<<<125<54
5H5B39C5CM C9=9<1B9DI• CCE=@D9?>  C@5396935H5B39C5CG9D8D85C1=5<125<81F5 8978C9=9<1B9DI G9D85138
?D85B  ,85B56?B5 G5 3B51D54"  D? 69>4 D85 =?CD @B?@5B <125< 3?>C945B9>7 =
<125<545H5B39C5CMC9=9<1B9DI/5CE2CD9DED5 6?B1<<<125<5441D1 G9D8<125< 1<<9>D?D8545D5B=9>546E>3D9?>
D? ?2D19> D85 G5978D 3?5669395>D ! ?6 AE5B95C  D81D 1B5 1CC97>54 D81D <125<  !5B516D5B
!  B5@B5C5>DC D85D8 89785CD F1<E5 ?6 !  !
 !  "> D89C 5H@5B9=5>D D85
6?<<?G9>76E>3D9?>CG5B54569>54D?45D5B=9>51=?B5CE9D12<5G5978D9>76?B3<1CC96931D9?>
   " 





 

)' ($%  &*  

/5D85>@B5493D1<125< "! 6?B5138AE5BI5H5B39C5 2I38??C9>7D8589785CD3?5669395>D?6!    "! 
!



$"-*

 ,85G1ID?69>4D85G5978D543?5669395>D1>4@B5493D1<125<82 "! ,?@B5F5>D291C9>D8541D1C5D4E5D?B1>4?=C1=@<9>7G5@5B6?B=543<1CC96931D9?>DB19>9>7
G9D8 ?6D855>D9B541D1C5D1CDB19>9>741D11>4D85B5CD1CD5CD41D1EC9>7 6?<416?B=?6
3B?CC F1<941D9?> ,89C1<<?GCECD?DB19>69F5D9=5C?>?>541D1C5D ,?5F1<E1D5D8556653D9F5>5CC?6?EB@B?@?C54=5D8?4G53?=@1B59DD?5H9CD9>7G?B4
5=25449>7=5D8?4CEC9>7D856?<<?G9>7DG?=5DB93C133EB13I1>4C3?B5 33EB13I1>4
C3?B5C ?> 29>1BI @B5493D9?>C 1B5 DI@931<<I 31<3E<1D54 2I 133EB1D5 @B5493D9?> 49F9454 2I 1<<@B5493D9?>C1>4D8581B=?>9J541F5B175?6@B539C9?>1>4B531<<B5C@53D9F5<I2EDC9>35D85
=?45<9>D89CCDE4I@B?@?C5485B5@5B6?B=C=E<D9 3<1CC3<1CC96931D9?>9D9C>535CC1BID?EC51
31<3E<1D9?>=5D8?43ECD?=9J546?B=E<D9 3<1CC3<1CC96931D9?> ?B5F1<E1D9?>?6D85B5CE<D9>D89C
CDE4I D85 133EB13I 1>4 D85 @B539C9?>  D85 B531<<  1>4 D85  C3?B5 6?B 1<< <125<C
G5B531<3E<1D54EC9>7D856?<<?G9>75AE1D9?>+?B?G5B      

 
 
 
 







 

'(
 

 

'(





'(

















?B6EBD85B5H1=9>5D855H@5B9=5>D1<B5CE<DCG51<C?=1@@54D851CC97>54<125<CD?
1DG? 49=5>C9?>C@1356?BF9CE1<9J1D9?>2IEC9>7D +' ,85D538>9AE5D +'9C?>5>?F5<D538>9AE5D81D31>@B?:53D8978 49=5>C9?>1<41D19>D?DG?49=5>C9?>C1>4=1@D85=.1>45B
&11D5>!9>D?>
 (>5CDE4I3?=@1B548?GG5<<D853<ECD5B9>7?6D8549665B5>D=5D8?4C
G9D8?2C5BF9>7D85F9CE1<9J1D9?>?6D85B5CE<DG9D8D +'1>46?E>4D81D?>5=?45<D81DEC5C
C5>D5>355=25449>7G9D8;>?G<54757B1@85=25449>7?ED@5B6?B=C1>?D85B=?45<D81D?><I
EC5CC5>D5>355=25449>785>5D1<  "!   ! 
97EB5C8?GC1@<?D?6D85133EB13I6?B5138G5978D6E>3D9?>6?B5138F1<E5?69>D85>
7B1= /5E>45BCD1>46B?=D85697EB5D81DD8525CD133EB13IG1C13895F546?B297B1=  
6?B1<<6E>3D9?>C ,12<5 C8?GCD85133EB13I1>4C3?B56?B5138G5978D6E>3D9?>6?B297B1= /56?E>4D81DD85@B?@?C541<7?B9D8=@B5493DCD8525CDG85>G5EC5D856E>3D9?>" 
,12<5  %,$('+#$) ,/ '/ $"#,!-',$(''-*0$'$"*&33EB13I
C3?B5E>3D9?>  "   "  " 
 
 
" 



" 

 
" 

"  "   "   " 

83

$"-*   *5<1D9?>C89@25DG55>>F1<E59>> 7B1=1>4133EB13IG9D81<< =?45<C G5EC54,12<5  C8?GC D85 133EB13I 1>4  C3?B5C 6B?= 5138 ?6 D85C5 5H@5B9=5>DC EC9>7 1
G?B4 5=25449>7 1>4 =1389>5 <51B>9>7 =?4E<5C  /5 14454 D85 1<7?B9D8= G9D8 D85 25CD
3<1CC96931D9?>133EB13I9>5138> 7B1=5H@5B9=5>D6?B3?=@1B9C?>,12<5  -*0'+(* (!,# )* $,$('-+$'"/(* & $'"/$,#,/( ++ ',$%
* +-%,+(!,# )* $,$('-+$'"'"*&51DEB5C/?B45=25449>7/?B45=25449>7/?B45=25449>7/?B45=25449>7/?B45=25449>7' 7B1=  
' 7B1=  &?45<C
0 ??CD%?79CD93*57B5CC9?>)5B35@DB?>*1>4?=?B5CD+.&#1331B4?5669395>D 
#1331B4?5669395>D"33EB13I
 
 

 




 C3?B5

 






    #  

97EB5 C8?GCD85=1@@9>7?6D85B5CE<DC?63<1CC96931D9?>EC9>729 7B1=#1331B4 " 1>4D81DEC9>7G?B45=25449>70 ??CD ">D85697EB55H5B39C5CG9D849665B5>D<125<C1B5
B5@B5C5>D542I49665B5>D3?<?BC1>4CI=2?<C ">1""2"" ?6D85697EB55H5B39C5CG9D8GB?>7
@B5493D9?>C1B5=1@@549>7B1I3?<?B%??;9>71D1"1>42"9>97EB5 G531>C55D81D2?D881F517B?E@?65H5B39C5C
G9D8D85C1=5<125<C9>3<?C5@B?H9=9DID?C?=55HD5>D /531>?2C5BF5D81DD8535>D5B?61"
81F5 1F5BI<1B75>E=25B?65H5B39C541D1G9D849665B5>D<125<C49CDB92ED54D978D<ID?75D85B1>4
6B?=1""D81DD85I1B51<=?CD1<G1IC9>3?BB53D<I@B5493D54 !?G5F5B2"4?>?DC8?GCE38
@?BD9?>C?6D8541D1 ,89C?2C5BF1D9?>CE775CDCD81DD85G?B45=25449>7=5D8?4CDBE77<5CD?
133EB1D5<I<125<AE9JJ5CD81D1B53<?C5D?=1>I49665B5>DDI@5C?65H5B39C5C G85>3?=@1B54D?
EC9>7D85@B?@?C54> 7B1=3<1CC96931D9?>=5D8?4%??;9>7 1D 1"" 1>4 2"" 9> 97EB5  G5 C55 D81D 6?B 2?D8 C5DC ?6 41D1 G85B5 D85
@B5493D9?>C1B53?BB53D5H5B39C5CG9D8D85C1=5<125< 1B549CDB92ED549>3<?C5@B?H9=9DID?5138
?D85BD?C?=55HD5>D ">1""D85B51B5=1>I1B51CG85B59D5=C?6D85C1=5DI@51B53<ECD5B54
D?75D85B1>4165G1B51CG85B59D5=C?649665B5>DDI@5C1B53<?C5D?5138?D85B (>D85?D85B
81>49D31>25?2C5BF54D81D9>2""D85B51B5=1>I<?31D9?>CG85B59D5=C?649665B5>DDI@5C
1B5 <?31D54 3<?C5 D? 5138 ?D85B  ?B 5H1=@<5 D? 3?=@1B5 G9D8 D85 <125< 49CDB92ED9?> 9> D85
DB1>C@1B5>DCAE1B5?61""1>42""6?B<125<CG9D8<125<C134;?1>4E9>1""5138<125<
1@@51BC D? 49CDB92ED5 9> 1 4569>9D5 B579?> 1>4 5138 B579?> 1@@51BC D? 81F5 >? 3?BB53D
@B5493D9?>C6?B41D1G9D8?D85B<125<C ">3?>DB1CD9>2""G5?6D5>69>4C=1<<B579?>CG85B5
D85<125<C1B549665B5>D2EDD85@B5493D9?>C1B53?BB53D ,89C133?E>DC6?BD85613DD81D@B5493D9?>
2IG?B45=25449>79CC5>C9D9F5D?D85@?C9D9?>3?BB5C@?>49>7D?D85F53D?BB5@B5C5>D1D9?>?6

84

D85 41D1 G85B51C > 7B1= C9=9<1B9DI 9C 12<5 D? =1;5 @B5493D9?>C B571B4<5CC ?6 D85 @?C9D9?>
3?BB5C@?>49>7D?D85F53D?BB5@B5C5>D1D9?>?6D8541D1

$"-*



 "

&1@C?6D85F53D?BC?2D19>546B?=51385H@5B9=5>DG5B5DB1>C6?B=549>D?DG?
49=5>C9?>CEC9>7D +'1>4D1;5>?>1@<1>5!!

,85 B5CE<DC 9> ,12<5  C8?G D81D @B5493D9?> EC9>7 1 G5978D9>7 6E>3D9?>" 9C 12?ED =?B5133EB1D5D81>3<1CC96931D9?>EC9>7D856E>3D9?> 1>41CD857B1@8C8?GCG531>
C1ID81D?D85BG5978D=?45<1<C??ED@5B6?B=C  3<1CC96931D9?> ,89C9>4931D5CD81D>?D1<<@B?2<5=C 3<1CC96954 E>45B D85 1@@B?@B91D5 <125< 1B5 C9=9<1B C5>D5>35C 1>4 D81D <51B>9>7 1<<
@B?2<5=C=1IB5CE<D9>>?9C5 ">D855H@5B9=5>DG53<1CC96ID855H5B39C5C9>D? 3<1CC5CG9D8 AE5CD9?>C@5B3<1CC1>4D85@EB@?C5?6D85G5978D6E>3D9?>9CD?45D5B=9>58?G=1>I<125<5441D1D5HDC1B5<9;5 D85AE5BID5HD (>5@9535?65F945>35CE@@?BD9>7D89C9CD81DD85B5
1B5?6D5>5H5B39C5C9>D8541D1C5DD81D81F5F5BIC9=9<1BC5>D5>35C1>41B579F5>D85C1=5
<125< ,81>;CD?D85@B5C5>35?6CE385H5B39C5CG5G5B512<5D?13895F518978<I133EB1D5
=E<D9 3<1CC3<1CC96931D9?>5F5>G9D81C=1<<41D1C5D/5 1<C? 6?E>4 D81D D85 =?45< G9D8 > 7B1= 71F5 12?ED  89785B 133EB13I 1D D85
3<1CC96931D9?> D1C; CE38 1C " 1719>CD +.&  ,89C 9>4931D5C D81D 1 G5978D54 > 7B1=
C9=9<1B9DI=?45<9C=?B556653D9F56?BD853<1CC96931D9?>?6C8?BD=1D85H5B39C5C">97EB5 1"1>42"2?D8=?45<CG5B5?2C5BF54D?81F5B579?>C1CC97>54D?E>9DC <C?
97EB5 C8?GCD81DD85G?B4 5=25449>7C=?45<MC=1@81CB579?>C1CC97>54D?D85E>9DD?
C?=55HD5>DC?D81D3<E=@C?65BB?BC1B5>?D93512<59>1""2EDD89C9C>?DD8531C56?BD85
@B5493D9?> 9> 2""  ,89C 9>4931D5C D81D 3<1CC96931D9?> EC9>7 > 7B1= 9C =?B5 6<5H92<5 9> =1;9>7
@B5493D9?>C(>5<9=9D1D9?>?6D89C5H@5B9=5>D9CD81D1<D8?E789>D89C5H@5B9=5>DG5EC5DG?;9>4C
?6 G5978D9>7 6E>3D9?>    1>4 "  G5 31> 3?>C945B ?D85B G5978D9>7 6E>3D9?>C 
H@5B9=5>D1<B5CE<DC1B55>?E78D?C8?GD8525DD5B@5B6?B=1>35?6 D85> 7B1=B5@B5C5>D1D9?>
=?45< D81> D85 G?B4 5=25449>7 =?45< 2ED G5 31> 5H@53D 1>?D85B =?45< I95<4C 1 =?B5
133EB1D5@B5493D9?>85 "  "!" "> D89C CDE4I G5 @B?@?C5 1> 1ED?=1D93 3<1CC96931D9?> 1<7?B9D8= D81D 31> 81>4<5 5F5> C8?BDC5>D5>35C 9> =1D85=1D93C 5H5B39C5C 2I 6?3EC9>7 ?> D85 ?@D9=1< 17B55=5>D ?6 1 C5D ?6
=1D85=1D931< @B?2<5=C  /5 3?=@1B54 D85 @B?@?C54 =?45< G8938 31<3E<1D5C D85 #1331B4
3?5669395>DD?5F1<E1D5D85C9=9<1B9DI?6> 7B1=G9D8D85@B5F9?EC<I5H9CD9>7G?B45=25449>7
=?45< H@5B9=5>D1<B5CE<DCC8?GD81DD8556653D9F5>5CC9><125<9>7=1D85=1D931<5H5B39C5C?6
D85 @B?@?C54 1<7?B9D8= EC9>7 G5978D54 > 7B1= 5H3554C D81D ?6 G?B4 5=25449>7 G9D8 1>
133EB13I?6 F5BCEC 1>41>C3?B5?6  F5BCEC  B5C@53D9F5<I /51<C?3?=@1B54D85DG?=5D8?4CEC9>7D +'1>46?E>4D81DD85@B?@?C5429 7B1=F53D?B
C@135 =?45< =145 =?B5 6<5H92<5 @B5493D9?>C B571B4<5CC ?6 D85 @?C9D9?> ?6 D85 F53D?B
DB1>C6?B=542ID +' ,85B51B5C?=5<9=9D1D9?>C?6D853EBB5>DCDE4ID81DC8?E<425=5>D9?>54CE381C8?G
D85=1D5B91<CEC546?BD855H@5B9=5>D1B51<<6B?=?><I?>5@E2<9C85BG89383?E<481F5<514D?
D858978133EB13I9>D85B5CE<DC 449D9?>1<5H@5B9=5>DCC8?E<42531BB954?ED9>6EDEB5G?B;
D?3?>69B=G85D85BD85133EB13I1>4C3?B531>25=19>D19>54G85>EC9>7=1D5B91<C6B?=
F1B9?EC@E2<9C85BC 
>?D85B1B51D81DC8?E<41<C?259>F5CD971D549C3B51D9>71=?45<D81D1<<?GC=E<D9@<5<125<C D? 25 1CC97>54 D? D85 C1=5 895B1B38I 1>4 9>3B51C5C F5BC1D9<9DI 6?B =1D85=1D931<@B?2<5=C ">D855H@5B9=5>DG5@B5@B?35CC54D8541D1D?DB51D1<<4979D5H@B5CC9?>C1CD85
C1=5C?9>6EDEB5G?B;5H@5B9=5>DCC8?E<42531BB954?EDD?3853;D859>6<E5>35?61>1<IJ9>7
=1D85=1D931<5H@B5CC9?>C?>D853<1CC96931D9?> /51<C?>554D?3?>4E3D1>?D85B5H@5B9=5>D
D?AE1>D96ID85B54E3D9?>9>=1>E1< <125<9>72EB45>1CG5<<1CD8549665B5>359>D85B1D5?6
=9C<125<<9>76B?=D85@5BC@53D9F5?6D8556653D?>54E31D9?>1<3?>C5AE5>35C &?B5?F5B D85 @B?@?C54 =5D8?4 >554C D? 144B5CC 5H5B39C5C D81D 31> 25 1CC97>54
=E<D9@<5<125<C &E<D9 <125<3<1CC96931D9?>9C1<C?G945<IEC549>=1389>5<51B>9>7,C?E=1;1C $1D1;9C  ,85B5 1B5 C9DE1D9?>C G85B5 =E<D9@<5 <125<C 1B5 79F5> 2531EC5 ?>5=1D85=1D931<@B?2<5=81C=E<D9@<5;>?G<54755<5=5>DC $! 
,89CG?B;G1C@1BD<ICE@@?BD542I#+)+ B1>D 9> 946?B+395>D9693*5C51B38#) !  
1>4#) !
H@<?B1D?BI#) $  1B<I1B55B#) $ #) !  
1>4'(#)') 
@37;/<
%/<2=;4=@3ABA(*/04,,(7404. >> 
63<$)/<5)C/<5=3<3<
,3@=A6=BB3FB1:/AA7471/B7=<D7/9<=E:32535@/>6
3;03227<54=@A=17/:;327/2/B/49,74,95-$/04.85:74(2 >> 63<'C3AB@7<
*50==ABA1/:/0:3B@330==AB7<5AGAB3;!75*,,+04.85-9/, 4+
#49,74(9054(254-,7,4*,5445<2,+.,08*5;,7=(4+(9(0404.>> 
6C@16)
)=@2 (31(9:7(2(4.:(.,4.04,,704.  >>   
=F% '63@35@3AA7=</</:GA7A=407</@GA3?C3<13A5:74(25-9/,"5=(2#9(90890*(2#5*0,9=
#,70,8,9/5+525.0*(2  >> 3D:7<  6/<5  33  /<2 '=CB/<=D/   %' #@3B@/7<7<5 =4 233> 0727@31B7=</:
B@/<A4=@;3@A 4=@ :/<5C/53 C<23@AB/<27<5 !75*,,+04.8 5- 9/,  54-,7,4*, 5- 9/, 579/3,70*(4 /(69,7 5- 9/, 885*0(9054 -57 536:9(9054(2 04.:0890*8 :3(4 (4.:(.,$,*/4525.0,8>>   
6/@;/ /=:)/@</@A&&=3E7B=
'63/11C@/1G1=;>/@7A=</;=<5
E=@2 D315:=D3/<24/ABB3FBB=E/@2A1=<D=:CB7=<<3C@/:<3BE=@9!!B3FB1:/AA7471/B7=<5:74(2
5-$/,57,90*(2(4+6620,+4-573(9054$,*/4525.= >>  
:/</5/< /8C;2/@%9I/>Q</@)/<5"5/B/
<=E:3253;/>1@3/B7=<4=@;=23:7<5:3/@<7<5036/D7=@A7<2757B/::3/@<7<53<D7@=<;3<BA536(4054!75*,,+04.85-9/,9/49,74(9054(254-,7,4*,54,(7404.4(2=90*8(4+45<2,+.,>>  86

/@2<3@ )=@:7<5&% @B74717/:<3C@/:<3BE=@9AB63;C:B7:/G3@>3@13>B@=< O /
@3D73E=4/>>:71/B7=<A7<B63/B;=A>63@71A173<13A93586/,70*,4;07543,49   >>   
@/=D/1=D/V3D7U#/D:=D7U/T3B7U
/<5C/537<23>3<23<B<5@/;0/A32B3FB
1/B35=@7H/B7=<E7B6E3756B7<54/1B=@A1/A3ABC2G5:74(25-4-573(9054(4+(9((4(.,3,49 >> 
932/'</57A/  6BB>A57B6C01=;B/7A677</57A/=C:7<  @/D3  =8/<=EA97 # =CH3  J5=C   79=:=D '  /5 =4 B@719A 4=@
3447173<BB3FB1:/AA7471/B7=<!75*,,+04.85-9/, 9/54-,7,4*,5-9/,:756,(4/(69,75-9/,
885*0(9054-57536:9(9054(204.:0890*8 >> 6/<  /<3;/@C   &/9/;C@/  
 &3<B7;3<B /</:GA7A =4 />/<3A3 BE33BA CA7<5/CB=/C5;3<B32A3<B7;3<B>=:/@7BG271B7=</@73A/<2/2D/<132E=@23;03227<5  9/25)(254-,7,4*,54548:3,72,*97540*8>>   =0/G/A67'
'D A7;7:/@7BG4=@@35C:/@7H7<57<B@/1:/AA43/BC@327AB@70CB7=<!75*,,+04.85-
9/,&54-,7,4*,54536:9,7&08054(4+!(99,74",*5.409054>>  
7<7AB@G =4 @BA /<2 &173<13A  *'  高等学校用教科書目録(令和４年度使用) [=@ 7563@2C1/B7=< '3FB0==9 /B/:=5 4=@ 7A1/: +3/@- 6BB>AEEE;3FB5=8>1=<B3<B ;FB.9G=9/AG=  . >24
#/:;3@
>24B=B3FB   6BB>A57B6C01=;8/:/<>24B=B3FB
%/;/9@7A6</<#/B<7/=DGC@<A
/G=CB/E/@3B3FB3FB@/1B7=<4@=;4C::
B3FB#=4A173<B7471/@B71:3A#5:7*,*5+,-57)0525.=(4+3,+0*04, >>  
%7BB3@    %6+(9, 54 9/, *53354 *57, 89(9, 89(4+(7+8 04090(90;, !/B7=</: =D3@<=@AAA=17/B7=<%=A3<0:/BB '63>3@13>B@=<>@=0/07:7AB71;=23:4=@7<4=@;/B7=<AB=@/53/<2=@5/<7H/B7=<
7<B630@/7<!8=*/525.0*(27,;0,< >> &16C0=BH &16/@>4#'3A1693"K6<3;C<2@37B7<53@7>>CB=;A1
CB=;/B71 /AA75<;3<B =4 ;/B63;/B71A AC0831B 1:/AA7471/B7=< :/03:A 49,74(9054(2 54-,7,4*, 5449,220.,49536:9,7(9/,3(90*8>>   
&63<'+/;/A67B/ #@76/@3443@</<!)C*@/4433
 /B6%'#@3B@/7<32 /<5C/53 =23: 4=@ 3<3@/: !# '/A9A 7< /B63;/B71A 2C1/B7=< ,:7!#  
(9/-57+:*(9054'5718/56
&63<'+/;/A67B/ #@76/@3443@</<!)C* 1@3E&33
0:/AA74G7<5;/B6 9<=E:3253 1=;>=<3<BA D7/ B/A9/2/>B7D3 >@3B@/7<32 03@B 49,74(9054(2 54-,7,4*, 54
790-0*0(249,220.,4*,04+:*(9054>>  &;7B6 %  < =D3@D73E =4 B63 B3AA3@/1B "% 3<57<3 049/ 49,74(9054(2 54-,7,4*, 545*:3,494(2=808(4+",*5.409054>>  &=@=E3@  &   :7B3@/BC@3 AC@D3G =< /:5=@7B6;A 4=@ ;C:B7:/03: :3/@<7<5 7,.54 #9(9,%40;,7809= >> '/9/;7  /7 + :/</5/<   7@=/97 "5/B/  2C1/B7=</: 3F>:/7</0:3 @31=;;3<23@
CA/53/<27BA34431B7D3<3AA7<6756A16==:AC;;3@D/1/B7=</AA75<;3<B  9/49,74(9054(2,(7404.4(2=90*8(4+45<2,+.,54-,7,4*,
'/9/;7:/</5/</7+"5/B/
'=E/@232C1/B7=</:3F>:/7</0:3@31=;;3<23@
AGAB3; F>:/</B7=< 53<3@/B7=< 0/A32 =< 0/G3A7/< 9<=E:3253 B@/17<5 >/@/;3B3@A 9/49,74(9054(254-,7,4*,54536:9,7804+:*(905454-,7,4*,!75*,,+04.8 >> N '7/<,:/</5/</7+"5/B/CB=;/B32;/B167<5=43F3@17A3AE7B69<=E:3253
1=;>=<3<BA !75*,,+04.8 5- 9/, 9/ 49,74(9054(2 54-,7,4*, 54 536:9,78 04 +:*(9054
54-,7,4*,>>  
'A=C;/9/A   /B/97A   C:B7:/03: 1:/AA7471/B7=< < =D3@D73E 49,74(9054(2 5:74(2 5-(9('(7,/5:804.(4+0404.  >>  (/<23@ //B3<7<B=<
(7AC/:7H7<52/B/CA7<5B&!5:74(25-3(*/04,2,(7404.
7,8,(7*/ >>   (/><79(3@<3@  #/BB3@<@31=5<7B7=<CA7<553<3@/:7H32>=@B@/7B;3B6=2:953(9054(4+",359,549752 >>  (73/A67;/
<=E:3253B@/17<5;/167<3A/1B=@7H/B7=<;/167<3A4=@9<=E:3253
B@/17<5<!75*,,+04.85-9/,54-,7,4*,54790-0*0(249,220.,4*,  >> 
)/<5  7<3;/BAC ' "9C0=   &67;/2/  
 '=>71E7A3 @3>@3A3<B/B7=< =4 :3/@<7<5/1B7D7B73A 4=@ <3E :3/@<7<5 >/BB3@< /</:GA7A 9/ 49,74(9054(2 54-,7,4*, 54 536:9,78 04
+:*(905454-,7,4*,>>  87

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Can We Ensure Accuracy andExplainabilityfor a Math Recommender
System?
Yiling DAIa*, Brendan FLANAGANb & Hiroaki OGATAa
Academic Center for Computing and Media Studies, Kyoto University, Japanb
Center for Innovative Research and Education in Data Science, Institute for Liberal Arts
and Sciences, Kyoto University, Japan*dai.yiling.4t@kyoto-u.ac.jpaAbstract: Providing explanations in educational recommender systems are supposed
to increase students’ awareness of the recommendations, trust toward the system,
motivation to adopt the recommendations. With the expectation to have a higher
prediction accuracy, more and more complex recommendation models are developed,
which are difficult to explain. It remains debatable that whether there exists a trade-off
between the accuracy and explainability of recommender systems. In this study, we
focus on the explainable math quiz recommender system--- Naïve Concept Explicit(Naïve CE) proposed in our previous work. We are interested in knowing whether the
explainable Naïve CE has a good prediction accuracy compared with a powerful but
less explainable model--- Matrix Factorization (MF). We also proposed a combinedmodel CE+MF to preserve the explainability of Naïve CE and predicting power of MF.
We then used a long-term quiz answering dataset to evaluate the models’ accuracy as
to predicting students’ correctness rate of the quizzes. The results revealed that 1) The
explainable model Naïve CE had a lower accuracy than the less model MF given thesparse dataset; 2) Combining two models achieved a moderate accuracy in predicting
students’ answers while preserving the explainability of Naïve CE. Our study served as
an example of how to develop an inherently explainable educational recommendersystem and how to improve the accuracy by integrating more complex models.
Keywords: Recommender system, math quiz, explainability, accuracy, matrix
factorization

1. Introduction
To improve learning, educational recommender systems provide recommendations based
different criterion including learning activities (Afzaal et al., 2021), knowledge states (Ai etal., 2019; Tang et al., 2019), learning goals (Heras et al., 2020; Huang et al., 2019), learning
styles (Heras et al., 2020; Klašnja-Milićević et al., 2011) , student profiles (Shanshan et al.,2021) and so on. Unlike consuming entertaining products such as movies or music, reading
learning materials or solving quizzes requires higher levels of motivation and cognitive
investment of users. Providing explanations of the recommendations to the students is
considered as a solution. Some promising results of the explanations’ effects have beenfound: increasing the students' attention towards recommended practices, the willingness toopen them (Barria-Pineda et al., 2021); increasing the students' trust in hints, perceived
usefulness of them, and the intention to use them again (Conati et al., 2021); increasing thestudents' perceived unexpectedness, novelty of recommended courses, and the interests inthem (Yu et al., 2021); increasing the students' usage of the recommended quizzes (Dai,Takami, et al., 2022; Takami et al., 2022).
Unlike traditional machine learning models such as decision tree and logistic
regression, recent models involving neural networks are becoming more and more difficult tounderstand (Khosravi et al., 2022). It is commonly considered that more complex models
have better performance in terms of predicting user’s behavior (Molnar et al., 2022; Rudin,88

2019). To achieve model transparency, explaining complex models becomes a research
field also known as “eXplainable Artificial Intelligence (XAI)” (Arrieta et al., 2020). However,there remain two open issues before suggesting researchers and practitioners to develop
more complex models and then struggle to explain it:
1. It is doubted that there is a trade-off between the model’s accuracy and complexity(Molnar et al., 2022; Rudin, 2019). As Gervet et al. (2020) observed in theirexperiments, the superiority of deep knowledge tracing models over logistic regression
models is influenced by the size and shape of the dataset.
2. Should we develop an explainable model in the first place and then improve theaccuracy or develop a complex model and try to explain it afterwards (Molnar et al.,2022)?
To address these issues, we focus on a specific educational context of recommending
math quizzes in this study. Previously, we proposed a simple and explainable recommendersystem named Naïve Concept Explicit (Naïve CE) model, which recommends quizzes basedon the estimation of the students' mastery level on math concepts (Dai, Flanagan, et al.,2022). The model also provides explanations of why the students should undertake the quizfrom the perspective of math concepts. We also conducted a real-life experiment where theconcept-based explanations showed positive effects in motivating students to attempt thequizzes (Dai, Takami, et al., 2022). In this study, we are interested in knowing whether the
explainable Naïve CE has a good prediction accuracy compared with a powerful but lessexplainable model--- Matrix Factorization (MF). We also proposed a combined model
CE+MF to preserve the explainability of Naïve CE and predicting power of MF. We used along-term quiz answering dataset in our learning management system to evaluate themodels’ accuracy as to predicting students’ correctness rate of the quizzes. The results
revealed that 1) The explainable model Naïve CE had a lower accuracy than the less modelMF given the sparse dataset; 2) Combining two models achieved a moderate accuracy inpredicting students’ answers while preserving the explainability of Naïve CE. Our study
served as an example of how to develop an inherently explainable educational
recommender system and how to improve the accuracy by integrating more complex
models.

2. Related Work
2.1 Explainability of Recommender Systems
Basically, there are two approaches to generate explanations in recommender systems--model-intrinsic and post-hoc (Zhang & Chen, 2020). In the model-intrinsic approach, theexplanation explains exactly how the model generates a recommendation. In educational
contexts, an example of model-intrinsic explanation can be to explain how the student’s
knowledge state is estimated and why a learning item is considered preferable to improve
his/her knowledge state (Dai, Flanagan, et al., 2022). Other model-intrinsic explanations
include rule-based (Conati et al., 2021), keyword-based (Yu et al., 2021), concept-based(Dai, Flanagan, et al., 2022; Rahdari et al., 2020), and parameter-based (Takami et al.,2022) explanations. In contrast, the post-hoc approach allows the recommending
mechanism to be a “black box” and generates the explanations afterwards. In educational
contexts, a post-hoc explanation for a recommended item can besomething not necessarily
related to the knowledge state estimation but instrumental in motivating the student to accept
the recommendation. For instance, an explanation showing how many students haveattempted this item may work for students who are weak to peer pressure (Takami et al.,2023). Feature-based explanations were adopted for “black-box” models but Swamy et al.
(2022) found that the explainers are not consistent on feature importance. “Black box” or“Deep” methods are commonly considered more powerful in predicting users’ behavior(Molnar et al., 2022; Rudin, 2019). However, this remains doubtful as Gervet et al. (2020)
found that deep knowledge tracing models worked better with larger datasets while logistic
regression models worked better with denser datasets. There is a concern of over-using

89

complex models while an explainable model which has a comparable accuracy is available(Khosravi et al., 2022; Molnar et al., 2022).
In the context of learning math, we consider that developing an explainable
recommender system in the first place is intuitive and straightforward. We proposed a simple
and explainable recommender system named Naïve Concept Explicit (Naïve CE) model,
which recommends quizzes based on the estimation of the students' mastery level on mathconcepts (Dai, Flanagan, et al., 2022). We also conducted a real-life experiment where theconcept-based explanations showed positive effects in motivating students to attempt thequizzes (Dai, Takami, et al., 2022). In this study, we aim to investigate the predicting
accuracy of Naïve CE compared with other classic recommendation models. We also
attempt to improve the accuracy of Naïve CE by combining it with more complex models
while preserving the explainability. This study serves as an example of developing inherently
explainable recommender system as suggested by Molnar et al. (2022).

2.2 Naïve CE and MF for Recommending Math Quizzes
As Birenbaum et al. (1993) suggested, identifying specific misconcepts and difficult areas is
more instructive than a test score for remediation in learning Algebra. Therefore, it isimportant to recommend math quizzes that address the students' weak points which arereadable math concepts. This motived us to propose a concept-explicit recommendersystem named Naïve CE (Dai, Flanagan, et al., 2022). Naïve CE assumes that solving aquiz requires the knowledge of related math concepts in the quiz. We utilized the studentquiz interactions and quiz-concept associations to estimate students’ mastery levels of theconcepts and the possibilities for them to answer the quizzes correctly. We thenrecommended the quizzes based on the possibilities and the expected learning gains interms of the mastery level updates of the concepts. Naïve CE is inherently explainable as
every step of the estimation is a shallow calculation which is understandable for humanbeings.
However, we also have a concern on the estimation performance of Naïve CE. In otherwords, how well can Naïve CE estimate students' mastery level and their probabilities tocorrectly answer the quizzes? Is there an inferiority in estimation performance comparedwith more accurate but less explainable models? As Barnes (2005) pointed out, it isdebatable whether an explicit model with expert-assigned concepts models student
performance better than an implicit model with latent factors. Therefore, we selected aclassic recommendation model matrix factorization with latent factors as a comparative
target of Naïve CE.
Matrix factorization (MF) (Koren et al., 2009; Takács et al., 2008) is a frequently usedmodel to recommend items that users may have interests. This model assumes a user's
interest towards an item comes from her/his preferences on some factors and therelatedness of the factors with the item. It then guesses the unseen user-item interactions bylearning from observed user-item interactions. Khosravi et al. (2017) applied MF in theirmodel to estimate students' knowledge gaps to answering the quizzes. To integrate thestrengths of both models, Abdi et al. (2018) fed the error in Bayesian knowledge tracing
model to an MF model, improving the accuracy of estimating student performance. Since MF
has been verified as a useful model to estimate student performance and shares somesimilarities in modeling the problem with Naïve CE, we chose MF as a comparative model indiscussing the estimation performance and model explainability. We also propose a method
to combine two models so that the readability of concepts in Naïve CE is preserved.

3. Math Recommender Models
3.1 Problem Definition
In learning management systems, the learning activity can be modeled as a sequence ofstudents' reactions towards learning materials. The task is to recommend learning materials
that fit to an individual student's learning progress. It is common that the observed student

90reactions are limited to a small set of the learning materials. As a result, predicting thestudent reactions on unseen learning materials is a key step. For the specific context ofsolving math quizzes, we formalize the problem as follows: Given a set of ݉ students, a setof ݊ math quizzes, and the student correctness rates on the quizzes ܴ ‫ א‬Թ௠ൈ௡ , we want toestimate the student correctness rates on unseen quizzes ܴ෠ ‫ א‬Թ௠ൈ௡ .

3.2 Naive Concept-Explicit Model (Naïve CE)
Suppose that we have a quiz “Let the set of all positive divisors of 12 be A. Fill in the ˎ with‫ א‬or ‫ב‬. (1) 2ˎA (2) 7ˎA (3) 12ˎA”. Solving the math quiz requires the knowledge of “set”
and “positive divisor”. The probability that a student can successfully solve a math quiz
depends on how s/he understands the required concepts. Motivated by this intuition, we
proposed Naïve CE (Dai, Flanagan, et al., 2022) and we review the mechanism of themodel as shown in Figure 1:
STEP 1 When given the observed student-quiz matrix whose entries indicate thecorrectness rates and the quiz-concept matrix whose entries indicate the relatedness of aconcept and a quiz, we calculate the students' mastery level on each concept by looking at
how they successfully solved quizzes related to the concept. Note that the quiz-concept
correspondence is extracted from the quiz information automatically, which is also readable
concepts to students.
STEP 2 We then estimate the probability of a student successfully solving a quiz by
considering how much of the required concepts has been mastered.
By doing this, the probabilities are modified by the inter-relationships between quizzes
and concepts. For instance, ‫ݏ‬ଵ successfully solved ‫ݍ‬ଵ in the history but got an estimatedsuccess of ͲǤ͸͹. This is because ‫ݍ‬ଵ requires the knowledge of ܿଵ and the student failed tosolve ‫ݍ‬ଶ which also requires the knowledge of ܿଵ . However, this model falls short in copingwith unseen concepts. For instance, ‫ݏ‬ଶ had not attempted any quizzes related to ܿଷ . As aresult, ܿଷ is ignored in STEP 2.

Figure 1. The mechanism of Naïve CE model.

3.3 Matrix Factorization (MF)
MF decomposes the observed user-item interaction matrix ܴ ‫ א‬Թ௠ൈ௡ into two matrices ܲ ‫א‬Թ௠ൈ௞ and ܳ ‫ א‬Թ௞ൈ௡ such that ܴ ൎ ܲܳ, where ݇ is the number of latent factors. By minimizing
the difference between the estimated and the observed interactions (also viewed as amachine learning process), we get full ܲ and ܳ, which help us estimate the unseeninteractions. The magic part of this model is that it supposes a user's interest on an item
comes from the synergistic effect of the user's preferences on the latent factors and theimportance of the factors to the item. This idea is somehow similar to Naïve CE except thatthe factors are “latent” and difficult to interpret. There is a variant of MF which considers user
bias and item bias. As a user may tend to highly rate all items or an item of low quality tends
to be rated low by all users, introducing bias parameters in MF helps model this situation. By
doing this, the sum of ܲܳ and bias better approximates the observed interactions but theintermediate value of ܲܳ is harder to interpret.

91

3.4 Concept-Explicit Matrix Factorization (CE+MF)

Figure 2. The mechanism of CE+MF.
In Section 3.2, we discussed that Naïve CE is easy to interpret since the concepts arepredefined and the computation is simple and straightforward. However, it simply gives up
guessing when encountering unseen concepts and quizzes. In Section 3.3, we describedthat MF is good at estimating the probable values for unseen interactions by iteratively
learning from the observed interactions. However, the latent factors of the consisting
matrices are difficult to interpret. To take the strengths of both models, we propose a simple
hybrid model called CE+MF. As illustrated in Figure 2, Naïve CE utilized the observed
student-quiz matrix and quiz-concept matrix to estimate student-concept matrix. The
student-concept matrix is again used to adjust the student-quiz matrix. MF simply
decomposes the observed student-quiz matrix into two matrices with latent factors. In
CE+MF, we first estimate the student-concept matrix as we do in STEP 1 in Naïve CEmodel. Then, we adopt MF model to update the student-concept matrix where the mastery
level on unseen concepts is modified. Last, we update the student-quiz matrix with theupdated student-concept matrix as we do in STEP 2 in Naïve CE model. As a result, CE+MF
model is supposed to have a higher predictive performance than Naïve CE model while
preserving the explainability on concepts.

4. Evaluation
In this study, we aim at investigating two aspects of the recommender models--- quizmastery estimation performance and explainability. For the quiz mastery level estimation
performance, we use a historical dataset collected in a learning system to evaluate whether
the models can correctly predict students’ answers for unseen quizzes. For explainability, weevaluate whether each step in the model can be explained and what user-friendly
explanations can be provided.

4.1 Quiz Mastery Level Estimation Performance
4.1.1 Dataset
We collected quiz answering data from our learning system (Flanagan et al., 2021)
generated by the first-year students of a Japanese high school from April 2021 to March
2022. During this period, the students attempted the math quizzes in different contexts suchas finishing the assignments, preparing an upcoming test, and self-oriented practicing. As
they attempted a quiz, they were required to check the answer and report whether theysolved the quiz successfully. Each attempt was recorded as a 0-1 score associated with thestudent id, quiz id, and timestamp. We computed the aggregated student-quiz correctness
rate by taking the average score of all attempts throughout the period. We did not conduct
any data filtering process as the temporal order of attempts and the number of attempts are
not essential in this evaluation framework. Finally, we obtained a dataset consisting of
27,431 attempts for 270 unique students and 1,919 unique quizzes. Table 1 shows the92

statistics of the number of attempts per student and per quiz. After converting the log data
into the student-quiz correctness matrix, only 23,155 pairs of students and quizzes were͓௢௕௦௘௥௩௘ௗ௣௔௜௥௦
observed, which indicates a very high sparsity of 95.53% (ͳ െ ͓௦௧௨ௗ௘௡௧௦ή͓௤௨௜௭௭௘௦).
Table 1. Statistics of the Dataset for Quiz Mastery Level Estimation Evaluation
mean
std
min
max

# attempts per student
101.596
97.6811808# attempts per quiz
14.29431.4041426

4.1.2 Metrics
Our main concern is to evaluate whether a model can predict a student's success probability
on a quiz. Therefore, we adopt two metrics to measure the agreement between theestimated probability and true correctness rates: Area under ROC curve (AUC) is
considered an effective metric to measure how well a model separate negative and positive
samples across different decision threshold choices (Bradley, 1997). Since the truecorrectness rates for student-quiz pairs are real numbers between 0 to 1, we first transformthe true correctness rates into 1 if it is greater than 0.5, 0 otherwise when applying AUC.
Root mean square error (RMSE) is used to measure the absolute differences between theestimated probability and the true correctness rates.
4.1.3 Implementation

Figure 3. The data splitting process.
As illustrated in the upper part of Figure 3, we set aside 20% of the student-quiz attempts astest data and all models were blind to these data during training or tuning process. For
models involving hyper-parameter tuning, we adopted a 5-fold cross validation approach toselect the best combination of parameters. As illustrated in the lower part of Figure 3, ineach fold, 20% of the data is used to validate the model performance and the average
performance of all the folds is treated as the final performance of a combination ofparameters. We adopted a grid-search approach to generate the combinations ofparameters. Only the performance of the best combination of parameters will be reported in
the following section.
The followings are some implementation details of the models: 1) Naïve CE. As
described in (Dai, Flanagan, et al., 2022), we adopted text mining techniques toautomatically extract math concepts from the quizzes. The entries of quiz-concept matrix
were computed using TFIDF (Salton & Buckley, 1988) weighting scheme. 2) MF. We
adopted stochastic gradient descent algorithm to obtain the matrices ܲ and ܳ whose producthas the minimum difference with the observed student-quiz correctness rate. We tuned threeparameters--- learning rate ߙ, regularization factor ߛ and number of latent factors ݇ for MF.
MF_bias and CE+MF_bias are variants with bias parameters of MF and CE+MF,

93respectively. The best combination of hyperparameters is ߙ ൌ ͲǤͲͳǡ ߛ ൌ ͲǤͳǡ ݇ ൌ ͷ and isused in all MF-related models. The code for MF-related parts was adapted from (Yeung,2020).
4.1.4 Results
Table 2. Quiz Mastery Level Estimation Evaluation ResultsNaïve CE
CE+MF
CE+MF_biasMF
MF_bias

AUC
0.639
0.688
0.692
0.772
0.799

RMSE
0.508
0.478
0.464
0.419
0.381

Table 2 shows the AUC and RMSE values for each model. Overall, Naïve CE has thelowest, MF has the highest, and CE+MF has the medium performance in both metrics. MF
with bias has a better performance both in separate and hybrid models. This result is
consistent with our expectation:z Naïve CE is straightforward but ignores the unseen concepts or quizzes. From theperspective of AUC, this model can discriminate a correct or incorrect answer. However,the performance is nearly a random model from the perspective of RMSE, which means
the detailed values of the correctness has a large gap with student's true mastery level.
z MF and MF_bias do a good job at approximating the observed student-quiz correctness
rates and therefore the latent factors help to predict the values for unseen pairs. With
the AUC value being close to 0.8, MF_bias is supposed to be practically useful toseparate a correct or incorrect answer (Mandrekar, 2010). Meanwhile, the RMSE value
is still high if we consider a situation where we mistake a student's correctness rate 0.98
into 0.6. However, whether the students can recognize the difference and how they
perceive the estimation needs to be further investigated and discussed.
z The hybrid model CE+MF achieves better performance than Naïve CE but still has a
distance to the one of MF. We consider a possible reason is the flaws in quiz-concept
matrix. First, not all necessary knowledge and skills for solving a math quiz can bedetected from the textual information of the quiz. Second, the relatedness of theconcepts to a quiz may not be correct just judging from their occurrences in the quiz.
Ingesting more elaborated domain models to Naïve CE and observe the performance
improvement is one of the future directions.

4.2 Explainability
We compare the explainability of different models as shown in Table 3:z At the lowest level, all the models can provide information about the estimated studentquiz correctness rate, which can be used to indicate the difficulty of the quiz whenrecommending the quiz.
z At the medium level, we try to further explain why the correctness rate is as it is. Naïve
CE, CE+MF, and CE+MF_bias can provide information about the students’ mastery
level of concepts, which is the rationale behind the student-quiz correctness rateestimation. However, MF suffers from provide human-readable information about how
the student-quiz correctness rate was estimated as the factors are latent. Fortunately,
we can leverage the user bias and item bias in MF_bias to provide some extra
information. Specifically, we treat the user bias as the student's general ability to solve
math quizzes and the item bias as the quiz's general difficulty to all students. Note that it
is potential to improve the accuracy of Naïve CE by introducing quiz bias or student
bias, which will be future work.
z At the highest level, we want to explain why the concept master level or quiz generaldifficulty is estimated. In Naïve CE, the concept mastery level is explainable as the quiz-94

concept associations is readable to human. In contrast, CE+MF or CE+MF_bias
involves a matrix factorization process in estimating the concept mastery level, which isdifficult to explain.
Table 3. Explainability and Explanations of Recommender Models.
Explainability Model
LowNaïve CEMF
MF_bias
CE+MF
CE+MF_bias
MediumNaïve CE
CE+MF
CE+MF_bias
MF_biasMF_biasHighNaïve CE

Argument
student-quizcorrectness
rate

User-Friendly Explanation
The estimated difficulty of this quiz foryou is 70%.

concept
masterylevel
quiz
generaldifficulty
student generalabilityYour mastery level of these concepts
can be improved by solving thisquiz: multiple, integer, proof
This quiz is difficult (84%) for most of
your classmates. Let’s have a try!
Your ability (23.4%) to solve quizzes islower than the average (56.9%) of
your classmates.
Integer is important in solving q1(50%),
q2 (30%), and q3 (20%). Since you
have mistaken q1 twice, you aresuggested to address this problem
first.

quiz-concept
associations

To summarize, Naïve CE is explainable until the highest level as every step in themodel is a shallow computation from observed data. MF is only explainable at the lowest
level as the latent factors are difficult to interpret. However, the variant MF_bias possesses
some additional information about the general information of the quiz difficulty and student
ability. This shows a direction to improve the accuracy of Naïve CE by introducing quizdifficulty and student ability parameters. CE+MF is explainable at the medium level as thewhole framework is identical to Naïve CE, but a local step involves a MF process.

5. Discussion
As the results in Section 4.1 show, MF has the highest performance of estimating quizmastery level while Naïve CE has the lowest, CE+MF has the medium performance. The
results in Section 4.2 show that we can preserve part of the explainability of an inherently
explainable model Naïve CE by combining it with a difficult-to-explain model MF. Given thefact that the quiz answering data set is sparse and the quizzes only have 14 answers on
average, we did observe a trade-off between the accuracy and the explainability of different
recommender models. As was explored in Gervet et al.’s work (2020), models’ performancevaries to the type and characteristics of the dataset and features. We think it is important toselect a proper recommendation model based on the learning context and data available.
Besides, we want to clarify that the explainability of the model is not necessarily equal to the
explainability to the students in the practical world. For instance, it would be sufficiently
explainable if the students are satisfied with the quiz general difficulty without the interest tounderstand how it is computed. In this case, MF_bias is not easy to explain in the sense thatevery step is understandable to humans but explainable to end users.

6. Conclusions and Future WorkIn this study, we focused on exploring the accuracy and the explainability of mathrecommender systems. We took a simple and explainable recommender model Naïve CE as
an example and compared its performance with a more complex but difficult-to-explain95

model MF. We also attempted to combine two models so that the strengths of both models
are integrated. Using a student quiz answering dataset, we found that the explainable modelNaïve CE had a lower accuracy than the complex model MF, and the combined model
CE+MF had a moderate accuracy. Our results also showed that it is possible to improve theaccuracy of an inherently explainable model and preserve the explainability by combining itwith more complex models.
Some directions of future work are: 1) Explore other models that can be integrated into
the framework of Naïve CE and introduce other learning related parameters such as quizdifficulty and student ability; 2) Investigate how dataset influences the performance of therecommender models; 3) Ingesting more elaborated domain models into Naïve CE andexplore the performance improvement; 4) Further explore the difference between the model
explainability and the practical explainability to students. Students with different levels of
motivations, information literacy, and curiosity may question the recommendations at
different levels. It may be important to personalize the explanations according to different
scenarios.

Acknowledgements
This work was partly supported by JSPS Grant-in-Aid for Scientific Research (B)JP20H01722 and JP23H01001, (Exploratory) JP21K19824, (A) JP23H00505, and NEDOJPNP20006.

References
Abdi, S., Khosravi, H., & Sadiq, S. (2018). Predicting student performance: The case of combining
knowledge tracing and collaborative filtering. Proceedings of the International Conference on
Educational Data Mining, 545–548.
Afzaal, M., Nouri, J., Zia, A., Papapetrou, P., Fors, U., Wu, Y., Li, X., & Weegar, R. (2021).
Explainable AI for Data-Driven Feedback and Intelligent Action Recommendations to Support
Students Self-Regulation. Frontiers in Artificial Intelligence, 4.
Ai, F., Chen, Y., Guo, Y., Zhao, Y., Wang, Z., Fu, G., & Wang, G. (2019). Concept-Aware Deep
Knowledge Tracing and Exercise Recommendation in an Online Learning System. Proceedingsof the 12th International Conference on Educational Data Mining.
Arrieta, A. B., Díaz-Rodríguez, N., Ser, J. D., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., GilLopez, S., Molina, D., Benjamins, R., Chatila, R., & Herrera, F. (2020). Explainable Artificial
Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.
Information Fusion, 58, 82–115.
Barnes, T. (2005). The Q-matrix method: Mining student response data for knowledge. Proceedingsof American Association for Artificial Intelligence 2005 Educational Data Mining Workshop, 1–8.
Barria-Pineda, J., Akhuseyinoglu, K., Želem-Ćelap, S., Brusilovsky, P., Milicevic, A. K., & Ivanovic, M.
(2021). Explainable Recommendations in a Personalized Programming Practice System.
Proceedings of International Conference on Artificial Intelligence in Education, 64–76.
Birenbaum, M., Kelly, A. E., & Tatsuoka, K. K. (1993). Diagnosing Knowledge States in Algebra Using
the Rule-Space Model. Journal for Research in Mathematics Education, 24(5), 442–459.
Bradley, A. P. (1997). The Use of the Area Under the ROC Curve in the Evaluation of Machine
Learning Algorithms. Pattern Recognition, 30(7), 1145–1159.
Conati, C., Barral, O., Putnam, V., & Rieger, L. (2021). Toward personalized XAI: A case study inintelligent tutoring systems. Artificial Intelligence, 298, 103503.
Dai, Y., Flanagan, B., Takami, K., & Ogata, H. (2022). Design of a User-Interpretable Math Quiz
Recommender System for Japanese High School Students. Companion Proceedings of the 11th
International Conference on Learning Analytics and Knowledge.
Dai, Y., Takami, K., Flanagan, Brendan, & Ogata, Hiroaki. (2022). Investigation on Practical Effects ofthe Explanation in a K-12 Math Recommender System. Proceedings of the 30th International
Conference on Computers in Education, 7–12.
Flanagan, B., Takami, K., Takii, K., Dai, Y., Majumdar, R., & Ogata, H. (2021). EXAIT: A Symbiotic
Explanation Learning System. Proceedings of the 29th International Conference on Computers in
Education, 404–409.

96

Gervet, T., Koedinger, K., Schneider, J., & Mitchell, T. (2020). When is Deep Learning the BestApproach to Knowledge Tracing? Journal of Educational Data Mining, 12(3), 31–54.
Heras, S., Palanca, J., Rodriguez, P., Duque-Méndez, N., & Julian, V. (2020). Recommending
Learning Objects with Arguments and Explanations. Applied Sciences, 10(10).
Huang, Z., Liu, Q., Zhai, C., Yin, Y., Chen, E., Gao, W., & Hu, G. (2019). Exploring Multi-Objective
Exercise Recommendations in Online Education Systems. Proceedings of the 28th ACM
International Conference on Information and Knowledge Management, 1261–1270.
Khosravi, H., Cooper, K., & Kitto, K. (2017). RiPLE: Recommendation in Peer-Learning Environments
Based on Knowledge Gaps and Interests. Journal of Educational Data Mining, 9(1), 42–67.
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado,
R., Sadiq, S., & Gašević, D. (2022). Explainable Artificial Intelligence in education. Computers
and Education: Artificial Intelligence, 3, 100074.
Klašnja-Milićević, A., Vesin, B., Ivanović, M., & Budimac, Z. (2011). E-Learning personalization basedon hybrid recommendation strategy and learning style identification. Computers & Education,
56(3), 885–899.
Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender
Systems. Computer, 42(8), 30–37.
Mandrekar, J. N. (2010). Receiver Operating Characteristic Curve in Diagnostic Test Assessment.
Journal of Thoracic Oncology, 5(9), 1315–1316.
Molnar, C., König, G., Herbinger, J., Freiesleben, T., Dandl, S., Scholbeck, C. A., Casalicchio, G.,
Grosse-Wentrup, M., & Bischl, B. (2022). General Pitfalls of Model-Agnostic Interpretation
Methods for Machine Learning Models. In A. Holzinger, R. Goebel, R. Fong, T. Moon, K.-R.
Müller, & W. Samek (Eds.), XxAI - Beyond Explainable AI: International Workshop, Held in
Conjunction with ICML 2020, July 18, 2020, Vienna, Austria, Revised and Extended Papers (pp.
39–68). Springer International Publishing.
Rahdari, B., Brusilovsky, P., Thaker, K., & Barria-Pineda, J. (2020). Using Knowledge Graph for
Explainable Recommendation of External Content in Electronic Textbooks. Proceedings of theSecond International Workshop on Intelligent Textbooks 2020 Co-Located with 21st International
Conference on Artificial Intelligence in Education (AIED 2020), 2674, 50–61.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions anduse interpretable models instead. Nature Machine Intelligence, 1(5), 206–215.
Salton, G., & Buckley, C. (1988). Term-Weighting Approaches in Automatic Text Retrieval.
Information Processing & Management, 24(5), 513–523.
Shanshan, S., Mingjin, G., & Lijuan, L. (2021). An improved hybrid ontology-based approach foronline learning resource recommendations. Educational Technology Research andDevelopment, 69(5), 2637–2661.
Swamy, V., Radmehr, B., Krco, N., Marras, M., & Käser, T. (2022). Evaluating the Explainers: BlackBox Explainable Machine Learning for Student Success Prediction in MOOCs. Proceedings ofthe 15th International Conference on Educational Data Mining, 98–109.
Takács, G., Pilászy, I., Németh, B., & Tikk, D. (2008). Matrix Factorization and Neighbor Based
Algorithms for the Netflix Prize Problem. Proceedings of the 2008 ACM Conference on
Recommender Systems (RecSys’08), 267–274.
Takami, K., Dai, Y., Flanagan, B., & Ogata, H. (2022). Educational Explainable Recommender Usageand Its Effectiveness in High School Summer Vacation Assignment. Proceedings of the 12th
International Learning Analytics and Knowledge Conference (LAK22), 458–464.
Takami, K., Flanagan, B., Dai, Y., & Ogata, H. (2023). Toward Trustworthy Explainable
Recommendation: Personality Based Tailored Explanation for Improving E-learning
Engagements and Motivation to Learn. The Companion Proceedings of the 13th International
Conference on Learning Analytics & Knowledge, 120–122.
Tang, X., Chen, Y., Li, X., Liu, J., & Ying, Z. (2019). A reinforcement learning approach topersonalized learning recommendation systems. British Journal of Mathematical and Statistical
Psychology, 72(1), 108–135.
Yeung, A. A. (2020). Matrix-factorization-in-python. In GitHub repository. GitHub.
https://github.com/albertauyeung/matrix-factorization-in-python
Yu, R., Pardos, Z., Chau, H., & Brusilovsky, P. (2021). Orienting Students to Course
Recommendations Using Three Types of Explanation. Adjunct Proceedings of the 29th ACM
Conference on User Modeling, Adaptation and Personalization, 238–245.
Zhang, Y., & Chen, X. (2020). Explainable Recommendation: A Survey and New Perspectives.
Foundations and Trends® in Information Retrieval, 14(1), 1–101.

97" !   !" " "   #" !
#" !"$  #" ! #"                   
     
    
/ /%27%- 9 %' .4  $.+4<++4)5:49+7675(2+38<./2+6+7,573/4-86+)/,/)')9/;/9/+8<+4++*
95/*+49/,>9.+)':8+5,9.+675(2+395,/=/94')9/;/9>/8)53658+*5,8+;+7'283'22
')9/;/9/+88:(')9/;/9/+8'4*9.+)':8+3/-.92/+/4+/9.+754+5,9.+3$+9.+7+,57+
4++*95'4'2>?+9.+)53654+4985,9.+')9/;/9>95/*+49/,>9.+)':8+5<+;+78/4)+
;'2/*8:(')9/;/9/+8'7+459*+,/4+*/9/8*/,,/):29956+7,573'8:,,/)/+49'4'2>8/85,9.+
')9/;/9>".+5(0+)9/;+5,9./87+8+'7)./895675658+'3+9.5*,57'4'2>?/4-'4')9/;/9>
'4* /*+49/,>/4- 9.+ 675(2+3'9/) 8:(')9/;/9> "5 '99'/4 9./8 -5'2 ;'2/* 8:(')9/;/9/+8
8.5:2* (+ *+,/4+* "./8 7+8+'7). 675658+8 '4 ')9/;/9> 35*+2 9.'9 7+67+8+498 9.+
7+2'9/548(+9<++4'4')9/;/9>'4*/988:(')9/;/9/+84'**/9/54/9)54897:)98'4')9/;/9>
'4'2>8/88>89+3</9.,:4)9/54895675;/*+./498,57*+7/;/4-8:(')9/;/9/+8'4*;+7/,>9.+
;'2/*/9>5,9.+*+7/;+*8:(')9/;/9/+8 75(2+3'4'2>8/8')9/;/9>35*+2;'2/*/9>).+)1 
63.)'8%7)()%62-2+ ,%73*8)2&))2%(348)(*36'908-:%8-2+463&0)1 730:-2+7/-007 2789()287%'59-6)/23;0)(+)%2(7/-007&=*-2(-2+%2(730:-2+463&0)17-271%00+63947 3;):)6731)8-1)78,)7)%'8-:-8-)7%6)23846%'8-')(79'')77*900=&)'%97)8,)789()280%'/7
8,) 7/-00 83 46%'8-') 8,) %'8-:-8-)7  #,)6) %6) 7):)6%0 789(-)7 8,%8 7944368  1%2= *3'97
46-1%6-0= 32 '327869'8-2+ 8,) +6394;%6) 8,%8 *%'-0-8%8)7 '31192-'%8-32 &= )2%&0-2+ 8,))<',%2+)3*463(9'87%2(34-2-327;-8,-28,)+6394!->%0!97(-%2%")8-%;%2"-%,%%2  ,)2)8%0    3;):)61378(323879443688,)0)%62-2+3*8,)463&0)1 730:-2+
7/-00-87)0*  63&0)1730:-2+-7%''3140-7,)(&=%2%0=>-2+8,)463&0)1*36190%8-2+%73098-32
%2(-140)1)28-2+-8737944368%8)%',78)4-72)')77%6= #,-7789(=*3'97)732%2%0=>-2+8,)463&0)1%2(():)0347%79443687=78)1*36463&0)1%2%0=7-7 "31)789(-)779443688,)-()28-*-'%8-323*8,)'%97)3*%463&0)1&=46)7)28-2+%1)8,3(*36 %2%0=>-2+ 8,) 463&0)1  %6-', )8 %0  463437)( % ;36/7,))8 8,%8 )2%&0)7 +6394 463&0)1
730:-2+ &= 6)4)%8)(0= %7/-2+ 8,) 59)78-32 $,= 928-0 % '327)2797 -7 6)%',)( ;-8,-2 8,)+6394%6-',!37%7   #,)463&0)1%2%0=7-71)8,3(746)7)28)(-28,)7)789(-)7'%2
()6-:)8,)'%97)3*%463&0)1&98(3238+9%6%28))8,):%0-(-8=3*8,)()6-:)('%97) 368,)
()6-:%8-32 3* :%0-( '%97)7 8,)6) %6) 789(-)7 -2 ;,-', 8,) 7=78)1 8%/)7 8,) 40%') 3* 8,)()6-:%8-32 36 %(%48-:)0= 79443687 8,) ()6-:%8-32 &%7)( 32 463&0)1 (31%-2 /23;0)(+)  36
-278%2')36-)8%0 *3'97)(32%2%0=>-2+8,)6)7)%6',463&0)13*8,))(9'%8-32%08)',2303+=
(31%-2 36- %=%7,-  ")8%    #,)= '327869'8)( %2 328303+= *36 '32(9'8-2+ 79',
6)7)%6',%2(463437)(%7=78)1*36+)2)6%8-2+59)78-3278,%8%003;8,)97)683'327-()68,)
)0)1)287 &%7)( 32 8,) 328303+=  "-2') 8,)7) 789(-)7 7944368 :%0-( %2%0=7-7 &%7)( 32 8,)7=78)17463&0)1(31%-2/23;0)(+)8,)7)789(-)7'%2320=7944368463&0)1%2%0=7-7;-8,-2
8,)7'34)3*8,)7=78)17463&0)1(31%-2/23;0)(+) "-2')8,)6)%6)1%2=(-**)6)288=4)7
3* ):)6=(%= 463&0)17 -8 -7 238 4377-&0) 83 ()7'6-&) %00 8,) 463&0)1 (31%-2 /23;0)(+) -2
%(:%2') $)1978463437)%+)2)6-'1)8,3(3*463&0)1%2%0=7-76%8,)68,%2,%:)8,)7=78)1
'%66=8,)/23;0)(+)3*8,)463&0)1(31%-2 63&0)1 %2%0=7-7 -7 8,) %'8-:-8= 3* 92()678%2(-2+ 8,) 7869'896) 3* 8,) 463&0)1 %2((-7'3:)6-2+ -87 '%97)7  2 8,) %2%0=7-7 8,) )0)1)287 8,%8 '31437) 8,) 463&0)1 7,390( &)
()6-:)(  3;):)6 -8 -7 (-**-'908 83 ()6-:) )239+, )0)1)287 8,%8 '%2 *361 8,) 463&0)1 98#,)6)*36);)()*-2)8,)'32(-8-3273*;,%87,390(&)()6-:)(-28,)463&0)1%2%0=7-7 $)
%073'0%6-*=;,%82))(783&)'327-()6)(83()6-:))0)1)2878,%87%8-7*= 8,)'32(-8-327 %2('327869'8%2 %2%0=7-77944368 7=78)18,%80)%(78,)%2%0=7-7%2(.9(+)7;,)8,)68,)%2%0=>)()0)1)287 7%8-7*= 8,)'32(-8-327 132+ 8,) 1%2= 8=4)7 3* 463&0)17 ;) *%') -2 (%-0= 0-*) 8,-7 6)7)%6', *3'97)7 32
463&0)17 8,%8 3''96 (96-2+ %'8-:-8-)7?463&0)17 8,%8 -28)6*)6) ;-8, %2 %'8-:-8=@7 79'')77*90
4)6*361%2') #,)6)*36)8,)463&0)1%2%0=7-7 -783()6-:)8,)79& %'8-:-8-)73*8,)%'8-:-8=

    
 




 
   

2 %'8-:-8= -7 %2 %'8-32
   
7)59)2') 8%/)2 83 %',-):)
    


% +3%0 %2( -7 '31437)( 3*7):)6%0
79& %'8-:-8-)7
-+96) 7,3;7 8,) %'8-:-8=  

13()0  '8-:-8-)7 ,%:) %
'32(-8-32 8,%8 -2(-'%8)7 8,) -%+6%13*'8-:-8=3()078%8) -2 ;,-', 8,)= '%2 &)
4)6*361)( #,)(-**)6)2')&)8;))28,)46) 78%8)%2(4378 78%8) '%2&)6)+%6()(%78,)79&%'8-:-8=@7)**)'8 #,)79& %'8-:-8=7)59)2')'%2%',-):)8,)7%1)%'8-:-8=%78,)36-+-2%0%'8-:-8=738,)
79& %'8-:-8-)71978,%:) 8,)7%1))**)'8%78,)36-+-2%0%'8-:-8= 2%((-8-328,)79& %'8-:-8-)7
7,390( ,%:) % '%97%0 6)0%8-32 ;-8, )%', 38,)6 ;,)6) 8,) 4378 78%8) 3* 8,) '%97%0 %'8-:-8=
&)'31)78,)'32(-8-32*368,)6)7908-2+%'8-:-8=     #,-7789(='327869'87% 7=78)1 83 7944368
97)6 %2%0=7-7 3*
%'8-:-8-)7 &%7)( 328,) %'8-:-8= 13()0 -+96)  7,3;7 8,)'32*-+96%8-32 3* 8,)7=78)1
8 '327-787 3*
%2-28)6*%')%:%0-(-8=.9(+1)28 *92'8-32
%2(%,-28*92'8-32 -%+6%13*"=78)132*-+96%8-32
1%+-2-2+ 8,)7')2)7-2;,-',8,)%'8-:-8-)7%6)%'89%00=4)6*361)( -7)**)'8-:)*36()6-:-2+8,)79& %'8-:-8-)7 #,)6)*36)396-28)6*%')0)878,)97)6()7'6-&)8,)%'8-:-8= 79& %'8-:-8-)7;-8,8,) 7')2) #,) :%0-(-8= .9(+1)28 *92'8-32 ',)'/7 ;,)8,)6 8,) ()6-:)( 79& %'8-:-8-)7 7%8-7*= 8,)'32(-8-327  8 '327-787 3* %2 )**)'8 79**-'-)2'= .9(+1)28 *92'8-32 8,%8 .9(+)7 ;,)8,)6 %00'%2(-(%8)79& %'8-:-8-)7,%:)%2)**)'8 %'3,)6)2').9(+1)28*92'8-328,%8',)'/7;,)8,)6
%0079& %'8-:-8-)7%6)'322)'8)(&=%'%97%06)0%8-32%2(%'327-78)2'=.9(+1)28*92'8-328,%8
()8)61-2)7;,)8,)68,))**)'83*8,))28-6)79& %'8-:-8-)77)59)2')-7'327-78)28;-8,8,%83*8,)36-+-2%0%'8-:-8= #,),-28*92'8-3279++)787;,%883'327-()683()6-:)%:%0-(79& %'8-:-8-)77)59)2') #,)97)6 '%2 -24988,)%'8-:-8=83&)%2%0=>)(8,)'%2(-(%8)79& %'8-:-8-)7%2(8,)
)**)'87%2('32(-8-3273*8,)%'8-:-8= '%2(-(%8)79& %'8-:-8-)7 -2838,)-28)6*%') $,)28,)97)66)59)787%:%0-(-8=.9(+1)283*8,)79& %'8-:-8-)78,):%0-(-8=.9(+1)28*92'8-32.9(+)7 ;,)8,)6
8,)'%2(-(%8)79& %'8-:-8-)7 )28)6)(&=8,)97)6%6):%0-(36238%2(463:-()76)%7327-*-8-7238:%0-( $,)28,)97)66)59)787%,-28 6)'311)2(%8-328,),-28*92'8-326)'311)2(78,)
%446346-%8)34)6%8-32&%7)(328,):%0-(-8=.9(+1)28 6)7908

99

-+96)  7,3;7          8,) 7=78)1 -28)6*%')
'327-78-2+ 3* %2 %'8-:-8=
(-740%= %6)% %2( %234)6%8-32 %6)%  2 8,)34)6%8-32%6)%8,)97)6
'%2 -2498 8,) %'8-:-8=
79& %'8-:-8= %2( )**)'8
  
%2( '32(-8-32 8,%8 8,)97)6()6-:)(-2,-736,)6   
%2%0=7-7 #,) %'8-:-8=
(-740%= %6)% 7,3;7 8,)   -24988)( 79& %'8-:-8-)7
%2(8,)-6
'%97%06)0%8-327  #,) 79&%'8-:-8=-77,3;2-28%&0)
*361%8;-8,8,) 2%1)3*
8,) 79& %'8-:-8= 32 834 "=78)128)6*%')%2( -87 )**)'8 %2('32(-8-32 &)03;  #,) '%97%0 6)0%8-32 -7 6)46)7)28)( &= 8,) %6' &)8;))2 79& %'8-:-8-)7  #,)7=78)1 7)%6',)78;379& %'8-:-8-)7 32);,37)'32(-8-32-78,)7%1)%78,))**)'83*8,)38,)6
%2( -**392((6%;7 %2%6'&)8;))28,)1#,)%2%0=7-779443684%2)0-28,)34)6%8-32%6)%'328%-278;3&988327 %&988328,%8
%7/783',)'/8,):%0-(-8=3*8,)-249879& %'8-:-8-)7%2(%&988328,%86)59)787%,-28 $,)2
8,):%0-(%8-32',)'/&98832-746)77)(-8(-740%=7%1)77%+)-2(-'%8-2+ 8,%88,)79& %'8-:-8=
-7:%0-(-*-8-7.9(+)(83&):%0-( *238-8(-740%=78,)6)%732;,=-8-7238:%0-( $,)28,),-28
&98832-746)77)(-8(-740%=7%1)77%+)8)00-2+8,)97)6;,%883(32)<8 
#,-74%4)6 463437)7 %2%'8-:-8=13()0%2(8,)%'8-:-8=%2%0=7-7 79443687=78)1&%7)(32-8 #,)%'8-:-8=%2%0=7-7 79443687=78)1463:-()7%2-28)6*%')8,%8 0)878,)97)6'327-()6238320=
8,)79& %'8-:-8=&98%073-87'32(-8-32%2()**)'8 8%073463:-()7% *92'8-328,%8.9(+)7;,)8,)6
8,) '%2(-(%8) 79& %'8-:-8= 7)59)2') -7 :%0-( %2( +-:)7 ,-287 8,%8 -2*361 8,) 97)6 ;,%8 83
'327-()683()6-:)%:%0-(79& %'8-:-8=7)59)2') 73962)<878)4;);-00 '32(9'8)<4)6-1)287
83):%09%8)8,):%0-(-8=3*8,)%'8-:-8=13()0%2(8,))**)'8-:)2)773*8,)463437)(7=78)1


.+4%3+25!/2;+7'05/+!&.+4-:'4-5*4'7!  #8/4-"+').+7
'8.(5'7*8 95 ))+88 75:6 522'(57'9/54 /4 75(2+3('8+* +'74/4- #!" !&
$!! "!   '7/). 58'8  75)+88*7/;+475(2+3!52;/4-</9. 559':8+4'2>8/8*'69/4-5<+7,:2:8/4+88"552895/9".+!:89'/4'(/2/9>75(2+3 !#&"# %!  
57/  '>'8./ %  !+9'     49525->('8+* ".5:-.9 7-'4/?'9/54 !:66579 !>89+3 95
75369 +'*/4+885,49+49/54!.'7/4-'4*9854-9+737')9/)+ $! !#&"#" $#   /?'2  :8*/'4'!+9/'<'4$!/'.''4  +;+2563+495,'75(2+3('8+*+'74/4-
'4'-+3+49!>89+38:66579+*!3'796.54+! 662/)'9/54#8/4-9.+5*+295
3675;+/-/9'2/9+7')> #!# $!! $#"! 


100

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Method for Estimating Learning Strategies
from Tools Using Bayesian Network
Kento KUWAJIMAa*, Atsushi ASHIDAb & Tomoko KOJIRIc
a

Graduate School of Science and Engineering, Kansai University, Japanb

Institute for Datability Science, Osaka University, Japan
Faculty of Engineering Science, Kansai University, Japan*k268505@kansai-u.ac.jpcAbstract: With the recent development of the robot technology, teacher robots will be
introduced into educational settings to instruct learners. Such teacher robots, likehuman teachers, will be required not only to teach knowledge to learners, but also tomonitor their learning process and motivate them. Positive learning attitudes can beascertained by changing the learning strategy. The fact that learners change theirlearning strategies may indicate that they are making efforts to find an appropriate
strategy. The learning strategies used by learners can be inferred from the tools they
use to learn. Therefore, we propose a method to estimate learning strategies using
Bayesian networks when learners change them.
Keywords: learning strategy, teacher robot, motivation, Bayesian Network

1. Introduction
In the future, it is expected that robots will play an increasingly important role in education,such as to provide lessons to learners. Such teacher robots will be required to provide
instruction equivalent to that of a human teacher.
When human teachers teach, they not only impart knowledge to learners, but also theyoften motivate learners to study. One of these efforts is to provide instruction that increases
the learner's motivation. In the future, such instruction will also be required for teacher robots.
However, current robot-based instruction focuses only on knowledge teaching (Ono et al.
2019, Tanaka & Matsuzoe, 2012), and there are few examples of instruction that is oriented
toward motivation. Therefore, this paper proposes a method for robot teachers to enhancelearners' motivation.
One of the methods for enhancing the motivation is to evaluate the learner's effort andingenuity in the learning activity and praise them. The effort and ingenuity are often appeared
as changes in learning methods during the learning process, so it is required to grasp thechanges in the learner's learning methods.
For a robot teacher to read changes in a learner's learning method, it is necessary toestimate changes in the learning method from what the robot can observe. Therefore, theresearch questions for this study are as follows:z What observable information can be used to estimate changes in learning methods?
z How to estimate learning methods using these observables tools?
Usually, learners use learning tools, such as notes, pencils, and rulers, and such tools are
fixed according to the learning methods. Based on this consideration, this studyfocuses on thelearning tools as observable information of the robot and proposes a method for estimatingchanges in the learning method from them.

2. Overview of Learning Method Estimation System
The learning method is a pair of learning strategies and learning means. A learning strategy
is a way for learners to approach learning in order to achieve their learning goals. An example
of strategies is "memorizing" when the learning goal is to learn English words. In order to101

practice the learning strategies, the learner selects a learning means, such as "writing
repeatedly" for the strategy "memorize." Many of the learning means use tools. For example,
to achieve the means of "writing repeatedly," a pencil that can be used for writing, and toachieve the means of "divide into sections," a multi-color pen that can mark the segmented
items in different colors might be used as tools. Therefore, the learning method, which is apair of strategies and means, is characterized by the tools.
In this study, for the teacher robot to estimate the learning method, Tools-meansstrategies model, which represents the relationship between tools, means, and strategies, isintroduced. By observing the tools through the sensor, the learning means and the learning
strategies that are currently taken are estimated by using Tools-means-strategies model.
As the first step of realizing
such teacher robot, we construct asystem that corresponds to a functionthat identifies the current learningmeans and learning strategy. An
overview of the proposed system isshown in Figure 1. The system has
Tools-means-strategies model and alearning record that stores the historyof the learner's learning strategies.
When the user of the system inputs
Figure 1. System Configuration
the tools that a learner used for thelearning, instead of observing it using the sensors, the learning method identification function
estimates the learning strategy based on Tools-means-strategies model and stores it as alearning record.

3. Method for Estimating Learning Strategy
3.1 Tools-means-strategies Model
Tools-means-strategies model proposed applied Bayesian networks to represents the causal
relationship between the strategy and the means, and between the means and the tool. A
Bayesian network represents causal inference by means of a directed acyclic graph structure,
where nodes represent variables and links represent probabilistic correlations among
variables. Each node hasa corresponding choice,each of which has aprobability that the choice
will be chosen when asuperior node makes adecision. Each alternative
within the same node is
exclusive and does notoccur simultaneously.
In
theToolsmeans-strategies model,
the
nodes
are
the
strategy, the means, andthe tool, and the links exist
between the strategy andthe means and between
the means and the tool.
When multiple tools are
Figure 2. Example of Tools-means-strategies Model
used at the same time, theset of tools are regarded as a single tool. Figure 2 shows an example of Tools-meansstrategies model for English vocabulary learning.

102

3.2 Learning Method Identification Function
The learning method identification function uses the Tools-means-strategies model toestimate the current strategy of the learner based on the input tools. The probability of eachstrategy is calculated from the model, and the strategy with the highest calculated value isdetermined to be in use. The probability that a learner is using a certain strategy A when a tool
B is observed is calculated by the following formula.
ሺܲ‫݀݁ݒݎ݁ݏܾ݋ݏ݅ܤ݈݋݋ݐܽ݄݊݁ݓܣݕ݃݁ݐܽݎݐݏݏ݁ݏݑݎ݁݊ݎ݈ܽ݁ܽݐ݄ܽݐݕݐ݈ܾܾ݅݅ܽ݋ݎ‬ሻൌ ሺܲ‫ܣݕ݃݁ݐܽݎݐݏ݃݊݅ݏݑ݂݋ݕݐ݈ܾܾ݅݅ܽ݋ݎ‬ሻ‫כ‬෍ሼሺܲ‫ܣݕ݃݁ݐܽݎݐݏ݃݊݅ݏݑ݄݊݁ݓݏ݊ܽ݁݉݄ܿܽ݁݃݊݅ݏݑ݂݋ݕݐ݈ܾܾ݅݅ܽ݋ݎ‬ሻ௔௟௟௠௘௔௡௦‫ כ‬ሺܲ‫ݏ݊ܽ݁݉݄ܿܽ݁݃݊݅ݏݑ݄݊݁ݓܤ݈݋݋ݐ݃݊݅ݏݑ݂݋ݕݐ݈ܾܾ݅݅ܽ݋ݎ‬ሻሽ
Let’s consider the example using the Tools-means-strategies model shown in Figure
2. If a pencil and a notebook are observed, the probability using each strategy can be obtainedby normalizing the values obtained in the following equation so that the sum is 1.
Memorize: 0.8 x (0.1 x 0.7 + 0.9 x 0.9) = 0.704
Analyze the structure: 0.2 x (0.4 x 0.7 + 0.6 x 0.9) = 0.164
Normalizing the obtained values and rounding to the fourth decimal place, theprobability of using the strategy of memorizing when using a pencil and notebook is 0.811 andthe probability of using the strategy of analyzing the structure is 0.189, which indicates thatthe learner is taking the strategy of memorizing.

4. Conclusion
In this study, we proposed a method for estimating a learner's learning strategy from the tools
used by the learner. Since there is a causal relationship from strategies to means and frommeans to tools, Tools-means-strategies model was introduced that can infer strategies basedon the causal relationships when tools are observed. Currently, we have constructed themodel for English vocabulary learning based on the results of a questionnaire survey asking
the strategies, means, and tools in the English vocabulary learning. In future, we need toevaluate the validity of this constructed model. In addition, we also need to develop otherfunctions of the teacher robot.
The current system treats tools as concrete instruments. However, just as notebooks
are being replaced by PC-based document creation tools in line with the recent digitalization,
the tools used may change in the future. Tools have functions, and tools with the samefunctions are likely to be used in the same way and in the same manner. For example, a pencil
and a tablet pen both have the function of writing, and a ruler and a triangular ruler bothmeasure straight lines. By expressing the tools by their function, not by the name, Toolsmeans-strategies model can be used even if tools change in the future. Therefore, we needto modify the tool nodes in Tools-means-strategies model to represent the their functions, notto their names.

References
Horino, M., & Ichikawa, S. (1997). Learning Motives and Strategies in High School Students’ English
Learning. Japanese Journal of Educational Psychology, 45, 140-147. [In Japanese]Ono, H., Koike, K., Morita, T., & Yamaguchi, T. (2019) Ontologies-Based Pupil Robot Interaction withGroup Discussion, 23rd International Conference on Knowledge-Based and Intelligent Information& Engineering Systems. Procedia Computer Science 159, 2071-2080.
Tanaka, F., & Matsuzoe, S. (2012). Children Teach a Care-Receiving Robot to Promote Their Learning:
Field Experiments in a Classroom for Vocabulary Learning. Journal of Human-Robot Interaction,
1(1), 78-95.

103

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

ExGen: Ready-To-Use Exercise Generationin Introductory Programming Courses
Nguyen Binh Duong TAa*, Hua Gia Phuc NGUYENa & Swapna GOTTIPATIa
School of Computing and Information Systems, Singapore Management University*donta@smu.edu.sgaAbstract: In introductory programming courses, students as novice programmers
would benefit from doing frequent practices set at a difficulty level and concept suitable
for their skills and knowledge. However, setting many good programming exercises forindividual learners is very time-consuming for instructors. In this work, we propose an
automated exercise generation system, named ExGen, which leverages recent
advances in pre-trained large language models (LLMs) to automatically createcustomized and ready-to-use programming exercises for individual students ondemand. The system integrates seamlessly with Visual Studio Code, a populardevelopment environment for computing students and software engineers. ExGen
effectively does the following: 1) maintaining a set of seed exercises in a personalized
database stored locally for each student; 2) constructing appropriate prompts from theseed exercises to be sent to a cloud-based LLM deployment for generating candidate
exercises; and 3) implementing a novel combination of filtering checks to automaticallyselect only ready-to-use exercises for a student to work on. Extensive evaluation using
more than 600 Python exercises demonstrates the effectiveness of ExGen ingenerating customized, ready-to-use programming exercises for new computing
students.
Keywords: introductory programming courses, exercise generation, large language
models, prompt engineering, auto-filtering.

1. Introduction
Despite recent advances in code generation using large language models (LLMs), e.g.,
OpenAI Codex, ChatGPT, Google LaMDA, etc., programming is still an essential skill thatcomputing students must master in the foreseeable future (Becker et al., 2023). It is wellknown that computer programming is a challenging subject for many new university students(Keuning et al., 2018). To learn effectively, students must practice on their own frequently withsuitable exercises designed for their level of programming skills and knowledge. However, it
is usually impractical for instructors to manually create enough exercises for all students, letalone creating customized or personalized exercises for each individual student on-demand,i.e., when they need to practice. Exercises from Internet platforms for coding practices andinterview preparations (Joshi et al., 2023) such as LeetCode.com might not be ready-to-use
in introductory courses as they are not customized for the skill levels of new computing
students. From our experience, many easy-level LeetCode problems can be quite challenging
for novice programmers (Ta et al., 2022). Intermediate or harder problems found online often
require advanced data structures and algorithm concepts not taught in an introductory course.
Automatic generation of programming exercises has been an important problem intechnology-enabled education (Zavala and Mendoza, 2018). Recently, the application of pretrained LLMs in supporting exercise and feedback generation has been gaining attention.
Sarsa et al. (2022) explored OpenAI Codex for the purpose of creating new Python exercises
and code explanations. They provided a comprehensive, mostly manual evaluation of theoutput generated by Codex. They showed that a large percentage of the generated exercises
were sensible and new, but most of them were not directly usable by students. However, they
did not address the problem of on-demand generation of ready-to-use exercises customized

104

for various levels of difficulty. Recently, Kazemitabaar et al. (2023), Becker et al. (2023) and
Finnie-Ansley et al. (2023) discussed and evaluated the use of LLMs to generate sample
solutions and to provide adaptive feedback on code submitted by students, but not exercise
generation.
In this work, we propose ExGen, a software tool for automatically generating new,ready-to-use exercises for individual students learning introductory programming in Python.
ExGen makes use of LLMs which have been pre-trained on a massive amount of text andcode such as OpenAI's GPT-3.5-Turbo. In ExGen, we implement carefully designed LLM
prompting strategies, automatic filtering of exercises, and seamless integration with Visual
Studio (VS) Code. ExGen makes it convenient for students to work on new exercises that are
customized for their skill levels. An innovative component of ExGen is that it makes use of
LLMs not just to generate new exercises, but also to automatically check if the newly
generated exercises are ready-to-use or not. In this way, ExGen is different from existing
research that investigated exercise generation based on pre-defined templates (Zavala andMendoza, 2018). We make the following contributions in this paper:z
We considered the problem of auto-generating ready-to-use exercises customized for acertain difficulty level and programming concepts, e.g., string, lists, etc., upon requests
from individual students.
z
We designed zero-shot and few-shot prompting strategies to obtain appropriate
responses from pre-trained LLMs. We then proposed a mechanism of chaining different
filtering checks which auto-select ready-to-use exercises instead of requiring humanexperts to inspect the LLM output and make decisions. The checks are based on popular
software engineering techniques such as unit testing combined with LLM-based checking
techniques.
z
We implemented a fully functional VS Code extension which makes it more convenient
for students to generate new exercises and practice coding. We noted that using existing
web interfaces such as ChatGPT or OpenAI's Playground requires a lot of copy/paste
actions for adjusting prompts and submitting requests.
z
We conducted an extensive evaluation of ExGen using more than 600 programming
exercises generated via the GPT-3.5-Turbo API. Each of these exercises was manuallyinspected to see if they are ready-to-use; and compared to ExGen's output. The result
demonstrated the effectiveness of ExGen and the proposed solution design.

2. Problem Statement
We are interested in auto-generating ready-to-use exercises, i.e., those that a student canwork on right away without further modifications by human experts. To be considered readyto-use, a generated exercise would need to have: 1) a clear problem statement, 2) working
solutions and test cases, and 3) an appropriate difficulty level as requested by the student.
This is a challenging problem, as Sarsa et al. (2022) found that most generated exercises
were not ready-to-use when using OpenAI Codex which is based on the GPT-3 model.
On top of generating clear, well-defined problem statements, producing exercises withthe right level of difficulty is of crucial importance to students learning programming. For
instance, beginners would be encouraged by easy exercises covering the concepts taught inclass as they are just getting started. On the other hand, above average students would liketo be challenged with harder problems which require a certain level of higher order thinking.
Many programming problems available on Internet platforms (Joshi et al., 2023) are notappropriate for students in our course due to them: 1) being very difficult, and/or 2) requiring
advanced concepts such as dynamic programming, tree-based data structures, etc., which
are not covered in an introductory course for first-year students.
In this work, we define three levels of difficulty for exercises which have been used in
our introductory programming course at Singapore Management University. The three levelsof difficulty, namely "easy", "intermediate", and "hard", are defined relative to each other.
These levels can cater to a cohort of students with wide-ranging levels of skills and abilities.
As the course is more about basic programming and computational thinking skills, and lessabout using Python, some basic exercises can be made harder by disallowing certain Python105

utilities such as sorting functions. We can also impose some constraints on the number ofloops due to efficiency reasons, and as a result, the difficulty is increased. In the following
three examples, we explain the difficulty levels for each exercise which has been used in ourcourse. The solutions are not included here due to space constraints.
Example 1: "Prompt the user for a message. Display the message character by character on
the same line but separate every two adjacent characters by a space". This is an "easy"
exercise which tests basic string manipulation skills using a "for" loop in Python. Most students,
including those who are new to programming, after being taught the concepts, can handle thisexercise without issues. We have been observing that a few might need help with printing out
all the characters on the same line. Note that the "easy" exercises are more than just trivial
programming tasks such as creating a list, adding two numbers, etc.
Example 2: an "intermediate" exercise
for string manipulation. The number oflines of code as well as the logic for a
possible solution to this exercise ismore complicated than that of Exercise
1. The student would have to figure out
the exact range for a loop from m to n,followed by a few complex conditional
statements to get the correct output.
Therefore, in the context of this paper,
Exercise 2 is more difficult than
Exercise 1.

Example 2: Define a function called dis_nums(m:int, n: int), m <=n. The function displays all
int numbers between m and n (both inclusive)
separated by spaces. Special cases: if the number
is a multiple of 3 but not a multiple of 5,
display '-' instead. If the number is a multipleof 5 but not a multiple of 3, display '*'instead; 3) if the number is a multiple of both 3
and 5, display '#' instead. For instance,
dis_nums(4, 16) shows: 4 * - 7 8 - * 11 – 13 14 #Example 3: Define a function get_longest_subsq(st)for returning the longest subsequence from st thatconsists of only the letters from the English
alphabet. If there are multiple subsequences ofthe same longest length, the function returns thefirst one. If st doesn’t contain any letter, '' isreturned. Implement this function using just one
for loop. For instance: get_longest_subsq('ab24[AaBbCDExy0longest$]') returns 'AaBbCDExy',and get_longest_subsq('12345 ') returns ''.

Example 3: a "hard" exercise, also forstring manipulation. This exercise is not
as straightforward as Exercise 1 or 2,
as it requires the student to keep trackof the starting and ending point of each potential subsequence of letters using a Boolean
variable, and the indices in the input. There is also the additional requirement of using a single"for" loop, which makes the exercise more challenging.
We note that it is not practical to manually create a lot of such ready-to-use exercises
on-demand, i.e., when students need them, that are personalized to individual skill levels andlearning concepts, as many students might be working concurrently on different kinds ofproblems at various difficulty levels. In the below section, we describe our approach torealizing automatic generation of ready-to-use exercises.

3. ExGen: Design and Implementation

Figure 1. Design and Implementation of ExGen
Figure 1 shows the interactions between students, instructors and ExGen in our programmingcourse. As ExGen aims to minimize instructor’s involvement, the only role of the instructor isto provide a limited number of sample (seed) exercises with three difficulty levels: “easy”,“intermediate”, and “hard”. Each student in the course can download these exercises into apersonalized database which is saved locally. Students can load any exercises from their own

106database, work on them, and use ExGen to generate more exercises to practice. Students
can choose to save new exercises into their personalized database for reference later.
ExGen provides functionalities for managing a personalized database of exercises foreach student, generating new exercises via pre-trained LLMs, and providing a suite of autofiltering checks to give students ready-to-use exercises at a difficulty level suitable for theirknowledge and programming skills. In the below, we provide a brief description of each ExGen
component as shown in Figure 1, and relevant implementation details.

3.1 Prompt management and LLM integration
ExGen generates new exercises by first constructing a "prompt" as a query to be sent to a
pre-trained LLM such as OpenAI GPT-3.5-Turbo. It obtains a list of candidate exercises forfurther filtering before presenting a few selected ones in an IDE, e.g., VS Code, for the student
to work on. Constructing good prompts for LLMs in many application domains, i.e., "prompt
engineering", is currently an active area of research (Liu et al., 2023). LLM prompts may
consist of questions, instructions, examples, etc., so that the model would reply with an output
which has some desired qualities and/or quantities. In "zero-shot" prompting, we can query
the model without providing any example of expected results. On the other hand, "few-shot"
prompts provide several actual examples to the model.
In the GPT-3.5-Turbo model, which is used in ExGen's implementation, input prompts
can be given as a series of messages with different parameters. Each message is an object
with a role, which is either "system", "user", or "assistant", together with the content of themessage. Usually, the first message would be a "system" message which helps define thebehavior of the AI assistant. We can then alternate between "user" and "assistant" messages,
in which "user" messages provide specific instructions for the AI assistant, and "assistant"messages can be used to provide examples of desired outputs, i.e., the "shots" in few-shot
prompting. All messages must be constructed and passed to the LLM in a single API call, sothat the model would have the right context when generating the new exercises. Note that
OpenAI's web APIs do not have the memory of past requests. In this work, we consider two
different strategies of prompting for exercise generation using GPT-3.5-Turbo, namely:z

Zero-shot Prompting (referred to as "zero-shot"): provides keywords on the
programming concepts and difficulty levels required by a student, without using any
examples of exercises. The prompt starts with a system message, e.g., "You are a helpful
teaching assistant for undergraduates who are learning introductory programming in
Python", and then adds a user message, e.g., "Give me three easy Python string
exercises with this keyword: house".
Message 1: {"role": "system", "content": "You are a helpful teaching assistant forundergraduates who are learning introductory programming in Python."}Message 2: {"role": "user", "content": "Give me a hard Python string exercise."}Message 3: {"role": "assistant", "content": "Here is one hard Python string exercise:{example of a relevant exercise from the database}"}Message 4: {"role": "user", "content": "Good. I want one more hard Python string
exercise. Print the result with the same format as the previous ones."}... (more messages for remaining examples)
Last message: {"role": "user", "content": "Good. I want {N} more hard Python string
exercises using this keyword: {keyword}. Print the result with the same format as theprevious ones."}Figure 2. Messages for Few-shot Promptingz

Few-shot Prompting (referred to as "few-shot"): provides a few actual examples on how
the desired output should look like. In essence, when requested by students, ExGen
constructs a series of messages following the GPT-3.5-Turbo model as shown Figure 2.
ExGen first inspects the current exercise that the student is working on. Second, it
searches for other exercises on the same concept, e.g., string, list, etc. and having asimilar level of difficulty in the student's database. ExGen then sequentially appends therelevant examples, each with a problem statement, solution, and test cases, with the107appropriate roles into the list of messages. The result will be like a conversation in which
the LLM plays the role of a teaching assistant guided via a multi-turn conversation withseveral concrete examples. The last message is a user message requesting the LLM togenerate N new exercises. Students can also specify some new keywords, e.g., "person","movies", etc., so that the generated exercises would have more variations. Note that we
can specify the difficulty levels, e.g., "easy", "hard", etc., and concepts, e.g., "string", "list",etc. in the "few-shot" prompt messages from Message 2 onwards in Figure 2.
In the implementation of "few-shot", ExGen uses three examples (3-shot) with the default
parameters of the GPT-3.5-Turbo model. We deploy our own cloud instance for running LLM
inferences using Azure OpenAI Service for better control and billing purposes. We also notethat previous work in exercise generation, e.g., (Sarsa et al., 2022), mostly made use of the
OpenAI Codex API, which has been deprecated since Mar 2023. The output of this stage is N
candidate exercises, each having: 1) a problem statement, 2) a solution, and 3) test cases.
ExGen will conduct auto-filtering for these exercises in the next stage.

3.2 Auto-filtering of generated exercises
In ExGen, we implement several methods to filter exercises generated by LLMs. The methods
are chained, i.e., a newly generated exercise must successfully clear the previous filter before
it is checked by the next filter, as shown in Figure 3. We consider standard software
engineering techniques as well as LLM-based approaches for filtering. First, ExGen will checkif the Python solution code can be compiled to bytecode without any errors. Then it will run all
the unit test cases which are g
generated bywith every
y the LLM togethergy new exercise.

Figure 3. Chain of Filters in ExGen
After a new exercise can pass the first two filters, ExGen will do an additional check
regarding the difficulty level of the exercise. The rationale for implementing this final check isthat LLMs such as GPT-3.5-Turbo are probabilistic models, and their answers might bedifferent from one interaction to the next. Therefore, it could be beneficial to ask an LLM toverify its own answers generated previously.
Message 1: {"role": "system", "content": "You are a classification model that willclassify the difficulty of Python exercises."}
Message 2: {"role": "user", "content": "I want to you classify this exercise: {example
exercise}"}Message 3: {"role": "assistant", "content": "Difficulty: " {example’s difficulty level}}... (more messages for remaining examples)
Last message: {"role": "user", "content": "I want you to classify this exercise:{candidate exercise}"}Figure 4. Example Messages for Difficulty Classification
For the difficulty check, ExGen will perform a classification task by first constructing anappropriate prompt as shown in Figure 4, and then asking the LLM to classify the generated
exercise in terms of difficulty levels. The prompt has two examples for each level of difficulty
in a series of user and assistant messages. For the last user message, we ask the LLM toprovide a classification for a new candidate exercise based on the provided examples. If the
difficulty of the new exercise matches the required difficulty level, it will be shown to the student
as a ready-to-use exercise using ExGen's VS Code extension. In addition, we observed that
GPT-3.5-Turbo can generate exercises covering the concepts specified in the prompt such as
lists, strings, etc. most of the time, so ExGen does not need to check if a requested
programming concept is included in the output.

108

3.3 Coding interface and database management
ExGen provides a convenient interface for students to work and generate exercises right in
VS Code. We have used ChatGPT as well as OpenAI's Chat Playground extensively, and we
observed that it takes a lot of time and effort to copy/paste prompts and exercises forgeneration tasks. A screenshot of ExGen's interface is shown in Figure 5. Students can clickon "+ More exercise" at the right bottom of the IDE to generate new ready-to-use exercises.
They also have the option to change the problem statement, solutions or test cases used as
examples in the prompt before clicking "Submit". ExGen will automatically connect to an LLM
deployment, generate, and filter
the exercises.
Students can also save
selected ready-to-use exercises
into his/her own personalized
database. To make it easy forstudents, we use the TinyDB
package, which is a portable
Pythondocument-oriented
database, instead of setting up
a fully functional SQL databaseengine like MySQL. The
database comes bundled in
ExGen so students can just use
it right away without further
setup or installation.

4. Evaluation

Figure 5. ExGen's VS Code Interface

In this section, we aim to answer the following research questions (RQs):zzz

RQ1: Which is the best prompting strategy for generating ready-to-use exercises?
RQ2: How effective is the auto-filtering approach implemented in ExGen?
RQ3: How much time does it take to generate a ready-to-use exercise?

4.1 Dataset and performance measures
We used a seed set that has 49 manually crafted Python exercises including all levels ofdifficulty and introductory programming concepts such as string, list, dictionary, tuple, etc.
There were 18 "easy", 17 "intermediate", and 14 "hard" exercises in the seed set. We thengenerated a total of 613 exercises (205 "easy", 270 "intermediate", and 138 "hard") using
ExGen via the API of GPT-3.5-Turbo LLM. We then manually inspected each of these
exercises to determine if they are ready-to-use. A ready-to-use exercise must satisfy thefollowing criteria used in the manual inspection: 1) a clear problem definition, 2) a correct
solution and test cases, and 3) a correct difficulty level as requested in the prompt.
We then measured the performance, i.e., accuracy, of each prompting strategy andfiltering check when compared to the manual inspection results. For example, when "few-shot"
prompting produces 5 ready-to-use out of 10 generated exercises, the performance of "fewshot" is calculated as 50%. On the other hand, the performance of a filtering check is
calculated by how often it agrees with the manual inspection result. For example, theperformance of the unit testing check is calculated as 70% if it provides the same result, i.e.,
ready-to-use or not, as the manual inspection for 7 out of 10 generated exercises. We
measured the individual performance of each filtering check, as well as the overall filtering
performance.

1094.2 Results and discussion
RQ1-Comparing different prompting strategies: Figure 6(a) shows the percentages ofready-to-use exercises for the two prompting strategies and three difficulty levels. The "fewshot" prompting performed much better than "zero-shot" prompting in all cases. This isbecause "few-shot" incorporates examples to better guide the LLM in generating new
exercises with appropriate levels of difficulty. We observed from the collected data that the"zero-shot" prompt usually produced many more trivial exercises, such as creating a list of
several numbers, printing out a message with certain parameters, etc. These are not even at
the "easy" level. This is understandable, as the "zero-shot" prompt does not have enoughinformation about how an easy or hard exercise would be like. As a result, about 57% of "easy"
exercises generated by "few-shot" prompting were ready-to-use, compared to just 31% ofthem in "zero-shot" prompting.
From Figure 6(a), it is observed that generating ready-to-use "intermediate" and "hard"
exercises is quite challenging, even with recent advances in LLMs. In the experiments, onlyaround 10% and 25% of "intermediate" exercises generated by "zero-shot" and "few-shot"
prompting were considered ready-to-use, respectively. Similarly, we obtained only several
ready-to-use "hard" exercises using "few-shot".
ϲϬ͘ϬϬй

ϭϬϬ͘ϬϬйǌĞƌŽͲƐŚŽƚ

ĨĞǁͲƐŚŽƚ

ǌĞƌŽͲƐŚŽƚ

ĨĞǁͲƐŚŽƚ

ϰϬ͘ϬϬй
ϱϬ͘ϬϬй
ϮϬ͘ϬϬй
Ϭ͘ϬϬй

Ϭ͘ϬϬй
ĂƐǇ

/ŶƚĞƌŵĞĚŝĂƚĞ

,ĂƌĚ

ĂƐǇ

(a) ready-to-use exercises (%)/ŶƚĞƌŵĞĚŝĂƚĞ,ĂƌĚ(b) exercises with correct difficulty levels (%)

Figure 6. Comparing two Prompting Strategies
Figure 6(b) shows the percentages of generated exercises which have the correctdifficulty levels (but might not be ready-to-use). By providing more detailed information about
difficulty levels with examples in the prompt, "few-shot" produced more exercises with thedesired difficulty levels. Around 40% and 25% of exercises generated by "few-shot" could beconsidered as "intermediate" and "hard", respectively. However, quite a number of these
exercises did not have clear problem statements, correct solutions and/or test cases, so they
were not considered ready-to-use. Nonetheless, we observed that it's possible to make somesmall modifications so that many of these more challenging exercises would be ready-to-use.
Table 1. Performance of the Auto-Filtering Checks
zero-shot(unit test)
Easy
68.6%
Intermediate 51.4%
Hard
51.5%

zero-shot(difficulty)
37.1%
79%
90.9%

zero-shot(chain)
69.5%
90.6%
98.5%

few-shot(unit test)
83.8%
55.3%
54.2%

few-shot(difficulty)
96%
78.8%
77.8%

few-shot(chain)
83.8%
90.2%
97.2%

RQ2-Evaluating ExGen's auto-filtering approach: The standalone performance of the unit
testing and difficulty checks, as well as the overall filtering performance (chain of all checks)
are shown in Table 1 (higher percentage is better). From the experiments, we noted that thesolutions of most generated exercises could pass the compilation check so that its result is
not included in Table 1. The unit testing check when used alone could correctly identify if a110candidate exercise is ready-to-use or not in many cases, especially for "easy" exercises
generated with "few-shot" prompting (83.8% accuracy when compared to manual inspection's
result). However, unit testing became less effective as the only filter for harder exercises asshown in Table 1. This is because although around 50-70% of candidates for harder exercises
had working solutions and test cases in our experiments, they were not at the right levels ofdifficulty, i.e., "intermediate" or "hard".
Table 1 also shows the performance of the difficulty check when used alone as an
exercise filter. It is not very effective when filtering "easy" exercises generated by "zero-shot"
prompting as there could be many trivial exercises. The difficulty check performed better whenfiltering "intermediate" and "hard" exercises, compared to the unit testing check.
We observed good performance for our auto-filtering approach when chaining all thethree filters, i.e., compilation followed by unit testing followed by difficulty check. The accuracyof the chained filters improved significantly for all prompting strategies when filtering"intermediate" and "hard" exercises. Overall, we noted that "few-shot" combined with thechained filters produced the best performance. Using this combination, we are more confident
that ExGen can provide ready-to-use exercises for students to work on right away.
It was also noted that the LLM used in ExGen tends to provide partially correct
solutions and/or test cases when generating "intermediate" and "hard" exercises. Althoughthese candidate exercises did meet the difficulty requirement, they failed ExGen's chained
filters in the end. We believe that with a few manual modifications by human experts, they canbe ready-to-use by students.
RQ3-Exercise generation time: We measured the average time to produce a ready-to-use
exercise using the two prompting strategies combined with the chained filters for various
difficulty levels. We noted that the generation time was quite reasonable for "easy" exercises,
which was around 10 and 20 seconds per exercise with "few-shot" and "zero-shot" prompting,respectively. An "intermediate" exercise required 53 seconds for "few-shot", and 90 seconds
for "zero-shot". Using a less effective prompt like "zero-shot" increased the average generation
time as there would be more candidate exercises which were not ready-to-use.
On the other hand, generating ready-to-use "hard" exercises could take too long to beconsidered on-demand (about 8 minutes for "few-shot", and 14 minutes for "zero-shot"). Thiswas because many candidate exercises would have to be generated before ExGen could find
one that met the "hard" difficulty requirement and has correct solutions/test cases. To this end,
we believe that "hard" exercise generation should be best done in a hybrid way. That is, ExGen
can provide initial generation and filtering support for course instructors to quickly modify the
LLM's output and make it ready-to-use.
Discussion: From the careful evaluation using more than 600 auto-generated exercises, we
believe that ExGen can provide students with ready-to-use programming exercises
customized at the right difficulty levels. LLM prompting strategies having sufficient details suchas examples have been shown to generate more exercises that are ready-to-use. In mostcases, our auto-filtering approach with a chain of software engineering and LLM-based checks
could work well regardless of the prompting strategy used. However, it is noted that
automatically generating truly ready-to-use, "hard" exercises is still a problem in AI-enablededucation despite recent progress in LLM and AI development.
Threats to Validity: We note that there are several limitations which may affect the validity ofthis study. First, the output from LLMs could be non-deterministic, which might impact thequality of generated exercises. We have tried to take this into account by evaluating many
exercises. Second, manual evaluation of exercises by human experts could be subjective,e.g., when assigning a difficulty level. For this, we have used the opinion of two different
programming experts on the set of generated exercises. Finally, as more advanced LLMs suchas GPT-4 API are still in limited availability, it is possible that the results obtained from thesemodels are different from what we have seen in this study. We plan to do further investigations
using exercises in Python and other programming languages once access to more powerful
LLMs becomes widely available.

1115. Related Work
Much research has been done in automatic generation of formative feedback and reference
solutions to code produced by students (Keuning et al. 2018, Koutcheme 2022, Ta et al. 2022).
Such feedback and reference solutions could be generated by current pre-trained LLMs tohelp novice programmers know how to proceed when facing coding issues. On the other hand,
not much has been done in generating ready-to-use programming exercises for students to
do more practices. Kurdi et al. (2020) conducted a systematic review of automatic exercise
generation in many different domains such as analytical reasoning, geometry, history, logic,
relational databases, programming, and science.
Zavala and Mendoza (2018) used Automatic Item Generation (AIG) to address the
problem of creating many similar programming exercises using pre-defined templates which
are used for quizzes. The main goal was to ensure consistency in testing many students withquestions of the same level of difficulty. On the other hand, ExGen focuses on generating
ready-to-use exercises for a specific difficulty level and concept that the student is currently
working on. We leveraged the latest advances in LLMs to autogenerate many novel exercises
and filter them to ensure that they are suitable for students. Exercises considered in ExGen
are different from other kinds of programming practices such as faded Parson problems(Fromont et al. 2023), which require students to fill in code in partially scrambled solutions.
The most relevant work in automatic exercise generation using pre-trained LLMs has
been done by Sarsa et al. (2022). The authors explored OpenAI Codex (which has been
deprecated since Mar 2023) for the purpose of creating new programming exercises and code
explanations. They found that many Codex-generated exercises were sensible and novel, but
may have confusing problem statements, missing or faulty test cases. Sarsa et al. (2022) did
not consider the implementation of an auto-filtering tool to supply students with ready-to-use
exercises at various difficulty levels.
Recently, there has been an increase in the number of works leveraging pre-trained
LLMs for educational purposes. Kasneci et al. (2023) discussed the potential benefits, forinstance content generation and personalized learning, as well as challenges, e.g., model
biases, system brittleness, etc., when applying LLMs to education. Similarly, Becker et al.
(2023) elaborated the educational opportunities of AI code generation, and how educators
should act quickly given these developments. Finnie-Ansley et al. (2023) reported theperformance of OpenAI Codex on real questions from programming courses when comparedto that of actual students. Denny et al. (2023) studied GitHub Copilot, a plug-in for IDEs like
VS Code, which is based on the Codex model, to see what kinds of problem it would not
perform well. The authors also found that prompt engineering can play an important role ininteracting with AI tools like Copilot. MacNeil et al. (2023) found that code explanations suchas line-by-line, high-level summary, and lists of important concepts generated by Codex and
GPT-3 were helpful to most students. Kazemitabaar et al. (2023) studied how Codex's code
generation capability could assist novice programmers via a controlled experiment for young
students. We note that our work focuses on automatic exercise generation and filtering, notfeedback and explanation for code submitted by students.

6. Conclusion
We have implemented and evaluated ExGen, a new software tool that automatically generates
ready-to-use exercises for computing students. ExGen manages a personalized database of
seed exercises and constructs appropriate LLM prompts to obtain candidate exercises for arequested difficulty level and learning concept. ExGen incorporates a novel combination of
auto-filtering checks which reduce the tedious work of manually inspecting all the generated
output. Our extensive evaluation with more than 600 programming exercises generated using
GPT-3.5-Turbo API demonstrated that ExGen can be used to provide students with ready-touse exercises. We plan to investigate new approaches for automatic prompt optimization sothat harder exercises can be generated more easily. As a VS Code extension, ExGen can bereleased to our incoming freshmen for further user evaluation in the next academic year.

112

Acknowledgements
This research is supported by the Ministry of Education, Singapore, under its Tertiary
Education Research Fund (Award No. MOE2021-TRF-014). Any opinions, findings andconclusions or recommendations expressed in this material are those of the authors and do
not reflect the views of the Ministry of Education, Singapore.

References
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023).
Programming Is Hard-Or at Least It Used to Be: Educational Opportunities and Challenges of AI
Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science
Education V. 1 (pp. 500-506).
Denny, P., Kumar, V., & Giacaman, N. (2023). Conversing with Copilot: Exploring prompt engineering
for solving CS1 problems using natural language. In Proceedings of the 54th ACM TechnicalSymposium on Computer Science Education V. 1 (pp. 1136-1142).
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023,
January). My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2
Programming Exercises. In Proceedings of the 25th Australasian Computing Education
Conference (pp. 97-104).
Fromont, F., Jayamanne, H., & Denny, P. (2023). Exploring the Difficulty of Faded Parsons Problems
for Programming Education. In Proceedings of the 25th Australasian Computing Education
Conference (pp. 113-122).
Joshi, I., Budhiraja, R., Dev, H., Kadia, J., Ataullah, M. O., Mitra, S., ... & Akolekar, H. D. (2023).
ChatGPT--a Blessing or a Curse for Undergraduate Computer Science Students andInstructors?. arXiv preprint arXiv:2304.14993.
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., ... & Kasneci, G.
(2023). ChatGPT for good? On opportunities and challenges of large language models foreducation. Learning and Individual Differences, 103, 102274.
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023).
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory
Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing
Systems (pp. 1-23).
Keuning, H., Jeuring, J., & Heeren, B. (2018). A systematic literature review of automated feedback
generation for programming exercises. ACM Transactions on Computing Education (TOCE),
19(1), 1-43.
Koutcheme, C. (2022). Towards Open Natural Language Feedback Generation for Novice
Programmers using Large Language Models. In Proceedings of the 22nd Koli Calling
International Conference on Computing Education Research (pp. 1-2).
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020). A systematic review of automaticquestion generation for educational purposes. International Journal of Artificial Intelligence in
Education, 30, 121-204.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A
systematic survey of prompting methods in natural language processing. ACM Computing
Surveys, 55(9), 1-35.
MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., ... & Leinonen, J. (2023). Experiencesfrom using code explanations generated by large language models in a web softwaredevelopment e-book. In Proceedings of the 54th ACM Technical Symposium on Computer
Science Education V. 1 (pp. 931-937).
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022). Automatic generation of programming
exercises and code explanations using large language models. In Proceedings of the 2022 ACM
Conference on International Computing Education Research-Volume 1 (pp. 27-43).
Ta, D., Shar, l. K., & Shankararaman, V. (2022). AP-coach: Formative feedback generation forlearning introductory programming concepts. In Proceedings of IEEE International Conference
on Teaching, Assessment, and Learning for Engineering (TALE).
Zavala, L., & Mendoza, B. (2018). On the use of semantic-based AIG to automatically generate
programming exercises. In Proceedings of the 49th ACM technical symposium on computerscience education (pp. 14-19).

113" !   !" " "   #" !
#" !"$  #" ! #"      
    
      "!  # !$"!$  # !$
99/<A/7 C /1 8>

   5 6)1,*:680,5:,+ 786.8(4405.  ,(*/ 786*,99 09 (**647309/,+ )?
:/, 05:,8(*:065 6- 6)1,*:9 5 :/, -089: 9:,7 6- +,90.505. ( 9?9:,4 ;905. 786.8(44,895,,+:6+,*64769,(.0<,5;9,*(9,05:694(33,89;)786*,99,9(5+
0+,5:0-?6)1,*:96=,<,8-6856<0*,786.8(44,890:*(5),+0--0*;3::604(.05,:/05.9
:/(:(8,56:,>730*0:3?9:(:,+05:/,;9,*(9,(5+964,4(?56:;5+,89:(5+:/(::/,
786*,99 09 (*/0,<,+ )? :/, 05:,8(*:065 6- 6)1,*:9 %, /(<, :/,8,-68, +,<,367,+ (:8(0505. 4,:/6+ :/(: 0+,5:0-0,9 6)1,*:9 )? *655,*:05. :/,4 =0:/ :/05.9 :/(: 56<0*,
786.8(44,89*(5468,,(903?04(.05,9;*/(98,*07,9!0403(8:6;9,*(9,98,*07,9
(8,(9,:6-786*,99,9=/69,9;)1,*:9(8,56:*3,(83?4,5:065,+%/,5=,+,*64769,
(8,*07,9,:05:694(33,89;)786*,99,94(+,;76-20:*/,5=(8,0:,490:0976990)3,:6
8,*6.50@,:/,9,0:,49=/,504(.0505.:/,*66205.9*,5,%,(396+,<,367,+(9?9:,4:/(: /,379 56<0*, 786.8(44,89 :6 ,>:8(*: :/, (7786780(:, 20:*/,5=(8, -864 8,*07,9
"/86;./:/,,>7,80,5*,6-8,=80:05.(8,*07,;:030@05.:/,786*,996-*66205.56<0*,
786.8(44,89 (8, ()3, :6 8,*6.50@, :/(: :/,8, (8, 9;)1,*:9 /0+05. 05 ( 786*,99
+,9*807:065=/69,9;)1,*:9(8,56:4,5:065,+=/0*//,379:/,4;5+,89:(5+/6=:6
0+,5:0-?:/,/0+05.6)1,*:9 6)1,*:680,5:,+786.8(4405.,>:8(*:0656-6)1,*:98,*07,:8(0505.9;7768:
9?9:,4+8(=05.9*,5,  

%0831B =@73<B32>@=5@/;;7<5%%&7AE723:GCB7:7H327<B63A=4BE/@37<2CAB@G)374 /HH/<  !<%%&>@=5@/;036/D7=@7A7;>:3;3<B32/A/<7<B3@/1B7=<03BE33<=0831BA +63<
23A75<7<5A=4BE/@3CA7<5%%&B631:/AA3A=4=0831BA<332B=0323A75<327</11=@2/<13E7B6
B63 @3?C7@3;3<B A>317471/B7=<A 3 5  CA3 1/A3  )= 23A75< / AGAB3; A= B6/B B63 235@33 =4
1=C>:7<503BE33<1:/AA3A7A:=E/<2B63235@33=41=<23<A/B7=<E7B67<1:/AA3A7A67567B7A
<313AA/@GB=723<B74G4C<1B7=<AB6/B1/<035@=C>32B=53B63@/<2B=723<B74GB63B67<5AB6/B1/<
1/@@GB6=A34C<1B7=<A/A=0831BA  =E3D3@7B1/<03274471C:B4=@<=D713>@=5@/;;3@AB=723<B74G
=0831BA4@=;B63CA31/A3B6/B/@3<=B3F>:717B:G;3<B7=<32/<2A=;32=<=BC<23@AB/<2B6/B
B63>@=13AA7A/11=;>:7A6320GB637<B3@/1B7=<=4=0831BA )63@34=@37<3FB@3;31/A3AB63G
;/G23A75</AGAB3;E7B6/A7<5:3=0831B1/::32IAGAB3;J/<2/AA75</::4C<1B7=<AB=7B $/<G ABC273A 6/D3 033< 1=<2C1B32 B= 63:> >@=5@/;;3@A C<23@AB/<2 B63 =0831B=@73<B321=<13>B#7/<*/@=G7/1/;/ (3<5-/B7; =3 E/@79/3
*7::73@A   A=;3 =4 E6716 A>317471/::G 4=1CA =< B63 036/D7=@A =4 B63 =0831BA 0727< ./E/7  3ABE7197"/G/@/;/<  :B6=C56B63A3ABC273A>@=D723/<C<23@AB/<27<5=4 B63 =0831B =@73<B32 1=<13>B B63G 2= <=B AC>>=@B B63 23A75< >@=13AA =4 %%& E6716 7A
3AA3<B7/::GB=723<B74G=0831BA7</11=@2/<13E7B6B63@3?C7@3;3<BA>317471/B7=<A !<=@23@B=
723<B74G=0831BA7B7A<313AA/@GB=7;/57<3B63>=B3<B7/:7<B3@/1B7=<A=4;C:B7>:3=0831BA4@=;
B6357D3<@3?C7@3;3<BA>317471/B7=< !<B631C@@3<BABC2GE33F>:=@3BE=93G@3A3/@16?C3AB7=<A
 +6/B 97<2 =4 B67<97<5 7A <313AA/@G B= 23@7D3 /23?C/B3 =0831BA 4@=; B63>@=13AA23A1@7>B7=<7<E6716AC0831BA/@3<=B1:3/@:G;3<B7=<32
+6/B97<2=4B@/7<7<57A/>>@=>@7/B3B=4=AB3@AC16B67<97<5)=23@7D3B63/<AE3@AB=B63A3?C3AB7=<AE323D3:=>32/B@/7<7<5;3B6=2/<23D/:C/B327BA
34431B7D3<3AA 

114)63>@=>=A32B@/7<7<5;3B6=2723<B7473A=0831BA0G1=<<31B7<5B63;E7B6B67<5AB6/B
<=D713>@=5@/;;3@A1/<;=@33/A7:G7;/57<3AC16/A@317>3A (7;7:/@B=CA31/A3A@317>3A
/@3/A3B=4/1B7=<AE6=A3AC0831BA/@3<=B1:3/@:G;3<B7=<32 %C@;3B6=217@1C;D3<BAB67A
274471C:BG0G231=;>=A7<5B63A3B7<B=A;/::3@AC0 >@=13AA3A=497B163<E/@3 =@3F/;>:3IB=
1==9;3/BJ1/<03@3E@7BB3</AI/AB=D363/BAC>/4@G7<5>/</<2B634@G7<5>/<1==9AB63
;3/B J+31/<@31=5<7H3B6397B163<E/@37B3;AE63<E37;/57<3B631==97<5A13<3A=7B7A
3/A73@B=5@/A>B6/BB63@317>3A/@33AA3<B7/::G/A3@73A=497B163<E/@37<B3@/1B7=<A/<2B63@30G
B=7;/57<3E6/B97<2=497B163<E/@37A7<D=:D327<B63@317>3 %C@AGAB3;0/A71/::G3<1=C@/53A
<=D713>@=5@/;;3@AB=723<B74G97B163<E/@34@=;B63E@7BB3<@317>3B6/BB63G7;/57<34@=;7BA
A13<3 )63AGAB3;/A9AB63CA3@AB= 3FB@/1BB6397B163<E/@34@=;B63@317>3@3E@7B3B63
@317>37</11=@2/<13E7B6B633FB@/1B3297B163<E/@3/<22@/E/>71BC@3CA7<5B63723<B74732
97B163<E/@3  )67A >@=13AA 7A @3>3/B32 C<B7: B63G /@3 /0:3 B= /11C@/B3:G 23>71B B63 1==97<5
A13<3B6/BB63G7;/57<34@=;B63@317>3 )6@=C56B633F>3@73<13=4@3E@7B7<5/@317>3CB7:7H7<5
B63>@=13AA=497B163<E/@3<=D713>@=5@/;;3@A/@3/0:3B=@31=5<7H3B6/BB63@3/@3AC0831BA
6727<57<B63>@=13AA23A1@7>B7=<E6=A3AC0831BA/@3<=B;3<B7=<32 !</227B7=<0G1=<B7<C7<5
B67A/1B7D7BGC<B7:B63G2@/EB63A13<3B6/BB63G7;/57<3B63G0357<B=5@/A>B6/B7;/57<7<5B63
A13<37A<313AA/@G4=@23@7D7<5/23?C/B3=0831BA 
  $!##'"# %#

@3?C7@3;3<BA>317471/B7=<7A/2=1C;3<BB6/BAC;;/@7H3AB634C<1B7=<A/<216/@/1B3@7AB71A
B6/B/AGAB3;B=0323D3:=>32A6=C:26/D3 !B1=<A7ABA=4/0/A3/1B7=<A3@73A3F3;>:74732
/1B7=<A3@73A/<2=B63@1=<27B7=<A/<27B7A3AB/0:7A6327<B63@3?C7@3;3<BA2347<7B7=<>6/A3
=4AGAB3;23D3:=>;3<BB=4=@;B63>=:71G4=@B63AC0A3?C3<B23D3:=>;3<B>:/< <3F/;>:3
=4/@3?C7@3;3<BA>317471/B7=<7A/CA31/A323A1@7>B7=<E671623A1@703AA>3174717<B3@/1B7=<A
03BE33</</1B=@/<2/AGAB3;7<B6/BA13<3/AE3::/A@3:/B321=<27B7=<A)316<=:=571@BA!<1     (7<13 /1B=@A /@3 B63 CA3@A =4 B63 AGAB3; B63 3FB3@</: 6/@2E/@3 B63 3FB3@</:
AGAB3;A @3:/B32 B= B63 B/@53B AGAB3; 3B1  B63 CA3 1/A3 23A1@7>B7=< 7A / AC;;/@G =4 B63@3?C7@3;3<BA4@=;B63CA3@A>=7<B=4D73E )63@34=@3B631=<475C@/B7=<=4B63AGAB3;3 5 
B63=0831BA/@3<=BBG>71/::G;3<B7=<32 75C@3  A6=EA /< 3F/;>:3 =4 B63 0/A71 /1B7=< A3@73A 4=@ B63 CA3 1/A3 IE7B62@/E7<5
;=<3G4@=;/<)$J  #(cash &#!&) #!'$!  #!#$#' !#$#" '#"! !""#!$## $##$#''$&"#&#!&!#  &#!&"# %"!#&#!&"#"/A71/1B7=<A3@73A4=@1/A6E7B62@/E/:4@=;/<)$  # ## #(7<13B630/A71/1B7=<A3@73A=4/CA31/A323A1@7>B7=<2=3A<=B/:E/GA;3<B7=<B63AC0831B
=@B/@53B=43/16/1B7=<B63>@=5@/;;3@E7::<332B=23@7D3=0831BAB6/B1/<03B63AC0831BA
=@B/@53BA !4B63A13<37<B63CA31/A31/<037;/57<32B63723<B74732=0831BAA6=C:2037<B63
7;/57</B7=< +3>@=>=A3/;3B6=24=@23@7D7<5/23?C/B3=0831BA0G6/D7<5B63>@=5@/;;3@A2@/E/ >71BC@3 =4 B63 >@=13AA7<5 A13<3  (7<13 B63 /23?C/B3 =0831BA /@3 B6=A3 23A1@7032 7< B63>71BC@3 B67A ;3B6=2 3<1=C@/53A >@=5@/;;3@A B= 23@7D3 AC0831BA /<2 B/@53BA 4@=; B63 CA3
1/A3B6/B/@3B63A/;3/AE6/BB63G7;/57<3 G6/D7<5B63;2@/EB63>71BC@3B63G031=;3
/E/@3B6/BB63=0831BAB63G/22B=B637;/53/@3B63=0831BAB6/BB63GA6=C:2723<B74G 

115  $"# # ##

#"" "!< =@23@ B= /1?C7@3 B63 B67<97<5 ;3B6=2 B6/B B63 =0831BA 7< B63 A13<3 =4 B63 CA3 1/A3
A>317471/B7=</@3B63=0831BAB=03723<B747327B7A23A7@/0:3B=B@/7<B63B67<97<5;3B6=20G/CA31/A3 E6=A3A13<37A3/AGB=7;/57<3 !<B67AABC2GE3CB7:7H3/1==97<5@317>3/A B63
AC0AB7BCB7=<4=@B63CA31/A3
@317>37A/A3?C3<13=4/1B7=<AB=B/93 E63<>@3>/@7<5/A>31747127A6 !<B67A1=<B3FB
B63AC0831BA=4B63/1B7=<A1/<03@35/@232/AB6397B163<E/@3 7B3;A/<2A=;32/B/3 5 
7<5@3273<BA1/<03D73E32/A=0831BA 037<5>@=13AA320GB6397B163<E/@3 !4E3D73E/<7B3;
=4 97B163<E/@3/A/AC0831B7D3=0831BB63@317>31/<03D73E32/AB637<B3@/1B7=<=4;C:B7>:3
97B163<E/@3 7B3;A  =E3D3@ 7</BG>71/:@317>3B6397B163<E/@3 7A <=B 23A1@7032 1:3/@:G74/B
/:: !<B67A@3A>31BE303:73D3B6/B@317>3A /@3 A7;7:/@B=CA31/A3A !47B031=;3A>=AA70:3B=
723<B74G97B163<E/@34@=;@317>3A <=D713 >@=5@/;;3@A 1/<:3/@<6=EB=723<B74G=0831BA4@=;
B63CA31/A3 /<2B63@30GAC113AA4C::G 723<B74G=0831BA )67A7AE6G=C@;3B6=2CB7:7H3A /@317>3
<=B/A=4BE/@3CA31/A3/AB63B@/7<7<5B/@53B +31/;3C>E7B6B634=::=E7<5AB3>AB=B@/7<
B63B67<97<5=4=0831B723<B7471/B7=<(B3> 3@7D3 B63=0831BA3 5 AC0831B/<2B/@53B 1=@@3A>=<27<5B=3/16/1B7=<=4B63@317>3(B3>  '3E@7B3B63/1B7=< E7B6B6323@7D32=0831BA(B3>  @/E />71BC@3=43/161==97<5/1B7=< 7<B63@317>3CA7<5B63723<B74732=0831BA (B3>  6319E63B63@B63>71BC@37<(B3> A/B7A473A B637;/57<32A13<3 !4<=B723<B74GB63
;7AA7<5=0831BA/<2@3BC@<B=(B3> A/<3F/;>:3 =4B63AB3>A:3BKA4=1CA=<B631==97<5/1B7=< I0=7:>=B/B=3AJ7<75C@3 !4E3/AAC;3 B63AC0831B=4B67A>@=13AA7A/I>3@A=<J /<2B63B/@53B7A/I>=B/B=J (B3> 
B631==97<5 /1B7=< 1/<03@3E@7BB3</AI/ >3@A=<0=7:A>=B/B=3AJ (B3> !4E32@/E/>71BC@3:793 B63=<37< 75 CA7<5B633FB@/1B32=0831BA(B3> /<2 47<2B6/B7B7A27443@3<B4@=;B63
7;/57<32 A13<3 /A A6=E< 7< 75  E3 3FB@/1B B63 ;7AA7<5 =0831BA 0G 3F/;7<7<5 B63
27443@3<13A03BE33<B632@/E<>71BC@375 /<2=C@7;/57</B7=<=4B63A13<3(B3>AC16
/AIAB=D3JI47@3JI>=BJ/<2IE/B3@J +3B63<@3BC@<B=(B3> /<2@3E@7B3 B63I0=7:>=B/B=3AJ/1B7=<=<B630/A7A=4B63A3 23@7D32=0831BA/AI / >3@A=<BC@<A=</AB=D3 B63AB=D3BC@<A=< B634:/;3 B634:/;363/BA / >=B B63>=B63/BAB63E/B3@/<2 B6363/B32E/B3@
0=7:AB63>=B/B=3AJ !<(B3>E32@/EB63A13<3A4@=;B63A3=0831BA/<274E3A/B7A4732E7B6
B63>71BC@3B63 >@=13AA3<2A7<271/B7<5B6/BE36/D323@7D32/23?C/B3=0831BA !4 <=BE3 5=
0/19B=(B3>/<2:==94=@B63;7AA7<5=0831BA/5/7<
 '!$('++'* * ('++'* ,+'&!'&* ,+))'+*
 +'&!'&*
 +))'+* ,+,,%)* ,+ % !.%/'&&!*"
&=B/B=A/:/2@317>3"
==97<5A13<32@/E<
7</11=@2/<13E7B6
723<B74732=0831BA

"
!;/57<321==97<5
A13<3%!%& # ##!'"#)63 =0831B 723<B7471/B7=< B@/7<7<5 AGAB3; 7A 7<B3<232 B= >@=D723 / >:/13 E63@3 CA3@A 1/<
3F>3@73<13 6=E B= 723<B74G =0831BA 0G CA7<5 @317>3A B63@30G 3</0:7<5 B63; B= /1?C7@3 B63
B67<97<5 <313AA/@G B= 723<B74G =0831BA 4@=; B63 >@=13AA   AGAB3; 1=<475C@/B7=< 27/5@/; 7A
A6=E<7<75  )63AGAB3;1=<A7ABA=4BE=7<B3@4/13A/<2/43320/194C<1B7=</<27B CB7:7H3A
@317>3A7</@317>32/B/0/A34=@B63B@/7<7<5 116)63723<B7471/B7=<7<B3@4/137A
/ A1@33< 4=@ 7<>CBB7<5 B63 =0831BA
&+!0+!'&!&+)
723<B74732 4@=; 3/16 /1B7=< =4 B63  
@317>3  )63 CA3@ 7<>CBA B63 AC0831B
/<2B/@53B 4@=;B6327A>:/G32 @317>3
'&0)%+!'&#A3<B3<13 /<2 CA3A B63; B= @3E@7B3!&+),&+!'&
B63 A3<B3<13  )63 1=<47@;/B7=<
7<B3@4/137A/A1@33<E63@3B63CA3@2@/EA/" (GAB3;1=<475C@/B7=<>71BC@3CA7<5B63=0831BAAC0831B/<2B/@53B
7<>CBB=B63723<B7471/B7=<7<B3@4/13 B=1=<47@;E63B63@/23?C/B3=0831BA6/D3033<7<>CB 4B3@
B63>71BC@37A2@/E<B631=<47@;/B7=< 7<B3@4/1357D3A /1=;;3<BB=B63CA3@/A97<574B63@3/@3
/<G:/197<5=0831BA !4<=B7B47<7A63AB63>@=13AA !4G3A7B/1B7D/B3A B63723<B7471/B7=<7<B3@4/13
/5/7< )63 723<B7471/B7=< 7<B3@4/13 /A9A B63 CA3@ B= @3E@7B3 B63 @317>3 A3<B3<13 CA7<5 B63:/197<5 =0831BA  !B B63< 16319A E63B63@ B63 E@7BB3< A3<B3<13A @3>@3A3<B B63 /1B7=< =4 B63=@757</:>@=13AA/<274<=B7B57D3A43320/19B=;=274GB63A3<B3<13A /5/7<

 



A3<B3<131=@@3A>=<27<5B=/</1B7=<7<B63@317>3 @3>@3A3<BAB6/B B63 AC0831B3F31CB3A B63/1B7=< /<2B63B/@53B7A;/<7>C:/B320GB63/1B7=< !4B63A3<B3<137A@3E@7BB3<0GCA7<5A3D3@/:
A3<B3<13A@3:/B32B=AC0 /1B7=<A 7B7A7;>=@B/<BB6/B B6347@ABAC0831B=4B63AC0 /1B7=<AB63
:/AB B/@53B =4 B63 AC0 /1B7=<A /<2 B63 :/AB /1B7=< 03 B63 A/;3 /A B63 =@757</: /1B7=<  )63
AC0831BA/<2B/@53BA=4=B63@/1B7=<A/@37<B3@;327/@GB6/B7A 7<AB3/2=47<271/B7<5/27@31B
;/<7>C:/B7=<=4B63B/@53B0G B63AC0831B7B;3/<AB6/B=B63@A/@3B=:2B=;/<7>C:/B3B63B/@53B 75C@37::CAB@/B3AB63@3:/B7=<A03BE33<B63=@757</:/1B7=</<2B63AC0 /1B7=<A (C>>=A3B6/B
B63=@757</:/1B7=<&7A/>@=13AA)!!&$''#!&
+!'&
4@=;AC0831BB=B/@53B  !4B63
/1B7=< 7A @3>:/132 0G AC0,!-!''#!&/1B7=<AB6347@ABAC0831B<332AB=+!'&*
03B63:/ABB/@53B<332AB=03
21"+/<2B63:/AB/1B7=<;CAB03& 
1 +!'&%63@ /1B7=<A / 0 /<2 1 /<2" '3:/B7=<A03BE33<=@757</:/1B7=</<2AC0=B63@=0831BA,-/<2. ;756B
/1B7=<A03 >:/132 /B D/@7=CA >=A7B7=<A
03BE33<B63;)6343320/194C<1B7=<16319AE63B63@B63@3E@7BB3<@317>3AA/B7A4GAC161=<27B7=<A !<
/227B7=<B=B6347@ABAC0831BB63:/ABB/@53B/<2B63:/AB/1B7=<7B<332AB=16319B6/BB63@3/@3<= =;7AA7=<A 7< B63 B@/<A43@ =4 /1B7=<A  )63 /1B7=<A /@3 1=<<31B32 E63< B63 B/@53B =4 B63
4=@;3@ /1B7=< /<2 B63 AC0831B =4 B63 :/BB3@ /1B7=< /@3 B63 A/;3  G 1=<A723@7<5 B63A3
1=<27B7=<AB6343320/194C<1B7=<23B3@;7<3AE63B63@B63@3E@7BB3<@317>37AB63A/;3/AB63
=@757</:=<30GB634=::=E7<5AB3>A
(B3> 3B3@;7<3B63AC0831B/<2B63B/@53B =43/16AC0 /1B7=< 7<B63@3E@7BB3<@317>3(B3>  !4 B63 >@3D7=CA B/@53B /<2 B63 AC0A3?C3<B AC0831B /@3 B63 A/;3 3:7;7</B3 B63
B/@53B AC0831B/<2>@3D7=CA/1B7=</<21=;07<3B63;7<B= =<3/1B7=<(B3>  !4B63@3E@7BB3<@317>37A16/<532B==<3/1B7=<1=;>/@37BAAC0831BB/@53B/<2/1B7=<
B=B63=@757</:/1B7=<
3@3E3A6=E/<3F/;>:3E7B6 B63 =@757</:@317>3I/>3@A=<0=7:A>=B/B=3AJ /<2B63
@3E@7BB3<@317>3/AI />3@A=<AB/@BA C> / AB=D3  B63 AB=D3BC@<A=</4:/;3 B634:/;363/BA/>=B/<2 B63 >=B0=7:AB63>=B/B=3AJ (7<13B63B/@53B =4 /<2B63AC0831B=4/@3
B63A/;3 7 3 I/AB=D3JB63A3/1B7=<A/@31=;07<32/A I/>3@A=<BC@<A=<B6347@3=4 /AB=D3 
A/@3AC:BE36/D3I />3@A=<BC@<A=<B63 4:/;3  B63 4:/;3 63/BA/>=B /<2 B63 >=B
0=7:AB63>=B/B=3AJ (7<13B63B/@53B =4 /<2B63AC0831B=4/@3B63A/;37<B63@3E@7BB3<@317>3 B63G/@31=;07<32B=1@3/B3 I/>3@A=<63/BA/>=BJ A/@3AC:B />3@A=<63/BA/>=B /<2   / >=B 0=7:A B63 >=B/B=3AJ 7A 1@3/B32  !< B63 A/;3 E/G I  / >3@A=< 0=7:A B63>=B/B=3AJ 7A 53<3@/B32  (7<13 7B 7A B63 A/;3 /A B63 =@757</: @317>3 B63 @3E@7BB3< @317>3 7A
@35/@232/AB63A/;3/1B7=</AB63=@757</:=<3 117)6343320/194C<1B7=<>@=D723A43320/197</11=@2/<13E7B6B63@3AC:B !4B63=@757</:
@317>3/<2B63@3E@7BB3<=<3/@3B63A/;37B57D3AI)63@3 E@7BB3<@317>37A/>>@=>@7/B3 J!4<=B
7B>=7<BA=CBB637</>>@=>@7/B3<3AA/<2AC553ABA6=EB=47F7B )63@3/@3BE=>=AA70:31/CA3A
=47</>>@=>@7/B3<3AA )6347@AB7AB6/BB63@37A/<=;7AA7=<7<B63B@/<A43@=4>@=13AA7<5 =@
3F/;>:374B63AC0 /1B7=<A/@3I />3@A=<AB/@BAC>/AB=D3/<2 />=B0=7:AB63>=B/B=3AJ
A=;3/1B7=<A;756B03;7AA7<503BE33<B63BE=/1B7=<AA7<13B63B/@53B=4/1B7=< /<2B63
AC0831B=4/1B7=</@327443@3<B !<B67A1/A3B6343320/194C<1B7=<A6=EAB63CA3@E63@3B63
1==97<5AB3>A/@3<=B1=<<31B32/<257D3A/67<B74B63AC0831B/<2B/@53B/@327443@3<B )63
A31=<21/CA37AB6/BB631=;07<32/1B7=<7A27443@3<B4@=;B63=@757</:=<3 !<B67A1/A3B63
AGAB3; 57D3A B63 1=;;3<B I)63 /1B7=< 7< B63 =@757</: @317>3 /<2 B63 AC0 /1B7=<A 7< B63
@3E@7BB3<@317>3/@3<=BB63A/;3J  !! ## #!

75C@3  A6=EA /< 7;/53 =4 B63 723<B7471/B7=< 7<B3@4/13 A1@33<  +63< / @317>3 </;3 7A
A3:31B32B63/1B7=<A7<B63A3:31B32@317>3/@327A>:/G327<B63@317>327A>:/G/@3/ )63CA3@
<332AB=7<>CBB63AC0831BB/@53B/<2/1B7=<3FB@/1B324@=;B63@317>37<B=B631==97<5/1B7=<
7<>CB /@3/  G A3:31B7<5 / @=E 7< B63 1==97<5 /1B7=< 7<>CB /@3/ /<2 1:7197<5 B63 I22 '=EJ
0CBB=</<3E@=E7A/2232C<23@B63A3:31B32@=E G1:7197<5=<3/1613::=4B63@=EB63
AC0831BB/@53B/<2>@=13AA1/<033<B3@32 )=23:3B3B63@=E1:719=<B63@=EB=0323:3B32
/<2 A3:31B B63 I3:3B3J 0CBB=<  )63 =@23@ =4 B63 7<>CB 7< B63 1==97<5 /1B7=< 7<>CB /@3/
@3>@3A3<BAB63=@23@=4B631==97<54@=;AB/@BB=47<7A6 +63<B63I"C25;3<BJ0CBB=<7A1:71932
B63 43320/19 4C<1B7=< 3D/:C/B3A E63B63@ B63 7<>CB 1==97<5 /1B7=<A /@3 />>@=>@7/B3 /<2
27A>:/GAB63@3AC:B7</;3AA/530=F   )!!&$''#!&()',)

,!-!*!'&
,!-!''#!&()',)*







     









 

  " !23<B7471/B7=<7<B3@4/13  !# #!

)63 1=<47@;/B7=< 7<B3@4/13 7A / A1@33< 4=@ 1@3/B7<5 / >71BC@3 =4 B63 1==97<5 A13<3 4=@ 3/16
/1B7=<CA7<5B63=0831BA723<B747327<B63723<B7471/B7=<7<B3@4/13 75C@3A6=EA/<3F/;>:3=4
B631=<47@;/B7=<7<B3@4/13A1@33< )63CA3@1/<A3:31BB63/1B7=<B=0323>71B327<B631==97<5/1B7=<27A>:/G:7AB +63</B/@53B/1B7=<7A16/<532/<3E1/<D/A4=@B63/1B7=</>>3/@A7<B63>71BC@31@3/B7=</@3//<2B63CA3@1/<2@/E/>71BC@3@3>@3A3<B7<5B63A13<3=4B63/1B7=< )63>71BC@37A2@/E<0G>:/17<5B637;/53A27A>:/G327<B637;/5327A>:/G/@3/=<B=B63>71BC@3
1@3/B7=< /@3/  )63 7;/53 27A>:/G /@3/ 27A>:/GA 7;/53A =4 =0831BA AB=@32 7< B63 AGAB3; 7</2D/<13B6/B;/G/>>3/@7<B631==97<5A13<3 )63CA3@1/</:A=/22B637@=E<7;/53A0G
2@/557<5/<22@=>>7<57;/5347:3A )=2@/EB63>71BC@3/CA3@A3:31BA/<=0831B4@=;B63:7AB
=4723<B74732=0831BA1:719A=<B637;/53@3>@3A3<B7<5B63=0831B/<2B63<1:719A=<B63I$/B16

118

(B@7<5 B= !;/53J 0CBB=< B=
               
27A>:/G B63 1=@@3A>=<27<5

7;/53  )63 CA3@ 1/< /:A=;=D323:3B3 /<216/<53
B63 A7H3 =4 B63 7;/53 +63< B63 1==97<5 A13<3A=4 /:: /1B7=<A /@3 2@/E<
CA7<5B63723<B74732=0831BA/<2 B63 I7<7A6 @3/B7=<J
0CBB=< 7A 1:71932 B63
AGAB3;
>@3A3<BA/;3AA/53 B= 1=<A723@
E63B63@
B63
2@/E<
>71BC@3A @3>@3A3<B B63
CA3@KA 7;/57</B7=< =4 B63
A13<3 )63 7<B3<B =4 B67A
;3AA/53 7A B= >@=;>B B63  
CA3@ B= B67<9 /0=CB" =<47@;/B7=<!<B3@4/13
E63B63@ B63@3 /@3 /<G;7AA7<5 =0831BA !4B63@3/@3B63AGAB3;;=D3AB=B63723<B7471/B7=<7<B3@4/13/5/7</<2 :3BA
B63CA3@ AC027D723B631==97<5/1B7=< CA7<5B63;7AA7<5=0831BA@31=5<7H327<B631=<47@;/B7=<
7<B3@4/13 !4B63@3/@3<= ;7AA7<5=0831BAB63=>3@/B7=< B3@;7</B3A +36/D323D3:=>32/<%%& B@/7<7<5AGAB3;4=@723<B74G7<5=0831BAB6/B/@3@3A>=<A70:34=@B63
AC0 /1B7=<A=4 @3?C7@3;3<BA>317471/B7=<A AC16/A CA31/A323A1@7>B7=<A )63B@/7<7<5AGAB3;
7<B@=2C13A /@317>3 7<AB3/2=4 / CA31/A3 /A/B@/7<7<5B/@53B E6716;/93A7B3/A73@4=@<=D713A
B=7;/57<3B63=0831BA/<2B637@7<B3@/1B7=<A 7<B63A13<3 %C@AGAB3;63:>AB63CA3@B=723<B74G
D/:72=0831BA0G6/D7<5B63; 2@/E/>71BC@3=4B631==97<5A13<3E7B6B63723<B74732=0831BA !B/:A= >@=D723A / 43320/19 4C<1B7=< B= 16319 E63B63@ / @3E@7BB3< @317>3 CA7<5 B63 723<B74732
=0831BA@3>@3A3<BAB63A/;3/1B7=<A /AB63=@757</:@317>3)63@317>3A6/<2:320GB63AGAB3;7<B67AABC2GE3@3:7<3/@:G1=<<31B32E7B6=<:G=<3AC0831B /<2 =<3B/@53B>3@ >@=13AA  =E3D3@B63@3 /@3 /:A= /1B7=<AB6/B6/D3;C:B7>:3B/@53BA!<AC161/A3AA=;3/1B7=<A/@3>3@4=@;327<>/@/::3: /<2 /@3B63@34=@3 <=B1=<<31B32:7<3/@:G )=1=>3E7B6AC16@317>3AB6343320/194C<1B7=<<332A B=037;>@=D32A=/AB=16319 B63
D/:727BGE7B6<=B=<:GA7;C:B/<3=CA /1B7=<A0CB/:A=/1B7=<A=11C@@7<5A3>/@/B3:G


)0+05'''(=(0  678,(85)1,*:80,5:,+86.8(4405.#905.;.4,5:,+,(30:?,1#/,1'-,*-2/,,*-$2*'+#"',"#!#,1,,-31'-,    
=(802(   , $0330,89      #9, 6- 30*, $09;(3 5<08654,5: 05 ",(*/05. (5+ ,(8505.
)1,*:80,5:,+86.8(4405.,/-!##"',%0-$&#  ,,2*#0#/!&-,$#/#,!#-,
-21&$/'!,,01'121#-$-+.21#/!'#,1'010,",$-/+1'-,#!&,-*-%'01077  
,9:=0*20(?(8(4(5  ,:/6+636.?(5+8*/0:,*:;8,6-$,/-!##"',%0-$&# 6+.-0'2+-,-$14/#'02*'71'-,77   
0(5$$(86?0(*(4(  ,(8505.)1,*:80,5:,+86.8(4405.65*,7:9"/86;./
$09;(35(36.0,9/,0!1'-,0-,#/,',%#!&,-*-%'#0   !,5.&%&(:046,"%  ,(8505.)1,*:80,5:,+86.8(4405.(8(+0.4$0((4,(9,+ ,(8505. (4,  306: !:;+? &# ,1#/,1'-,* -2/,* -$ 2*1'+#"'  10
..*'!1'-,0     ",*/5636.0*8:95*  #!&',%-2/0#*$ 1&"'1'-,',.,#0#
",0-   (@@(5    /1-,-+6 ," 5-,-+6 ',  (#!1/'#,1#" &',)',%  2,'-/ '%&!&--*12"#,10#/!#.1'-,0-$ (#!1/'#,1#"0'!-,!#.10 %68205.86;7 ,768:965
"0!65556<(:065(5+",*/5636.?05647;:,8!*0,5*,+;*(:065

119" !   !" " "   #" !
#" !"$  #" ! #"                    !  #     !" # !       " # 
:,;(64< (* 17  <4>4.,6<0,/482=5466=A34.3<0;?4<060,<80<=>90B,7480>0B>=1<977?6>4:60
:0<=:0.>4@0= <,>30< >3,8 <06C482 =9606C 98 A3,> 4= A<4>>08 :6,C ,8 0==08>4,6 <960 48
<0,/482 ,.,/074. ,<>4.60= >9 ,.;?4<0 342360@06 64>0<,.C '0@0<,6 =C=>07= 3,@0 -008
/0@069:0/>9=?::9<>>3047:<9@0708>91.<4>4.,6<0,/482=5466=39A0@0<>30<04=89
=C=>07 >3,> 19669A= >30 60,<80<= >3485482 ,8/ >30 :<9.0== 91 <0,/482 ,<>4.60= ,> >30
.98>08>60@06,8/:<9@4/0=100/-,.5>3,>.,:>?<0=>30=07,8>4..9880.>498==:0.414.>9>30<0=0,<.3/97,48(30<019<0A0:<9:9=0,=?::9<>=C=>0719<.<4>4.,6<0,/48291
,.,/074. :,:0<= A34.3 3,= , 1?8.>498 >9 306: 60,<80<= 9<2,84D0 >30 .98>08>= 91
,.,/074.:,:0<=>9:<979>0,A,<080==910==08>4,6@40A:948>=>3,>08.9?<,20=>304<.<4>4.,6<0,/482-,=0/98=07,8>4..9880.>498==:0.414.>9>30<0=0,<.3/97,48 <4>4.,6<0,/48291,.,/074.:,:0<=<0=0,<.3,.>4@4>C98>9692C/C8,74.
48>0<@08>498-,=0/9898>9692C

  
",(+05.(5+>90;05.:2033::<*/(:<5+,9:;(5+05.(+=(5*,+;,?;:*90;0*(33@,?(40505.;/,4
(5+>90;05.7(7,9:(9,*(33,+/0./ 3,=,330;,9(*@;/,5,,+;6+,=,367:<*/:2033:/(:),,5
7605;,+6<;059,*,5;@,(9:05:,*65+(9@(5+/0./,9,+<*(;065 59,:765:,;6;/,,?7(5:065
6- ;/, 5<4),9 6- 3,(95,9: >/6 ,5;,9 <50=,9:0;0,: (5+ ;6 :6*0(3 +,4(5+: 4(5@ <50=,9:0;0,:
/(=, ),.<5 6--,905. =(906<: :;<+@ (5+ (*(+,40* :2033: (: 7(9; 6- -09:; @,(9 ,+<*(;065 69
9,4,+0(3 ,+<*(;065  90;0*(3 9,(+05. :2033: >/0*/ 9,8<09, 3,(95,9: ;6 ,?(405, ;,?;: -964
4<3;073, 7,9:7,*;0=,: 9(;/,9 ;/(5 9,3@05. :63,3@ 65 >/(; 0: >90;;,5 73(@ (5 ,::,5;0(3 963, 05
9,(+05.(*(+,40*(9;0*3,:;6(*8<09,/0./ 3,=,330;,9(*@ $6+,=,367;/,:,:2033:0;0:,::,5;0(3;6<5+,9:;(5+;/,>90;,9:;/6<./;:05(>(@;/(;(336>:;/,9,(+,9;6-6336>;/,096>5;/6<./;:
(5+-694;/,096>50+,(:(-;,9;/696<./3@,?(40505.;/,46990:,;(3  <2,  (930:3,  
!,(9:65!  ,;(3   
26*/0:;(;,:;/(;;/,9,(9,;/9,,(:7,*;:;6*90;0*(39,(+05.(*;0=0;0,: 26*/0 
0.<9,  $/,-09:;0:;/,(:7,*;6-,?(40505.65,:6>5<5+,9:;(5+05.*6479,/,5:065
4650;6905. )@ ,?(40505. >/,;/,9 65, <5+,9:;(5+: ;/, (<;/69: (9.<4,5; *699,*;3@ ;/,:,*65+0:,?(40505.;/,>90;,9:36.0*)@,?(40505.;/,36.0*(3305,6-;/,>90;,9(5+;/,;/09+
0: ;/, 796+<*;0=, (:7,*; 6- +,,7,505. 65,: ;/05205. )@ *65:0+,905. 5,> 796)3,4: (5+(3;,95(;0=,:63<;065:;6;/,>90;,9:36.0*;/(;65,/(:<5+,9:;66+ 
5 .,5,9(3 >, +,-05, *90;0*(3 9,(+05. (: 796+<*;0=, (*;0=0;0,: ;6 +,,7,5 65,: 6>5;/05205.>/03,<5+,9:;(5+05.(5+,?(40505.;/,(<;/69:36.0*>/03,9,(+05.(*(+,40*(9;0*3,:
(5+;/0::;<+@(04:;6+,=,367*90;0*(39,(+05.:2033: 5;/0::;<+@>,,5*6<9(.,3,(95,9C::,3- +0(36.<,05>/0*/;/,(<;/69:;/6<./;:(5+65,: 6>5 ;/6<./;: 65 ;/, (<;/69: ;/6<./;: -694,+ -964 ;/, (<;/69: (9.<4,5;(;065 (9,05;,9(*;0,+ (:,+65;/,0+,(>,79676:,(:<7769;:@:;,4-69*90;0*(39,(+05.6-(*(+,40*
7(7,9:>/0*//(:(-<5*;065;6/,373,(95,9:69.(50A,;/,*65;,5;:6-(*(+,40*7(7,9:)@
*647(905.;/,090+,(:>0;/;/6:,6-(<;/69:(5+(-<5*;065;6*647(9,;/,69.(50A,+*65;,5;:
6-9,(+05.(5+;679646;,(>(9,5,::6-*90;0*(3=0,>7605;: 




120

         !90;0*(3!(7,9",(+05.#<7769;#@:;,4

         &, +,=,367,+ ( *90;0*(3 9,(+05. :<7769; :@:;,4 ;/(; ,5*6<9(.,: 796+<*;0=, +0(36.<,  $/,:@:;,405;,9-(*,0::/6>5050.<9, $/,:@:;,4/(:(-<5*;065;6:<7769;;/,69.(50A(;065
6- ;/, *65;,5;: 6- (5 (9;0*3, )@ *647(905. ;/, (<;/69C: (5+ 65,: 0+,(: (5+ ( -<5*;065 ;6
*647(9,;/,69.(50A,+9,(+05.*65;,5;:(5+79646;,(>(9,5,::6-*90;0*(37,9:7,*;0=,: $/, :@:;,4 0: ( :,9=,9 *30,5; 46+,3 ;/(; *(5 ), <:,+ (5@>/,9, (5+ (5@;04,  @<736(+05.(!-03,6-(5(9;0*3,3,(95,9:*(59,(+0;05;/,:(4,-694(;(:;/,@<:<(33@9,(+0;(5+*6479,/,5+;/,(9;0*3,>0;/;/,09+0(36.<, $/,3,(95,99,(+:;/,7(7,9+0:73(@,+05;/,
7(7,9+0:73(@(9,(0.  00 5;/,8<,:;06579,:,5;(;065(9,(0.  000(8<,:;065;/(;
,5*6<9(.,: +,,7,505. (5+ *65:0+,9(;065 6- ;/, *65;,5; 6- ;/, 7(7,9 , .  &/(; (9, ;/,+0--0*<3;0,: ;/(; 3,(95,9: /(=, ;/(;
79,=,5; ;/,4 -964 (*/0,=05. ;/,09   3,(9505. .6(3:B 0: 79,:,5;,+ )@;/,:@:;,4 %:05.;/0:8<,:;065(:(.<0+,-699,(+05.*6479,/,5:065
   ;/, $/6<./; 9.(50A(;065 (70.  0 796=0+,: ( ;/05205.
69.(50A(;065 ,5=09654,5; ;/(; 
  
(336>: ;/, 3,(95,9 ;6 :,7(9(;, ;/,(<;/69:0+,(:-964/0:69/,96>5
(5+ ;6 69.(50A, ;/,4 9,3(;0=, ;6,(*/6;/,9690 ,;(3     

   
  
$/,3,(95,99,(+:;/,(9;0*3,>/03,    

4(9205.0;(5+,?79,::,:/0:69/,90.<9, ?(473,6-79,:,5;05.56+,:;/(;,5*6<9(.,
<5+,9:;(5+05. 6- ;/, (<;/69:
:,3- +0(36.<,),;>,,5;/,(<;/69(5+/0: /,9;/6<./;
(9.<4,5;: (5+ 0+,(: 65 ;/, 4(7<:05.;/,8<,:;065       &/,5(3,(95,9(++:(8<,:;065;/(;/, :/,;/052::/6<3+),*65:0+,9,+(:(8<,:;06556+,
65 ;/, ;/6<./; 69.(50A(;065 4(7 ( 7<973, 79,+0*;065 56+, 3(),3,+ @6<9 79,+0*;065 0:
(<;64(;0*(33@ (++,+ (: ( */03+ 56+, ;6 ,5*6<9(., ;/, -694(;065 6- /0: /,9 0+,( 0.  0 &/,5;/,3,(95,99,:765+:;6;/0:79,+0*;06556+,(569(5.,05;,979,;(;06556+,3(),3,+121

(<;/69:*3(040:(<;64(;0*(33@(++,+0. 00 :;/,3,(95,9*65;05<,:;69,(+/,69:/,
0:,5*6<9(.,+;6+0.+,,7,99,3(;0=,;6/0:69/,979,+0*;065,+0;05.(5+76:0;06505.;/,56+,:
*65;,5;:>/,5;/,(<;/69:*3(04(77,(9:3(;,9 *90;08<,56+,;/(;*90;08<,:;/,(<;/69:
0+,(: 3(),3,+ '6<9 90;08<, 0: (3:6 (<;64(;0*(33@ (++,+ 0.  000  $/0: *90;08<, 56+,
9,8<09,: ;/, (::0.54,5; 6- :,4(5;0* ;(.: ;/(; ,?79,:: ;/, *90;08<, =0,>7605;: =(3<,
1<+.4,5;: 0::<,: 6705065: ,;*  >/0*/ (9, /0,9(9*/0*(33@ (99(5.,+ 05 (+=(5*, :6 ;/(;
3,(95,9:*(5*647(9,;/,0979,+0*;065:>0;/;/,(<;/69:*3(04:(5++,,7,5;/,09;/05205. 5;/0:>(@;/,56+,:(9,79,:,5;,+:;,7)@:;,705;/,69+,96-79,+0*;06505;,979,;(;065(5+*90;08<,-69,(*/8<,:;065(5+;/,:,*6.50;0=,(*;0=0;0,:(9,(04,+(;4(205.;/,(<;/69:
*3(04: 9,3(;0=, ;6 ;/,09 6>5  $/,:, (9, 7,9-694,+ 05 ( */(05 302, 4(55,9 :@:;,4(;0*(33@
9,79,:,5;05.;/,796*,::6-05;,95(3:,3- +0(36.<,+<905.9,(+05.05(405+4(7-694(;   

96:: 9,-,9,5*05.-<5*;065),;>,,5(56+,(5+(9;0*3,;,?;:0:(3:6796=0+,+ 56+,(5+;/,
;,?;0;9,-,9:;6(9,/0./30./;,+05;/,:(4,*6369(**69+05.;6;/,56+,;@7, $/0:0:(*/0,=,+)@ *65=,9;05. ;/, (9;0*3, -03, 9,(+ 05 7+- -694(; ;6 $ -694(; ,5*36:05. ,(*/ ;,?; 6- ;/,(9;0*3,>0;/;(.:(5+.0=05.0;(<508<,(5+(::0.505.;/,<508<,6-,(*/56+,;6;/,;,?;
05-694(;065:,3,*;,+)@;/,3,(95,9 $/0:4,*/(50:4(336>:;/,:@:;,4;6*(7;<9,;/,;/6<./;:
6- 4<3;073, 3,(95,9: >/6 /(=, 9,(+ ;/, :(4, 7(7,9 (5+ >/(; ;/,@ 9,(+  $/<: ;/, :@:;,4
,5()3,:3,(95,9:;669.(50A,;/,09;/05205.>/03,,?(40505.;/,*65;,5;:6-;/,7(7,9(5+;/,
9,:<3;:6-;/,09;/05205.(:(7796790(;,+<905.9,(+05.(5+;6*647(9,;/,9,-,9,5*,:;9<*;<9,
6-;/,7(7,9>0;/;/,4(7:6-6;/,93,(95,9:>/6/(=,9,(+;/,:(4,7(7,9(-;,99,(+05.0;     $/0::;<+@-6*<:,+65;/9,,(:7,*;:6-+,=,36705.*90;0*(39,(+05.:2033: &,+,-05,+ 
(: ( 796+<*;0=, (*;0=0;@ ;/(; +,,7,5: 65,: ;/05205. >/03, <5+,9:;(5+05. (5+,?(40505.;/,(<;/69:36.0*059,(+05.(*(+,40*(9;0*3,: $/,-<5*;065(30;@6-;/,:@:;,4*6<3+
653@ ), 9,(30A,+ )(:,+ 65 ;/, 65;636.@ 05 >/0*/ *65*,7; +,-050;065: :7,*0-@ :,4(5;0*
9,3(;065:/07: :7,*0-0* ;6 ;/, 05;,330.,5; ;<;6905. :@:;,4 9,:,(9*/ +64(05 /,5    +,
30=,09("   0A6.<*/0"   &,>033=,90-@>/,;/,9;/,*90;0*(39,(+05.*6479,/,5:065:<7769;:@:;,4+,:*90),+05
#,*;065*(5),<:,+05(*;<(379(*;0*, -;,9*65-09405.;/(;;/,:@:;,4>692:(:05;,5+,+>,>033*65+<*;(8<,:;0655(09,:<9=,@;6+,;,9405,>/,;/,9<:05.;/,:@:;,4;69,(+7(7,9:
/(:(76;,5;0(33,(+05.;6(*/(5.,053,(95,9:(;;0;<+,:;6>(9+*90;0*(39,(+05.*6479,/,5:065    
308!/0$64@04<,&83,8.482.<4>4.,6<0,/482=5466=A4>3>0.389692C=C=>07,>4.<0@40A
97:?>0<=/?.,>498    
?50# ,<64=60
(30/0@069:708>91<0,/482.97:<0308=498(30$B19</,8/-995
91&0,/482   #9<<4='%%34664:=!"9A64>0<,.C484>=1?8/,708>,6=08=04=.08><,6>9=.408>414.64>0<,.C'.408.0/?.,>498    
$59.34+(30&960=91'><?.>?<,6=:0.>=91(0B>=98<4>4.,6&0,/482(30?660>4891>30
<,/?,>0'.399691/?.,>49848>30)84@0<=4>C91(95C9  
%0,<=98 %  &90360< ! & 960    ?11C      0@069:482 0B:0<>4=0 48 <0,/482
.97:<0308=498*3,>&0=0,<.3,=>9',C-9?>&0,/4828=><?.>498   "4D92?.34&(?>9<4,69898>96924.,60824800<482#0A080<,>49897:?>482    3>>:= /949<2  "9<4 # ,C,=34 +  '0>,   $8>9692C-,=0/ (39?23> $<2,84D,>498 '?::9<> 'C=>07 >9
%<97:>&0,/480==918>08>498'3,<482,8/>=!982>0<7%<,.>4.0(309?<8,691819<7,>498,8/'C=>07=48/?.,>498    122

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Overcoming Barriers to Sustainable
Dissemination of L2 Learning Resources:
An Integrated Framework for Creating andDistributing Dialogue Scenarios
Emmanuel AYEDOUNa*, Yuki HAYASHIb & Kazuhisa SETAba
Faculty of Engineering Science, Kansai University, Japanb
Graduate School of Informatics, Osaka Metropolitan University, Japan*emay@kansai-u.ac.jp
Abstract: Language learning resources, such as dialogue scenarios, are important for
supporting second language (L2) learning by engaging learners in simulated
conversations or role-plays. However, there are often barriers to the wider
dissemination of these resources, including the lack of a standard format for distributing
them, the difficulties of maintaining and updating them over time, and the need for
effective means of tracking their use and impact. This paper presents an integrated
framework for fostering the wider dissemination of L2 learning resources, with a focus
on dialogue scenarios. The framework is designed to address the above challenges by
providing a novel approach for creating and distributing dialogue scenarios, as well asa set of guidelines to ensure that they are pedagogically sound and compatible withlearning management systems (LMS). The paper also discusses the potential benefitsof the framework for L2 learners and educators and outlines future directions for the
project.
Keywords: Dialogue Scenarios, L2 learning, Authoring interface, SCORM-compliance

1. Introduction
The Second language (L2) learning has the potential to be a transformative experience,
opening up new opportunities for communication and cultural exchange. Yet, many learners
are unable to access high-quality language learning resources due to financial, geographic,or technological barriers (Zimmerman, 2014).
On the other hand, the use of learning management systems (LMS) in education, andmore specifically in computer-assisted language learning, has become increasingly common
in recent years, as they provide a convenient platform for organizing and delivering course
materials, managing assessments, gathering learning data, and facilitating communication
between learners and instructors (Watson & Campana, 2017). In order for an LMS to beeffective, it must be properly configured and populated with high-quality learning resources(Kataoka et al., 2018).
For instance, learning resources such as dialogue scenarios are an effective tool for
L2 learning, as they can engage learners in simulated conversations or role-plays, helping
them to develop their language skills and confidence in using the target language (Willis &Willis, 2007). Overall, the use of dialogue scenarios in L2 learning can support learners’communicative competence (Johnson & Valente, 2008), cultural awareness, and cancontribute to a more effective and engaging language learning experience (Ayedoun et al.,2021). However, the distribution of these learning resources over LMS has historically been
limited, with most being developed for a specific course or program and not widely available
for reuse. In addition, the implementation of new dialogue scenarios requires a significant
amount of knowledge engineering effort and expertise in dialogue design (Ayedoun et al.,2021), which can be a barrier to their wider dissemination in learning management systems

123

(LMS). Other barriers may include the lack of a standard format for distributing resources, thedifficulties of maintaining and updating materials over time, and the need for effective meansof tracking their use and impact.
In this paper, we present an integrated framework for overcoming these barriers andfostering the sustainable dissemination of L2 learning resources, with a focus on dialogue
scenarios. The framework is designed to provide a standard format for creating anddistributing resources, as well as a set of guidelines to ensure that they are pedagogicallysound and compatible with learning management systems (LMS). Then, we discuss thepotential benefits of the framework for L2 learners and educators and outlines future directions
for the project.

2. Proposed Framework
We identified several key requirements for building a desirable Integrated Language Learning
Support Framework (I2LSF) that puts educators at the heart of learning resources design fora more sustainable dissemination of L2 learning resources. We propose that such an
integrated framework should ideally be flexible, collaborative, easy to use, analytics-driven,
integrate with other tools, and inclusive. Additionally, it should have a sustainable plan, andfocus on accessibility to all learners.
Moreover, we also anticipate that such a framework could lead to the development of
cost-efficient intelligent tutoring systems (ITS) that have the ability to reach a large number ofstudents while still maintaining a high level of personalization, adaptivity, and interactivity. In
other terms, as ITS are scaled up and move beyond the standard three or four-component
model, there is a potential for adaptation to users to occur at various levels of granularity.
Previously, adaptation in tutoring systems was thought to consist of two main components:“macro-adaptivity” or the “outer loop”, and “step-based adaptivity” or the “inner loop” (VanLehn,
2006). However, the I2LSF framework as proposed here, and fulfilling the prerequisites
described above may also hold the potential for "meta-adaptivity" in which the system adapts
to the user by directing them to a different language learning activity that may suit better
learners’ needs. Additionally, research on "micro-adaptivity" has explored the benefits of using
learning data to fine-tune interactions at a level below the problem step, such as keystrokelevel inputs, emotion detection, and the presentation and timing of feedback (Graesser, 2011).
In the context of the I2LSF framework, we believe that micro-adaptivity may be achieved at
the interaction level with the presentation of specific conversational feedbacks.
In sum, from the above and inspired by Nye’s previous work on the potential ofintelligent support systems to improve education in resource-constrained countries (Nye,2015), we propose that a desirable learning support framework for creating and distributing
learning resources in the context of L2 learning should feature the following four connected

Figure 1. Outlines of our proposed I2LSF with the four levels of adaptation.

124

levels or loops of adaptivity: macro-adaptivity, step-based adaptivity, meta-adaptivity, andmicro-adaptivity. These four loops are likely to remain important decision points for designers
in the I2LSF, as they can be thought of as boundaries within the learning support system. In
addition, each of these types of adaptivity has its own advantages and can be used to provide
a more effective and efficient learning experience, by adjusting to the student's needs,
preferences, and progress. Such an I2LSF could also be helpful to teachers in teaching
language by providing them analytics on how the students are progressing and giving themsuggestions on how to improve their teaching. On the other hand, incorporation of AI based
techniques such as Adaptive learning, Self-generated data and Natural Language Processing
can make the system more effective and efficient in the way it assesses the learner's progress
and gives the feedback which can help the learner to identify his strong and weak areas.
In sum, from a theoretical point of view, if achieved, the I2LSF framework proposed in
the context of study hold the potential to allow educators to deliver learning resources that canbe adjusted and tailored to the individual needs and preferences of each student with user
adaptation being carrying out at four levels, as depicted in Figure 1.

3. Potential and Future Directions
While our project is still in the early stages of development, we believe it has the potential to
make a significant impact on the field of L2 learning and are excited to see how it evolves in
the future. In addition to providing access to a larger number of high-quality resources, theframework has the potential to support the professional development of scenario authors bygiving them a platform to share their work and receive feedback from their peers. Anotheradvantage of the framework is that it makes learning resources available from anywhere withan internet connection, making language learning more convenient and accessible. It could
also facilitate collaboration among authors and institutions, enabling the creation of more
complex and nuanced scenarios that better reflect the diversity of L2 learners.
To evaluate the effectiveness of the proposed framework, we plan to conduct a seriesof case studies with a diverse group of L2 learners and educators. These experimental
evaluations will examine the impact of the framework on learners’ language skills and attitudes
towards L2 learning, as well as the adoption and use of the platform by scenario authors.

References
Ayedoun, E., Hayashi, Y., & Seta, K. (2021). Authoring Tool for Semi-automatic Generation of TaskOriented Dialogue Scenarios. In Proceedings of the 29th International Conference on Computersin Education, 41-50.
Graesser, A. C. (2011). Learning, thinking, and emoting with discourse technologies. American
psychologist, 66(8), 746.
Johnson, W. & Valente, A. (2008). Tactical Language and Culture Training Systems: Using Artificial
Intelligence to Teach Foreign Languages and Cultures. Proceedings of the Twenty-Third AAAI
Conference on Artificial Intelligence, AAAI 2008, 1632-1639.
Kataoka, Y., Thamrin, A., Murai, J., & Kataoka, K. (2018). Effective use of learning management system
for large-scale Japanese language education. In Proceedings of the 10th International Conferenceon Education Technology and Computers (ICETC '18). Association for Computing Machinery, New
York, NY, USA, 49-56.
Nye, B.D. (2015) Intelligent Tutoring Systems by and for the Developing World: A Review of Trends
and Approaches for Educational Technology in a Global Context. International journal of artificial
intelligence in education, 25, 177-203.
VanLehn, K. (2006). The behavior of tutoring systems. International journal of artificial intelligence ineducation, 16(3), 227-265.
Watson T., R., & Campana, K. (2017). A review of learning management system use in language
education. Language Learning & Technology, 21(3), 84-104.
Willis, D. & Willis, J. (2007). Doing Task-Based Teaching; Oxford University Press: Oxford, UK; 56-63.
Zimmerman, K. (2014). A literature review of the challenges & best practices for English language
learners." In National forum of multicultural issues journal, 11(1), 1-7.

125

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Learning Support System to Understand
Others Through Dramatic Script Reading
and Its Evaluation
Hanano OKAMOTOa, Yuki HAYASHIb* & Kazuhisa SETAb
College of Sustainable System Science, Osaka Prefecture University, Japanb
Graduate School of Informatics, Osaka Metropolitan University, Japan*hayapy@omu.ac.jpaAbstract: To form and maintain smooth interpersonal relationships in various
situations in social life, the ability to appropriately understand the mental states ofothers, such as their intentions and feelings, is required. However, it is difficult to infer
the unobservable mental states of others, and it is also difficult to learn from others how
to perform inferring activities that are not externalized. In this study, we focus on“dramatic scripts” as a learning material that contributes to the training of the ability tounderstand others and propose a learning support system that aims to improve theability to understand others and to cultivate an attitude of understanding others throughinference activities about the emotions and conflicts of characters.
Keywords: Ability to understand others, dramatic scripts reading, learning support
system

1. Introduction
It is required to form and maintain smooth interpersonal relationships in various situations insocial life. For this purpose, it is necessary to appropriately understand the mental states ofothers, such as their intentions, beliefs, and feelings (Apperly, 2012; Epley & Caruso, 2012).
An increased understanding of the mental states of others will lead to consideration for others
and two-way communication, thereby promoting smooth interpersonal relationships. In
addition, an attitude that strives to understand others who have different backgrounds fromoneself is an important element forming the basis for understanding others (Shih et al., 2009;
Stephan & Finlay, 1999).
On the other hand, it is not always easy to infer the mental states of others that cannot
be explicitly captured because the intentions and emotions of others are not always expressedin everyday communication. When the actual feelings of others differ from their expressed
behaviors, it is necessary to infer their emotions based on contextual factors other than facial
expressions, such as the background situation in which they are placed. Since the mental
states behind such utterances and actions are complex and chaotic (Berrios, 2019), there isno unique model for inference activities, and it is also difficult to learn from others suchinference activities inside an individual.
In this study, we focus on the “dramatic scripts” as promising learning material for
cultivating the skill of understanding others. In dramatic script reading, it is required tounderstand the mental states of others from the viewpoint of characters with different
backgrounds from the reader by inferring the intentions and feelings of the characters basedon the clues in the chain of utterances (dialogues) made by the characters. Such activities
may contribute to the training of the inference activities.
Based on the above, this study sets the research question: “How can we realize alearning support system that improves the ability to recognize and infer the mental states ofothers?” More specifically, we are interested in the following two questions: “RQ1: Does
understanding the characters in dramatic scripts from multiple viewpoints contribute to theimprovement of the ability to understand others?” and “RQ2: Does broadening the learner’s126thoughts based on the inferences results of others contribute to the improvement ofunderstanding others?” In this paper, we propose a learning support system that aims to
improve the ability to understand others and to cultivate an appropriate attitude toward
understanding others through dramatic script reading.

2. Basic Philosophy of Fostering Ability to Understand Others
2.1 Difficulties in Understanding Others
In this study, the learning goal of the learners is to be able to appropriately recognize and infer
the mental states, such as intentions and emotions, from others’ utterances and actions andto be able to think based on the other’s point of view. In order for learners to learn these skills
and attitudes, the following learning difficulties need to be reduced:z
Difficulty 1: It is difficult to infer the mental states of others because they do not always
express their emotions as they are but may intentionally control their emotions or express
false emotions (Gross, 2002). It is also not easy to imagine the contextual factors that are
not always expressed in the behavioral process of others because they may choose
different behaviors from learners.
z
Difficulty 2: It is not easy to consciously train oneself to acquire an attitude of trying tounderstand the other’s point of view without immediately judging the beliefs and actionsof those who are different from oneself.
zDifficulty 3: It is difficult to learn desirable inferential activities because people’s mental
states are complex (Berrios, 2019), and no unique inferential process can be used as amodel. It is also difficult to learn from others how to perform the inference activities thattake place inside a person’s mind.

2.2 Dramatic Script Reading for Developing Ability to Understand Others
In this study, we focus on dramatic scripts as a learning material for developing the ability tounderstand others. The act of inferring the intentions and feelings of the characters in dramatic
scripts from the chain of their utterances (dialogues) and deciphering the conflicts between
them is an activity to understand the intentions, feelings, and thoughts of others from theviewpoint of the characters (others) who are placed in a different background from oneself.
According to Ishino, acting in theater is putting oneself in the other’s place and assuming theother’s identity, which shifts one’s point of view to the other’s (Ishino, 2015). Watanabe andKusumi also point out that actors have the experience of carefully inferring the beliefs, desires,and motives of the characters they play to perform them naturally (Watanabe & Kusumi, 2021).

2.3 System Design for Developing the Ability to Understand Others
As an approach to RQ1, we provide an inference activity support environment that allows
learners to externalize the inference results of the characters’ (others’) mental states indramatic scripts in mind map format as an opportunity to engage in inference activities about
the mental states of others. In addition, we provide “inquiries” that encourage in-depth
exploration of each viewpoint. The inquiries are intended to stimulate the learners’ curiosity
about the characters and further activate their critical reasoning.
The learner’s unique way of inference results will be externalized in the mind map.
Therefore, as an approach to RQ2, we develop a system that captures the differences on themap between different learners and provides information that stimulates each learner’s
inferential activities. This approach aims to promote learning from the inferences of others andto encourage more critical inference activities. More specifically, we aim to encourage new
insights from learners by providing highlight and focus functions for each viewpoint so thatlearners can compare their own and others’ ideas about each viewpoint on the map. In addition,
we implement an advice presentation function based on the differences between the two maps.

127

Advice is offered to encourage awareness of viewpoints and inquiries that the other explores
in depth but has not yet explored the learner’s own and how others think about the sameinquiries.

3. Learning Support System for Developing Ability to Understand Others
3.1 Environment to Support Inference Activities of Character’s Mental State
Based on the system design described in Section 2.3, we implemented a learning support
system for developing the ability to understand others as a web application. Figure 1 shows
the interface of the developed system. The learner can confirm the dramatic script in Figure1(a). Here, learners grasp the flow of the entire story. After reading through the story, learners
select one character they want to understand more deeply and explore the character in depth.
In Figure 1(b), viewpoints for deciphering the mental states of others (e.g., the identities
of characters) are displayed in a network structure. This network is a starting point to consider
the character’s mental state. The network is classified into two types: one for inferring thecharacter’s personality (Figure 1(i)) and the other for inferring the character’s utterances andactions (Figure 1(ii)). The network for personality inference indicates stable elements related
to the character’s personality development as viewpoints. The network for utterance andaction inference indicates elements related to utterances or actions of the character inadvance as viewpoints. By developing these networks, the learner systematically infers thecharacter.
The system displays inquiries that delve into the understanding of the character inFigure 1(c) by clicking on a viewpoint node. For example, when the learner clicks on the
viewpoint “unfulfilled desires,” adaptive inquiries (e.g., “What are the character’s unfulfilleddesires?” and “Whom does the character admire?”), that delve deeper into the viewpoints are
displayed. This is intended to activate critical reasoning for each point of view and to stimulate
the learner’s curiosity about the character. Figure 1(d) shows a choice (e.g., ‘causality’) toconsider the connection between what has been inferred under each viewpoint. This isintended to encourage reinterpreting a single interpretation from various perspectives throughactivities considering the connections between perspectives. In the same area, options arealso displayed for a more profound investigation of the character’s thoughts (e.g., ‘reasonsthat led to the inference,’ ‘interpretation,’ and ‘utterances that show the interpretation’). This isintended to promote the externalization of the learner’s inference activities process.
On the interface, clicking on any node and selecting an item from these options adds
a node for entering the answer below the node. The selected choice is displayed on the link,
and the node can be filled in with the learner’s inferences and interpretations from the drama
script. In addition, a link is created between the nodes by selecting any two nodes andFigure 1. Interface for Inference Activities of Character’s Mental State.

128

choosing one of the options (e.g., causality). The inquiry associated with a viewpoint isdisplayed in a blue link, while others are in gray.
The above functions are intended to externalize the results of inferences about themental states of characters (others) made inside the learner’s mind and the viewpoints that
led to such inferences and to task the learner with critically digging into them. Currently, we
set 21 viewpoints and related 42 inquiries for personality inference and 7 viewpoints andrelated 19 inquiries for deepening into the character from the viewpoints of the utterances andactions of characters based on the related research (Aikawa, 2015; Murayama, 1975) topersonality development, human behavior, etc.

3.2 Environment to Compare Inference Activities That Differ from Learners’ Own
By selecting the other learner’s map, the learner’s and the other’s maps are displayed, as
shown in Figure 2. By comparing their own and others’ maps, learners gain new viewpoints
on understanding others and engage in activities to broaden their thoughts.
In the interface, when a learner clicks on a viewpoint node on his/her map, a node of
the same viewpoint on another learner’s map is enlarged and highlighted in the center of thearea. In the example shown in Figure 2(e), the learner selects “desire (red frame).” In addition,
by selecting an utterance in the dramatic script from the area shown in Figure 1(a), the nodecorresponding to that line is enlarged and highlighted in the center of each map.
In addition to the visual support, the system is equipped with a function for presenting
feedback (advice) focusing on the differences between the two externalized maps, as shownin Figure 2(g). The following three types of advice are generated and presented based on atemplate as clues to expand their thoughts by relativizing the different results of others’
reasoning activities:z
Advice that captures differences in viewpoint: The advice that captures the difference
between the viewpoint nodes used by other learners and not by the learner. e.g., “The X(other) is deepening his/her thoughts about the character “Elizabeth’s experience”(viewpoint), but you have not yet done it. Think about the viewpoint in your way”.
z
Advice that captures differences in inquiries: The advice that captures the differences
in inquiries used by other learners and not by the learner. e.g., The X (other) has been
deepening his thoughts about the inquiry: “What are the character’s unfulfilled desires?”but you have not yet done it. Think about the inquiry in your way.
z
Advice that captures the same inquiries: The advice that captures inquiries and theirinterpretations used by both learner and the other. e.g., The X (other) gave aninterpretation as “disappointment in myself” to the inquiry: “What emotions does thecharacter have?” Do you have any new thoughts about this?

Figure 2. Interface for Inference Activities of Character’s Mental State.

129

4. Evaluation
4.1 Experimental Setting and Procedure
An evaluation experiment was conducted to verify the usefulness of the developed system. In
this evaluation, we report on the effects of the developed system on dramatic script reading
toward understanding others (i.e., RQ1 and RQ2), as well as on the effects of the system ontheatrical activities. Four adult members of a theater troupe and one acting instructor
participated in this evaluation. The dramatic script used in the experiment was “Witch Village
Forest,” written by Miki Kimito. The script is 20 pages long (about 600 lines), and the screening
time is about one hour.
First, the troupe members were asked to read through the script to understand thecontents before using the system. No intervention, such as time control, was used here. Eachmember was cast with the expectation that they would play one or two characters. In order topractice map comparison activities and acting instructions after using the system, oneutterance of a particular scene was designated as the target of inference activity. Then, themembers were asked to conduct the inference activity (Section 3.1) of the mental state of twoor three characters’ personalities (the character played by each member and the character ofthe dialogue partner in the dramatic script) using the system until they were satisfied with theirperformance. After finishing the inference activity, they were asked to conduct a comparison
activity (Section 3.2) with maps created by different troupe members. The maps to becompared were the inferences of the same utterances of the same character, and they wereasked to reflect on their inference results while using support functions. If any new findings
emerged, they were reflected in the map.
After completing the map, the troupe members practiced acting with an acting
instructor. Each member was asked to perform a part of a pre-designated scene, and the
instructor was asked to guide the performance while checking the map created by themembers. Finally, a questionnaire survey on a 5-point scale (Table 1) and an interview-style
discussion were conducted. Here, we asked each participant to share his or her opinions oneach question and asked them to share their various insights about the system.

4.2 Results
The results of the questionnaire (N=4) are shown in Table 1. Questions (A) to (D) were asked
to evaluate the support function of RQ1, which aims to capture characters from multiple
perspectives, and questions (E), (F), and (G) were asked to evaluate the support function of
RQ2, which aims to learn from the insights of others and to broaden one’s thoughts. The
results showed that all questions were generally evaluated positively, confirming that theenvironment for inference activities of the character’s mental state and the presentation of
viewpoints and inquiries helped encourage the members to dig deeper into their charactersand to perceive others from various angles.
Table 1. Questionnaire Survey
(A)
(B)
(C)
(D)
(E)
(F)
(G)

Question
Did “viewpoints” for personality and utterance/action inferences help you better understand thecharacter from various angles?
Did “questions” for each viewpoint make the activities to understand the character more active?
Did the support function in the mind map interface help you to organize and infer the character’smental state?
Did a series of inferential activities help you better understand the character’s mental state?
Did the highlighting function help you compare your thoughts with others?
Did the advice provided by the system lead you to interpretations or insights that you had notthought of on your own?
Did a series of activities comparing thoughts with others broaden your thoughts?

130

Average
4.75
4.25
5.00
4.25
4.25
4.00
4.75

In an interview-style discussion, in response to the question to the instructor, “How has
the use of the maps created by the troupe members affected your instruction?” the instructor
commented that “Although I found it somewhat difficult to use in the field, it was easy to advise
because I knew the actor’s thoughts in advance.” In the discussion, many members statedthat using the system helped them expand their thoughts, such as “Comparing inference
results of myself with the other made my thoughts more flexible” and “It became easier torevise my thoughts.” It was also commented that the system was effective in motivating themembers to dig deeper into their characters, such as “I wanted to think about what others
have inferred” and “Since others were able to infer more viewpoints than I could, I thought I
would consider them more myself.”
In the discussion, many members stated that using the system helped them expand
their thoughts, such as “Comparing inference results of myself with the other made mythoughts more flexible” and “It became easier to revise my thoughts.” It was also commented
that the system was effective in motivating the members to dig deeper into their characters,such as “I wanted to think about what others have inferred.”

5. Concluded Remarks
In this study, we developed a learning support system that aims to improve the ability tounderstand others and to cultivate an appropriate attitude toward understanding others byusing dramatic scripts as learning materials. Experimental results show that the system has
the potential to facilitate inference activities that contributed to understanding others, as wellas activities to infer the mental states of others from multiple viewpoints.
Future issues include improving the system’s usability and confirming its usefulness
through long-term practice.

Acknowledgments
We sincerely thank the Osaka LaLaSong Musical Theater Company and its members forcooperating in this study.

References
Aikawa, T. (2015). A Cognitive Study of Understanding Others’ Intentions. Human and socioenvironmental studies, 29, 35–48.
Apperly, I. A. (2012). What is “theory of mind”? Concepts, cognitive processes and individual
differences. Quarterly Journal of Experimental Psychology, 65(5), 825–839.
Berrios, R. (2019). What is complex/emotional about emotional complexity?. Frontiers in psychology,
10, 1606.
Epley, N., & Caruso, E. M. (2012). Perspective taking: Misstepping into others’ shoes. In Handbook ofimagination and mental simulation (pp. 295-309). Psychology Press.
Gross, J. J. (2002). Emotion regulation: Affective, cognitive, and social consequences.
Psychophysiology, 39(3), 281–291.
Ishino, Y. (2015). The Effect of a Relative Analysis of Oneself and Understanding Another Person by
Impersonation: The Development of a Method for Recreating Various Scenes in Real Life
Situations through Problem-Finding Fieldwork. Journal of lifology, 27, 17–28.
Murayama, S. (1975). Six dimensions of life styles in college students. Bulletin of the Faculty ofSociology, Kansai University, 7(1), 161–174.
Shih, M., Wang, E., Trahan Bucher, A., & Stotzer, R. (2009). Perspective taking: Reducing prejudice
towards general outgroups and specific individuals. Group Processes & Intergroup Relations,12(5), 565–577.
Stephan, W. G., & Finlay, K. (1999). The role of empathy in improving intergroup relations. Journal ofSocial issues, 55(4), 729–743.
Watanabe, T., & Kusumi, T. (2021). Experimental investigation of the effects of a theatrical activity onsocial abilities. Cognitive Studies, 28, 122–138.

131

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Development of Estimation Method for
Learner’s Emotional Concealment During
Learning Using Biometric Information andFeedback Model
Koichi SHINOHARAa*, Keiichi MURAMATSUb & Tatsunori MATSUIca
Guraduate School of Human Sciences, Waseda University, Japanb
Global Education Center, Waseda University, Japanc
Faculty of Human Sciences, Waseda University, Japan*koichis9911@toki.waseda.jp
Abstract: Although it is important to understand learners’ emotional situations inteaching and learning processes, learners sometimes do not express negative
emotions intentionally. This study has termed this intentional non-expression ofnegative emotions “emotional concealment,” and examined its estimation fromlearners’ biometric information. In particular, we focused on learners’ facial features,
and attempted to clarify their relationship with emotional concealment. Using theseresults, we attempted to quantify emotional concealment from the time-series data of
biometric information and detect it as an outlier. Additionally, we attempted to develop
a feedback generation system to inform teachers about emotional concealment.
Keywords: Emotional concealment, learning, estimation, feedback, micro-expression

1. Introduction
Inability to understand the content of another person’s speech is common in communication.
However, continuing a conversation without resolving this situation will result in deteriorationin its quality. Since the teaching-learning process can be considered communication
between a teacher and learner, the same can happen during learning. In an asymmetrical
relationship between the teacher and learner, the learner often does not consciously express
negative feelings, such as not understanding (Shinto, 2016). The persistence of this state ofmind ultimately leads to a decline in learning quality. This study terms this act of
unconscious non-expression of negative emotions as “emotional concealment” (Gross,2015). Thus, emotional concealment is defined as consciously not expressing one's negative
emotions even though one is aware of them. We hypothesized that emotional concealment
could be estimated using biometric information. Furthermore, we attempted to quantify
emotional concealment based on biometric information and detected it as an outlier. We
created a feedback model for the teacher based on the results. It is expected that theteacher will be able to understand the learner’s mental state more accurately by estimating
the emotions that the learner has concealed. As a result, higher-quality learning will be
realized.

2. Overview of the Study
This study aimed to develop a method for the quantification and estimation of emotional
concealment using biometric information, and generate a feedback model. In Study 1, we
conducted an experiment to determine whether emotional concealment occurs during132

learning, and if so, what kind of biometric information it relates to. The results suggest thatemotional concealment occurs during learning, and that respiration and pulse wave values
are higher during emotional concealment. Additionally, facial expressions of the learners
differed between emotional concealment and normal conditions. In Study 2, we conducted
an experiment to investigate the differences in facial expressions during learning between
emotionally concealed and normal conditions. We analyzed the intensity of facial featurepoints, called Action Units. Results showed that the number of Action Units around themouth was higher during emotional concealment. In Study 3, we analyzed the time-series
data of the Action Units obtained in Study 2 using an outlier detection method, assuming a
relationship between the time interval showing the outlier and emotion concealment. We
were able to detect anomalous values, but were unable to derive a significant relationshipwith the emotional concealment time interval reported in the learners’ reflection reports. The
detailed methods and results of Study 4 are described below.

3. Analyzing the Feature Points of Learners’ Faces in Emotional Concealment
and Normal Situations during Learning
From the discussion in Studies 1 to 3 (Shinohara, Muramatsu & Matsui, 2021), it was notpossible to derive a significant relationship between the time interval showing abnormal
values and emotional concealment in the Action Unit time-series data. Therefore, in Study 4,
we conducted an additional experiment in which we asked the learners to watch on-demandcontent, and captured their facial expressions. Furthermore, we reanalyzed 67 facial
landmark outputs from Open Face. The experiment was conducted with 26 Waseda
University students (19 men and 7 women). As specific instructions for the experiment,
learners were given the definition of emotional concealment, were told that they would be
photographed during the experiment, and that they would attend the class twice, once toconceal their emotions when negative emotions occurred and the second time as usual. We
obtained biometric data from learners who were watching 25-minute on-demand content on
knowledge representation. We asked the learners to report whether they experienced
emotional concealment during learning, and if so, at what time of day, while watching thevideo recordings after the experiment. Learners were asked to report their reflections oneach slide of the experimental stimulus. The results showed a difference between emotional
concealment and normal conditions around the mouths of all the learners (Figures 1, 2, and3). They further suggested that the characteristic values around the mouth in the emotional
concealment interval repeatedly rose and fell, as reported in all learners’ reflection reports.
However, for feature points other than the lower lip, some of them, such as those that riseafter falling during the emotion concealment interval, do not appear in the emotion
concealment interval. Figure 4 shows a graph of the characteristic values of the upper lip. It
suggests that the upper lip rises and falls not only during emotional concealment, but also
during normal conditions, with no common tendency among many learners. We
hypothesized a relationship between emotional concealment and micro-expressions aroundthe mouth (Ekman & Friesen, 1987).

Figure 1. Characteristic score graph of the lower lip of learner A.

133

Figure 2. Characteristic score graph of the lower lip of learner B.

Figure 3. Characteristic score graph of the lower lip of learner C.

Figure 4. Characteristic score graph of the upper lip of learner A.

4. Summary and Future Tasks
This study aimed to estimate the emotional concealment state using the learner’s biometric
information, and generate feedback using the results. The results of Study 4 suggest thatemotional concealment is related to the learners’ facial expressions. The feature points
around the mouth moved in detail when the learner was emotionally concealed. Futurestudies should develop a method for quantifying and estimating learners’ emotional
concealment, and create a feedback model based on the developed estimation method.
Using the developed feedback model, it is necessary to verify whether learning effectimproves in actual learning situations.

References
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L. P. (2018). Openface 2.0: Facial behavior analysis
toolkit. 13th IEEE international conference on automatic face & gesture recognition, 59-66.
Ekman, P., & Friesen, W. V. (2003). Unmasking the face. Malor Books.
Gross, J. J. (2015). Handbook of Emotional Regulation. Guilford Press.
Horiguchi, Y., Matsui, T., & Kojima, K. (2008). Detection of student’s subconscious turns on studyingin e-learning system. The 22nd Annual Conference of the Japanese Society for Artificial
Intelligence. (in Japanese)
Kayatani, Y., Otani, M., Chu, C., Nakashima, Y., & Takemura, H. (2019). Predicting Laughters in
Comedy Drama with Subtitles and Facial Expression. The 33rd Annual Conference of theJapanese Society for Artificial Intelligence. (in Japanese)
Shinohara, K., Muramatsu, K., & Matsui, T. (2021). Development of estimation method for Learner's
emotional concealment during learning Using Biometric Information. Japanese Society for
Information and Systems in Education 2021 Student Research Presentation, 53-54. (inJapanese)
Shinto, T. (2016). The Meaning of Negative Emotions in Learning Community: The Concept of
Cognitive Apprenticeship. Ritsumeikan teacher educational studies, 91-100. (in Japanese)134

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Development of a Learning Support
System for playing Ryuteki in Gagaku for
BeginnersaYasushi UENOa*, Masato SOGAb**
Graduate School of Systems Engineering, Wakayama University, Japanb
Faculty of Systems Engineering, Wakayama University, Japan*s236034@wakayama-u.ac.jp, **sogamasato@gmail.com

Abstract: Ryuteki is one of the wind instruments in gagaku, a traditional music of Japan
since ancient times. This instrument is difficult for beginners to acquire playing skills.
The best way to learn how to play ryuteki is to learn from an expert, but the opportunity
is limited due to time or cost constraints in gagaku. Therefore, in this study, wedeveloped a system that automatically judges whether the pitch and timing of the soundare correct when a beginner plays ryuteki using sound processing technology, andpresents feedback on the results.
Keywords: Learning support, Ryuteki, Gagaku, Skill, Sound processing

1. Introduction
Gagaku is a traditional Japanese music that is also called “the world’s oldest orchestra”.
It was introduced from Asian countries such as China and Korea around the 5th century, andit fused with Japan’s ancient songs and dances. It was completed as a unique musical formof Japan in the middle of the Heian period, 9th century, and it has been continuing.
One of the wind instruments used in gagaku is “Ryuteki”, which is the subject of thisresearch. It is a transverse flute made of bamboo, and it has a similar structure to the flute inWestern music (Left in Figure 1). Because it has a wide range of tones among the instruments
used in gagaku, it has a role of decorating the main melody played by Hichiriki (oboe in
Gagaku), as well as a role of playing “Ondo”, which is a solo performance by a Ryuteki player
at the beginning of a piece.
There are various skills required to play Ryuteki, and some of them are common withthe skills for playing flute, but there are many techniques and practice methods that are uniqueto gagaku, so it is difficult to learn gagaku performance skills even for those who are familiar
with music. Especially, the differences from Western music can be seen in the notation of
Ryuteki and “shoka”, which is one of the practice methods of gagaku. The notation of Ryuteki
is very different from the staff notation that is often used in Western music. It has shoka written
in katakana vertically in the center, and fingering written in kanji vertically on the left side (Rightin Figure 1).
Shoka is a practice method of singing the melody of a piece out loud while tapping thebeat with hands and knees, without using a score. By doing this, one can grasp the flow of thepiece and learn how to breathe and play in that piece. Gagaku pieces are difficult tounderstand the flow by just looking at the score, and to be able to play them, one has to not
only read the score, but also understand the flow and playing style that are not written in thescore by singing shoka.
One can only practice playing with an instrument after being able to sing shoka, so there
are many things that are necessary to be able to play a piece, and it can be said that it isdifficult for beginners to learn how to play.
In order to learn the music of gagaku, one must grasp the flow of the music using singing
and other methods, and therefore it is essential to practice under the guidance of an expert.
However, currently, beginners have limited opportunities to receive guidance from instructors,

135and even if they can receive guidance once, there is often a long period of time before they
can receive guidance again. Even if they practice on their own during that time, it is difficult for
beginners to judge whether they are playing well or not due to the characteristics of gagaku
music, and therefore it is difficult for them to practice satisfactorily and efficiently improve theirperformance skills.

Figure 1. Ryuteki.and score (Ryuteki (left) and score for Ryuteki (right))

2. Research objective
It is very difficult for beginners to practice on their own and acquire gagaku performanceskills. Therefore, in this research, the learners are beginners who have acquired someknowledge about Ryuteki (a type of flute used in gagaku) and the music they want to practice,
however, have difficulty performing the music with Ryuteki. The purpose of this research is todesign and develop a learning support system that can help beginners judge whether they areplaying correctly or not and acquire performance skills even when they practice on their own
without the help of a teacher.
Specifically, we developed a system that uses audio processing technology to judge
whether the pitch and timing of the performance match the musical score, and provide
feedback on the results. The goal of this system is to enable beginners to judge whether they
are playing “Hyojo Etenraku”, a piece that beginners often encounter first as an example ofgagaku music, correctly or not on their own, and to improve their own performance by using
this system.

3. Related work
As previous research on learning support systems for musical instrument performance,
Takekawa et al. developed a system that uses piano fingering recognition technology to checkthe performer’s fingering in real time, and provides intuitive information on fingering accuracy,
performance accuracy, fingering information, and keystroke information. They are necessary
for performance, to enhance keystroke position and fingering proficiency in the initial stage ofperformance (Yoshinari Takegawa et al. 2011).
Kikukawa et al. developed a learning support system that can measure the position andangle of the bow during the performance of bowed string instruments such as erhu using amagnetic position sensor, diagnose the movement of the bow in real time, and enable
beginners to acquire performance skills for bowed string instruments (Fumitaka Kikukawa,
Masato Soga et al. 2014).
These studies target instruments such as piano and erhu that are played by hand or arm
movements, but Ryuteki is a wind instrument that produces sound by blowing air into it
internally, so it is not possible to evaluate whether it is producing correct sound or not bymeasuring body movements as in previous studies. Therefore, in this study, we decided to136use audio processing technology as a means of evaluating performance for wind instruments
by directly evaluating the sound produced.

4. Prototype system
This system is designed for learners who have learned about the music from an
instructor and want to practice on their own, so it is desirable that learners have some priorknowledge of Ryuteki performance and how to read the musical score of “Hyojo Etenraku”.
We will explain how to use the system as follows.

4.1 Selecting the score to play

Figure 2. Score selection screen before performance.

When a learner starts the system, a screen like Figure 2 will appear. On the left side of
the screen, part of the musical score of “Hyojo Etenraku” is displayed, and by clicking on “Nextpart of musical score” at the bottom of the screen, the learner can select the score he/shewants to evaluate by displaying it. The frequency in the upper right shows the frequency thatis currently used as the standard for judging the sound, and the learner can change thestandard frequency by 1Hz in the range of 400Hz to 460Hz by clicking on the “+” and “-”buttons on the left and right. The color indicates whether the sound is not sounding (gray), intune (green), slightly high (red), or slightly low (blue) within that scale, so the learner can check
his/her own sound and tune it before actually evaluating his/her performance.

4.2 Playing the score
To practice playing a score the learner wants to play, he/she selects the score andpresses “Start” to switch to the mode where he/she can play and evaluate the score. In thismode, the numbers on the left side of the screen change color from gray to white in order from
top to bottom. These numbers indicate the beats of the score, and the first 3/4 are set up to
prepare for playing before entering the performance, and the learner is supposed to prepare
for playing during these two beats and start playing according to the score from the next beat.
The gagaku score has a length of two beats per large character of katakana singing (thepart that says “Toraro…” in this score), and the learner is supposed to play along with thenumbers that light up in order at regular intervals. Figure 3 shows the screen display duringperformance, and in this figure, it is playing the first 4th beat after starting the performance.
Also, if the learner feels that his/her performance is not going well, he/she can stop playing bypressing the stop button at the bottom right of the screen, go back to the scene where he/she
selects the original score, and press start again to start playing from scratch.

137

While the learner is playing, the system internally processes information about the soundthat is currently being played and stores it in an array. This array consists of three types ofarrays: an array that stores information on the scale (array 1), an array that stores thefrequency deviation between the correct sound and the sound being played for that scale(array 2), and an array that stores how many seconds after pressing the start button this soundwas played (array 3), and information on sound is stored in an array.

Figure 3. Screen during performance

4.3 Evaluation of performance
When the learner finishes playing until the last beat, the system compares the threearrays recorded during the performance with the score data and judges whether the scale
matches for each beat on the score and whether there is any pitch deviation, and presents theresults as feedback. Figure 4 shows an example of the screen display when giving feedbackon the evaluation of performance. In this figure, it can be seen that the correct sound is
produced during the first three beats, but the fourth beat is slightly high, and from there, either
the sound is wrong for three beats, or the timing of switching or starting or ending the soundis off.

Figure 4. Screen for displaying results after performance

After checking the evaluation, if the learner is not satisfied with his/her performance,
he/she can press the “Repeat” button to redo the performance of the same score, or if he/shefeels that he/she has practiced enough, he/she can press the “Next part of musical score”
button to practice the continuation of the score.
In this way, by checking from the evaluation of performance, he/she can find out whichparts are easy to make mistakes or deviate in sound, and find out what he/she needs toimprove in his/her performance, and by repeating practice, he/she can deepen his/her
understanding of the music and practice to be able to play well. This is a system that allows
him/her to practice to be able to play well.

138

5. Evaluation Experiment
The purpose of the experiment using this system was to investigate, using aquestionnaire, whether beginners, who are the target of this study, can expect to improve theirperformance of “Hyojo Etenraku” by practicing with the proposed method, and whether theevaluation of Gagaku performance by the proposed method is useful for learning Gagaku. In
addition, we also asked for answers on improvement points and functions that they want us to
add to this system for future research.
The subjects who cooperated in the experiment were four people (two men and two
women) who had experience playing Ryuteki. Of the four subjects, two were beginners, one
was an expert who had performed in concerts and other events, and the remaining one wasan instructor who taught Ryuteki.
In this experiment, we explained how to use the system and that it was a system built
assuming that beginners would practice by themselves. We asked them to practice playing forabout 10 minutes using the system. After that, we asked them to answer a questionnaire onwhether they expected to improve their Ryuteki performance by using this system

5.1 Evaluation of usefulness of the system
To evaluate the performance and functionality of the system, we conducted aquestionnaire with four questions using a five-point scale. The questions and the results areshown in Table 1. The results are indicated by average points of five-point scale.
Table 1. Questions and results on usefulness of the system

No. Question sentence
(1)
(2)
(3)
(4)

I understood which parts I played well through the system.
I understood which parts I was weak at through the system.
I felt that I could expect to improve my performance by using thesystem.
I felt that the method of evaluating by pitch in this study was usefulfor learning Gagaku.

Average point
3.75
4.25
4.25
4.25

5.2 Free comments
In parallel with the multiple-choice questions, we asked for opinions on improvement
points and other things that they wanted us to incorporate in a free-writing format. A summaryof the answers is as follows.
x
x
x
x
x
x
x

I want them to put in a sound at the timing when the beat changes.
I want a little more time between pressing start and starting to play.
I felt that the judgment of pitch and tempo was too strict.
I want them to evaluate my performance not only by the color of the numbers but also
by words.
It would be interesting to show a message by the system with a chat bot.
I don't know if I can match with others in an ensemble, so I felt it would be nice if what I
played was recorded and I could play it back and get feedback later.
I felt that if I eliminated the Gagaku-like quality from my performance, I could get ahigher score.

5.3 Discussion
5.3.1 Discussion on the usefulness of the system
From the results of the questionnaire from question number (1) to (3), it was found thatby using this system, they were able to evaluate their performance, which they could not judge

139when they practiced alone before, and that they expected to improve their performance. Also,
since we obtained a high evaluation for question number (4), we recognize that we haveobtained agreement for the method of evaluating Ryuteki performance based on pitch andtiming.
5.3.2 improvements of the system
In the free comments, some users expressed their wish to have a sound cue at thetiming when the beat changes, and to have a little more time between pressing the start buttonand starting the performance, as they felt they could not perform well in the scenario of playing.
In actual gagaku ensemble, there is no conductor who keeps the beat, so players have tocatch the flow of the beat by themselves, and in this system, we did not play a sound at thebeat timing to create a similar situation. However, considering the opinions of the experimental
results, and also considering that this system was designed with the purpose of enabling selfpractice, we realized that we need to design it with more user-friendliness and playability.

6. Conclusion
In this study, we developed a prototype system of learning support for playing Ryuteki
for self-learning. We evaluated the system, and we got the results that the system can support
for learning Ryuteki. However, we also got opinions which tell some points to be improved.
We also think the number of participants of the evaluation is only a few. We would like toincrease the number of participants in future.

Acknowledgement
This work was supported by JSPS KAKENHI Grant Number JP 17H01996.

References
Gagaku Ryutekihu (Score for Ryuteki), Tenrikyodoyusha, 2018.
Yoshinari Takegawa, Masahiko Tsukamoto, Tsutomu Terada:
Design and Implementation of a Piano Practice Support System using a Real-Time Fingering
Recognition Technique., The 2011 International Computer Music Conference (ICMC 2011), 387394
Fumitaka Kikukawa, Masato Soga, Hirokazu Taki:
Development of a Gesture Learning Environment for Novices' Erhu Bow Strokes. 18th International
Conference on Knowledge-Based and Intelligent Information & Engineering Systems (KES2014): 1323-1332
Marimo Kumaki, Yoshinari Takegawa, Keiji Hirata
Design and Implementation of a Positioning Learning Support System for Violin Beginners, Using
True, Vague and False Information, Journal of Information Processing, vol.26, pp.285-293, 2018https://doi.org/10.2197/ipsjjip.26.285
Jürgen Konczak,Heidi vander Velden &Lukas Jaeger
Learning to Play the Violin: Motor Control by Freezing, Not Freeing Degrees of Freedom
Journal of Motor Behavior, Volume 41, 2009 , pp. 243-252,
https://doi.org/10.3200/JMBR.41.3.243-252
Yoshitaka Kashiwagi, Youji Ochi
A Study of Left Fingering Detection Using CNN for Guitar Learning, 2018 International Conference onIntelligent Autonomous Systems (ICoIAS2018), IEEE
DOI: 10.1109/ICoIAS.2018.8493656

140

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

An Adaptive Learning Support System
based on Ontology of Multiple
Programming Languages
Lalita NA NONGKHAIa*, Jingyun WANGb & Takahiko MENDORIc
Graduate School of Engineering, Kochi University of Technology, Japanb
Department of Computer Science, Durham University, UKcGraduate School of Engineering, Kochi University of Technology, Japan* 256010u@gs.kochi-tech.ac.jpaAbstract: This research proposes to develop an adaptive ontology-based learning
support system for computer programming learning. Firstly, the system adopts apreviously developed ontology called CONTINUOUS, which represents programming
concepts and their relation in a graph and makes use of its content to serve as hints
within programming questions. Secondly, we design an adaptive strategy forrecommending suitable exercises to learners, which uses CONTINUOUS as metadata
of exercises and the Elo rating system to estimate learners' skills. This work aims todesign a system to provide personalized exercise path for the learner and evaluate its
effectiveness.
Keywords: Adaptive Learning Support, Ontology, Recommendation System,
personalized learning, programming learning

1. Introduction
Computer programming utilizes coding, debugging, and evaluating of solutions (Friday et al.,2019). However, the complexity of programming problems can confuse learners regarding thenecessary knowledge for the solution (Yorah et al., 2016). To improve the efficiency of learning
CT (Computational Thinking), this research focuses on developing a web-based Learning
Support System (LSS) based on ontology to recommend suitable programming exercises forlearners. There are several educational systems leverage ontology. For instances, Ibrahim etal. (2017) utilized three ontologies to suggest suitable course for university students, andBouihi et al. (2019) proposed an approach to apply their ontologies in recommendation system
to support learners within their learning experiences by recommending appropriate materials.
In our previous work (Lalita et al., 2022), we developed an ontology named
CONTINUOUS, encompassing common concepts from Python, Java, and C#, describing
programming concepts, sub-concepts, and concept relation. This research adapts
CONTINUOUS to visualize programming concepts as a graph to help learners understand therelated concepts in a specific programming language. Furthermore, the design principle of
CONTINUOUS determines knowledge points (KP) as minimal syntax of a specificprogramming language and enables KP to serve as a hint of each programming question to
guide learners in solving a specific problem.
Moreover, this research also focused on designing a personalized recommendation
approach to enhance learner programming skills. Previously, Zheng et al. (2022) have deviseda knowledge structure tree for C programming, leveraging it as meta-data in theirrecommendation system. They used the matching degree calculation to match score between
the student’s cognitive level and exercise difficulty to facilitate personalized exercise
recommendations. Similarly, Michlík et al. (2010) have employed linear extrapolation toestimate potential knowledge levels in exercise recommendations, with the goal of aiding
learners in achieving favorable exam results within a limited time. To enhance the assessmentof learner’s understanding level, we adapt the Elo Rating System (ERS) (Elo, 1978), which is141

a method used to assess and measure the relative proficiency of the participants incompetitive games and has also been utilized in education research to estimate learner’s skill
and item difficulty (Radek, 2016). Additionally, Klinkenberg et al. (2011) demonstrated theefficacy of ERS in assessing player skill and item ratings within the Math Garden, amathematics game for primary students.
In summary, the contribution of this research is the adoption of CONTINUOUS in an LSS
and the design of an adaptive strategy for exercise recommendations to explore whether thisLSS can effectively enhance learners’ programming skills.

2. Development of an LSS Based on Ontology in Computer Programming
2.1 The adoption of CONTINUOUS with the system
Our system utilizes CONTINUOUS to visualize programming concepts as a graph to guidelearners with a clear understanding of the relation between concepts. For instance, as shownin Figure 1, if learners decide to learn the “data_structure” concept in Python, they have theoption to practice with “List” or “Dictionary.” Opting to study the “List” concept necessitates
understanding of the associated “list_method”, which serves as a function for manipulating
elements within the “List” concept.

Figure 1. A Python-based visualization of programming concepts from CONTINUOUS.
In addition, the contents in CONTINUOUS are also used as a hint in programmingquestions, depending on the code answer in each question. Currently, our system contains
questions from a question bank, which has 68 questions in three levels of difficulty (easy,
standard, and difficult) involving concepts in CONTINUOUS. For example, when learners aretasked with verifying the equivalence of input numbers, which involves the concepts of“conditionals” for decision-making and “standard_input” for keyboard input, the system willassist learners in grasping essential concepts (“conditionals” and “standard_input”) for
addressing the question.

2.2 The design of adaptive strategy to recommend suitable programming exercises
To recommend suitable programming exercises for learners, the system will update alearner’s skill based on the formulation of ERS presented by Radek (2016) in an educationalsetting as follows:ߠ௦ ‫׷‬ൌ  ߠ௦ ൅ ‫ ܭ‬ή ൫ܿ‫ݐܿ݁ݎݎ݋‬௦௜ െ ܲሺܿ‫ݐܿ݁ݎݎ݋‬௦௜ ൌ ͳሻ൯ሺͳሻ

142

From formulation 1, we denote ߠ௦ as learner’s skill of a learner ‫ݏ‬, the value of ‫ ܭ‬as alearning rate, which is a weight value in the formulation, ܿ‫ݐܿ݁ݎݎ݋‬௦௜ as the correctness of codeanswer of a learner ‫ ݏ‬on an exercise ݅ and ܲሺܿ‫ݐܿ݁ݎݎ݋‬௦௜ ൌ ͳሻ as the probability of a correct
answers as shown in formulation 2:ܲ ሺܿ‫ݐܿ݁ݎݎ݋‬௦௜ ൌ ͳሻ ൌ ͳሺͳ ൅ ݁ ିሺఏೞ ିௗ೔ ሻ ሻሺʹሻThe initial value of ߠ௦ is set to 0. After code submission, the system evaluates it with thetestcases. If all testcases are passed, ܿ‫ݐܿ݁ݎݎ݋‬௦௜ is set to 1; otherwise, it is set to 0. Learner’s
skill is estimated in the range of 0 to 1 and compared within three difficulty levels of exercises.
If the result is nearly 0 indicates a slight increase of maintain the same, while the result close
to 1 which means learner’s skill increase, leading to the system providing the upper level. If
learners reach the highest level within the current concept, the system will show the practiceprocess of one of its related concepts to learners.

3. Conclusion and Future Work
This research focuses on developing an LSS, which adopts CONTINUOUS to visualize
programming concepts as a graph, enabling learners to understand the relation between
concepts within a specific language. Additionally, the system makes CONTINUOUS serve as
exercise metadata and uses the programming concepts of CONTINUOUS as hints to support
learners during problem-solving. Furthermore, this research designs an adaptive strategy forexercise recommendations, utilizing the ERS formulation to estimate and update learners’
skills. In the future, we plan to conduct several experiments to evaluate the system and answer
the research questions: whether our system which provides personalized exercise
recommendation based on individual understanding levels, can effectively enhance learners’
programming skills. Learning log data will also be collected and analyzed.

References
Bouihi, B., & Bahaj, M. (2019). Ontology and Rule-Based Recommender System for E-learning
Applications. International Journal of Emerging Technologies in Learning 14(15), 4–13.
Elo, A. (1978). The rating of Chessplayers, Past and present. New York: Arco Publishers.
Friday, J. A., Solomon, S. O., Jarkko, S., & Sunday, A. (2019). A Systematic Review of Computational
Thinking Approach for Programming Education in Higher Education Institutions. Proceedings of
19th Koli Calling International Conference on Computing Education Research on Proceedings(pp. 1-10). Association for Computing Machinery.
Ibrahim, M.E., Yang, Y., & Ndzi, D. (2017). Using Ontology for Personalised Course Recommendation
Applications. International Conference on Computational Science and Its Applications. Springer,
Cham.
Klinkenberg, S., Straatemeier, M., & van der Mass H.L.J. (2011). Computer adaptive practice ofMaths ability using a new item response model for on the fly ability and difficulty estimation.
Computers & Education 57(2011). 1813-1824.
Lalita, N., Jingyun, W., & Mendori, T. (2022). Developing an Ontology Of Multiple Programming
Languages From The Perspective Of Computational Thinking Education. Proceedings of 19thInternational Conference on Cognition and Exploratory Learning in Digital Age (pp. 66-72).
International Association for Development of the Information Society.
Michlík, P., &amp; Bieliková, M. (2010). Exercises recommending for limited time learning. Procedia
Computer Science 1(2), 2821-2828.
Radek, P. (2016). Applications of the Elo rating system in adaptive educational systems. Computers &Education 98(2016). 169-179.
Yorah, B., & Macro, A. G. (2016). Why is programming so difficult to learn? Patterns of Difficulties
Related to Programming Learning Mid-Stage. ACM SIGSOFT Software Engineering Notes,
41(6), 1-6.
Zheng, W., Du, Q., Fan, Y., Tan, L., Xia, C., &amp; Yang, F. (2022). A personalized programming
exercise recommendation algorithm based on Knowledge Structure Tree. Journal of Intelligent &Fuzzy Systems 42(3), 2169-2180.

143

" !   !" " "   #" !
#" !"$  #" ! #"                            
        

   6 ;12: :;<-@ >. *25 ;7 9.*42A. .*96260 +@ #.*,1260 26 >12,1 * 8.9:76
4.*96:+@;.*,1260*97+7;26*4.*96260:2;<*;276+.;>..6;1.97+7;*6-;1.8.9:766
;12: 8*8.9 >. 9.879; *6 .?8.925.6; ;7 26=.:;20*;. ;1. 2589.::276 7/ * 1.48.9 76 *
9.,282.6;26*0.6.9*44.*96260:2;<*;276+.;>..61<5*6:&./7,<:.-76;1.97+7;:
+.1*=279 ;1*; 5*3.: ;1. 4.*96.9 >*6; ;7 ;.*,1 ;1. 97+7; ,76,.6;9*;260 76 2;:>.*36.::.:&.;1.6,76-<,;.-*6*--2;276*4.?8.925.6;26>12,1>.,1*60.-;1.
*::2:;2608.9:76/975*1<5*6;7*97+7;;79.*42A.;1.97+7;:+.1*=27957-.46;1.
/<;<9.>.>244,758*9.;1.:.;>7.?8.925.6;:;7.?*526.;1.+.1*=279*457-.4 &.*397+7;::<8879;260+.1*=279!








 

    26)')28=)%67%2-2'6)%7-2+291&)63*63&387,%:)&))2-2863(9')(-28,)*-)0(3*)(9'%8-32
%2( :%6-397 )**)'87 ,%:) &))2 6)4368)(  %2  79++)78)( 8,%8 63&38%77-78)( 2+0-7,
0%2+9%+) 0)%62-2+ -7 136) )**)'8-:) 8,%2 8)<8&33/7 36 '31498)6&%7)( 0)%62-2+ -2 8)617 3*
',-0(6)27'32')286%8-32-28)6)78%2(0)%62-2+%2 2%((-8-32%789(='32(9'8)(&=%879>3)
79++)78)(8,%8%63&38746)7)28%8-323*'366)'8%27;)67%2(8)%',-2+;%7)**)'8-:)-20)%62-2+2+0-7,;36(7%879>3)
 ",)630)3*8,)63&38-783&)8%9+,8;,-', %**)'87 238 320= 8,) )**)'8-:)2)77 3* 8,) 0)%62)67 0)%62-2+ &98 %073 8,) 0)%62)6A7
138-:%8-32830)%62 ",)6)*36)8,-7789(=%-1)(836)%0->)0)%62-2+&=8)%',-2+-2;,-',8,)
0)%62)60)%627&=8)%',-2+%63&38 "3%',-):)8,-7+3%08,-7789(=)<%1-2)78,)630)3*63&387
-2 8)%',-2+ %2( *3'97)7 32 ;)%/ 63&387   ;)%/ 63&38 4)6*3617 %'8-327 %*8)6 )0-'-8-2+
%77-78%2')*63138,)67/%(% 8;390(&)97)*9083()7-+279',0)%62-2+&=+-:-2+8,)
63&38%630)79',%7;)%/2)7783)2'396%+)8,)0)%62)6838)%',;,-',;390(6)7908-28,)
'3140)8-32 3* 8,) 0)%62-2+ 463')77  3;):)6 46):-397 789(-)7 ,%:) &))2 :%+9) -2 8,)-6
()*-2-8-3273*;)%/2)77 28,-7789(=;)%/2)77;%7()*-2)(%78,)()7-6)83,)04 ",)6)*36)
;) '32(9'8)( 8;3 796:)=7 83 :)6-*= ;,%8 /-2( 3* ;)%/2)77)7 '%2 86-++)6 0)%62-2+6)0%8)(
%77-78%2')-20)%62-2+7-89%8-327 ='314%6-2+8,)7)8;3)<4)6-1)2878,);)%/2)77)73*8,)
63&38-20)%62-2+7-89%8-327;)6)'0%6-*-)(   

78,)*-67878)43*<4)6-1)28 ;)796:)=)(%-(;36/)67-146)77-3273*8,)-6%-(6)'-4-)287
-24)6732834)67320)%62-2+7-89%8-327       2320-2)59)78-322%-6)796:)=;%7'32(9'8)((96-2+8,))<4)6-1)28 ",)4%68-'-4%287;)6)
%7/)(836)7432(83)4-73()7-2+)2)6%00)%62-2+7-89%8-3276)+%6(-2+8,)-6)<4)6-)2')7;-8,
%77-78%2')6)%7327*36%77-78%2')%2(-146)77-3273*8,)%77-78)(4)6732 ",)6);)6)
144

4%68-'-4%287-28,))<4)6-1)28 1%0)7%2( *)1%0)7;-8,%1)%2%+)3*  =)%67
78%2(%6(():-%8-32  ",)496437)3*8,))<4)6-1)28;%78397)%59)78-322%-6)83%7/4%68-'-4%28783%27;)6
59)78-327%&398)4-73()73*%77-78%2')-2)%','311320)%62-2+7-89%8-32 28,)%77-78%2')
)4-73();,-',;%7%2)<4)6-1)28%078-190978,)0)%62-2+7-89%8-32%440-)(8371%00):)6=(%=%'873*/-2(2)77'314%77-32 &6-)*796:)=;%7'32(9'8)(320)%7)8)0097%&398=396
'314%77-32(96-2+)(9'%8-32%2(0)%62-2+ @",-7796:)=%2(8,)314%77-32!'%0)&=-/9',- ;)6)97)(%76)*)6)2')783'6)%8)8,)%77-78%2'))4-73()7 %68-'-4%287;)6)%7/)(
836)7432(32%43-287'%0)6%2+-2+*631?2):)6(-(@83?(-(136)@836)'36(8,)-6)<4)6-)2')
%77-78-2+38,)67&%7)(328,)314%77-32!'%0) %68-'-4%287;)6)%073%7/)(83463:-()%
*6)) ()7'6-48-32 3* 8,) 6)%7327 *36 8,)-6 %77-78%2')  7 *36 8,) -146)77-327 3* 8,) %-()(
4)673278,)4%68-'-4%287;)6)%7/)(836)7432(83-8)1779',%7&6-+,8@83?(%6/%2(78632+ 83 ;)%/ *631 8,) %(.)'8-:) 4%-67 -2 8,) :%0-(%8)( 7'%0) *36 1)%796-2+ 4)6732%0-8=
4)6')48-32%7;)00%771%68@83?(91&%2(%4463%',%&0)83-2%4463%',%&0)*36%838%03* -8)1736-$37,-(% 

28,-7)<4)6-1)28(%8%*6313*8,)
79&.)'87;)6)97)(*368,)%2%0=7-7 ",)(%8%;)6)
%2%0=>)(97-2+)<4036%836=*%'836%2%0=7-7*36)%',3*8,) %77-78-:))4-73()7 2-8-%00=*36)%',%-()4-73();)97)(8,)%-7)6)=)60/-21)%796)3*7%140):%0-(-8= ",)6)790873*8,)1)%796)1)287,3;)(%6)79083* 36,-+,)6*36%00%77-78%2'))4-73()7-2(-'%8-2+8,)
4377-&-0-8=3*%438)28-%0*%'836 %7)(328,)%&3:)6)79087*%'836%2%0=7-7;%7'32(9'8)(&=
92-*=-2+8,)1%<-1910-/)0-,33(1)8,3(*36*%'836)<86%'8-32 ",)6)79087-()28-*-)( *%'8367
*368,) %77-78%2'))4-73()7 ",)7)*%'8367;)6)%'8-:-8=73'-%&-0-8=(-0-+)2')6)0-%&-0-8=
430-8)2)77%**-2-8=%+6))%&0)2)776)%'8-:-8=63&9782)77%2(-14%8-)2') :)6%00)%',*%'836,%(%,-+,:%09)*368,)*%'83603%(-2+73*8,)-8)17-2'09()(-28,)*%'836,-+,)<40%2%836=
43;)6*36-2863:)68)(78%8-')8' %2(03;%7%2)<40%2%8-32*368,)*%'836"%&0)  "%&0)  )790873*%'8362%0=7-7*364-73() 
("+#"((*!+$#!+#&(&"&(()%
#(!$%%+
!"'(!'' '! #(&($#'('(#(






 
  

  
        
 
  
 
       

  *8)6'32(9'8-2+8,))<4)6-1)28 *%'8367;)6)-()28-*-)(*36%00 )4-73()7 %',*%'836
8)2()(83&)03;79++)78-2+8,%88,),)04)6*)08731)/-2(3*;)%/2)7783;%6(8,)%77-78-2+
4)6732 8-74377-&0)8,%8)%',*%'836;%7',%6%'8)6->)(&=%1%896)-146)77-323*8,)%'8-:-8=
*%'836 %2( % 7,= -146)77-32 3* 8,) 73'-%&-0-8= *%'836  2 1%2= )4-73()7 8,)6) ;)6) %073
()7'6-48-3273*7=14%8,=79',%7?&)'%97)*)084336@%2(3*'32')6279',%7?&)'%97)
8,)= 033/)( 0-/) 8,)= ;)6) -2 8639&0) @ ",-7 -7 '327-()6)( 8,) 7%1) 8)2()2'= 3* )138-32%0
59%0-8=%7-246):-397789(-)732,)04-2+&),%:-36%/%196%"%/%+-  #)%/2)77)7
,%:)&))279++)78)(83&)/-2:30:)(-20)%62-2+7-89%8-327 ((-8-32%00=8,)*6))()7'6-48-327
-2(-'%8)(8,%813784%68-'-4%287%7791)(%*6-)2(0-/)4)6732 ",-7789(=%073%88)148)(8397) 63&387 %7 0)%62-2+ 4%682)67  ",-7 ;%7 (32) 83 '%4896) 7-1-0%6 86)2(7  ",)6)*36) %2
%((-8-32%0)<4)6-1)28;%7'32(9'8)(&=6)40%'-2+8,)%77-78%28;-8,%63&38 145  
"3'32(9'8<4)6-1)28 6):-7)(8,)46)7)28%8-323*8,)%7791)(7-89%8-32%2(%77-78%2')
)4-73()7%2(8,)%-()4-73()783*-88,)63&387')2%6-3 ",))4-73()798-0->)(-2<4)6-1)28 ;)6) %&786%'8)( %2( 8,3639+,0= (-7'977)( -2 8,) 0%&36%836= 83 '6)%8))4-73()7 %(%48)( 83
'-6'9178%2')7%440-'%&0)838,)63&38       
!-1-0%683<4)6-1)28 %2320-2)796:)=;%7'32(9'8)(*36<4)6-1)28 %68-'-4%287;)6)
%7/)(836)7432(83%+)2)6%00)%62-2+)4-73()6)+%6(-2+8,)*6)59)2'=3*%77-78%2')6)%7327
*36%77-78%2')%2(-146)77-3273*8,)6)'-4-)283*%77-78%2') ",)6);)6) 4%68-'-4%287
1%0)7%2(*)1%0)7;-8,%1)%2%+)3*  =)%6778%2(%6(():-%8-32  
2 <4)6-1)28  8,) 7%1) 59)78-322%-6) %7 -2 <4)6-1)28  ;%7 97)( %2( 8,)4%68-'-4%287;)6)%7/)(836)7432(8359)78-327%&398)4-73()73*%77-78%2');-8,8,)63&38
-2 )%', +)2)6%0 0)%62-2+ 7-89%8-32  ",) 59)78-322%-6) ;%7 ()7-+2)( -2 8,) 7%1) 1%22)6
)<')488,%88,)59)78-327;)6)',%2+)(838,)*6)59)2'=3*%77-78%2')  

7 -2 <4)6-1)28  %2 )<4036%836= *%'836 %2%0=7-7 ;%7 '32(9'8)( 32 8,)  %77-78%2')
)4-73()7 %2( % 0-/);-7) 463')(96) *36 *%'836 %2%0=7-7 ;%7 *3003;)(  -2) *%'8367 ;)6)
-()28-*-)(2%1)0=%'8-:-8=73'-%&-0-8=-28)00-+)2')6)0-%&-0-8=7-2')6-8=*%:36%&-0-8=63&9782)77
'0%6-8=%2(;%618, :)6%00)%',*%'8368)2()(83,%:)%,-+,*%'83603%(-2+:%09)*368,)
-8)17-2'09()(-28,)*%'836%2(%7-2<4)6-1)28 %03;)<40%2%836=:%09) ",)7869'896)3*
)%',*%'836;%77-1-0%6838,%8-2<4)6-1)28 79++)78-2+8,%88,)6)7432()287*)088,)7%1)
7)27)3*;)%/2)77%7-2<4)6-1)28   

!-1-0%6 83 <4)6-1)28  8,) 6)79087 3* <4)6-1)28  79++)78 % 7)27) 3* ;)%/2)77 -2 8,)0)%62-2+ 7-89%8-327  132+ 8,)7) 73'-%&-0-8= -28)00-+)2') 6)0-%&-0-8= %2( *%:36%&-0-8= *%'8367
;)6)-()28-*-)(-21%2=)4-73()7 %2=*%'8367;-8,7-1-0%67869'896)7;)6)-()28-*-)(&98136)
4)6732%0-8=6)0%8)(*%'8367;)6)-()28-*-)(-2<4)6-1)28 ",-7'%2,)046)%0->)8,)63&387&),%:-36 3;):)6-28,-7)<4)6-1)28;)46)7)28)(320=-1%+)73*8,)63&38%2(;)6)92%&0)
83 :)6-*= -87 &),%:-36  2 8,) *9896) 8,) -146)77-327 '32*-61)( -2 8,) )<4)6-1)28 ;-00 &)
6)*0)'8)( -2 % &),%:-36%0 13()0  ",-7 -2*361%8-32 ,%7 &))2 -2'09()( -2 8,) 4378)646)7)28%8-327 


  




*6776.:%7
758*9*;2=.:;<-@76;1..-<,*;276*4<:.7/
175.97+7;:/79,124-9.6        
*;:<A7."#*6*3*
#1.-2//.9.6,.7/.?,.44.6,.26.-<,*;276*4:<8879;97+7;:*//.,;:
,124-9.6C:4.*962606042:1=7,*+<4*92.:           B 26*8*6.:.3*-*
<5*6-.8.6-.6;>.*397+7;:/79,9.*;260:@5+27;2,9.4*;276:>2;11<5*6:7<96*47/;1.!7+7;2,:"7,2.;@7/*8*6  B
26*8*6.:.
23<,12 572@*9273*0*3<:<9<(#1.:",2.6,.7/,758*::276)#73@7*>*:125*
"17;.626*8*6.:.792''7:12-*   ",2.6,.""$"26*8*6.:.
*3*5<9*'#*3*02  :@,17470@7/+.1*=279;71.487;1.9:  26*8*6.:.

146

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Bio-Inspired Method for Personalized
Learning Path Recommendation Problem
Yaqian ZHENGa, b, Deliang WANGc, Yaping XUa, b, Ziqi MAOa, b, Yaqi ZHAOa, b & Yanyan
LIa, b*a
School of Educational Technology, Beijing Normal University, Chinab
Research Center for Knowledge Engineering, Beijing Normal University, Chinac
Faculty of Education, The University of Hong Kong, Hong Kong*liyy@bnu.edu.cn
Abstract: The recommendation of personalized learning paths is recognized as one of
the most challenging aspects in the field of e-learning. In the existing literature,
numerous approaches have been proposed to identify appropriate learning paths for
e-learners, taking into consideration multiple perspectives. However, the current stateof research lacks a unified framework that effectively integrates the most vital
parameters associated with the learner, learning object (LO), and domain knowledge
to generate optimal learning paths. To address this challenge, a novel bio-inspired
approach is proposed for solving the personalized learning path problem. In thismethod, we initially incorporate the learner, LO and domain knowledge models into a
unified mathematical model. Then an enhanced ant colony optimization algorithm isutilized to determine the optimal personalized learning paths for learners. To
investigate the effectiveness of the proposed method, we performed several
computational experiments based on six simulation datasets. The results indicate thatthe proposed method surpasses other competing methods in terms of performance androbustness, showcasing its superior effectiveness.
Keywords: E-learning, learning path recommendation, unified model, ant colony
optimization algorithm

1. Introduction
E-learning has gained significant prominence in the education sector, providing remarkable
opportunities for learning (Al-Fraihat et al., 2020). However, e-learners, particularly those withlimited experience in a specific domain, may face challenges such as information overload
and learning disorientation when confronted with a wide range of resources (Meng et al.,2021). In order to tackle this issue, several methods have been proposed for the automatedgeneration of personalized learning paths for e-learners. These methods aim to optimize
learning paths from various perspectives, including domain knowledge structure and learning
preferences, among others. Nevertheless, there is currently no unified framework that
effectively incorporates the most significant parameters linked to the learner, learning object(LO), and domain knowledge in order to create optimal learning paths. To this end, a novel
bio-inspired approach is proposed to discover personalized learning paths for e-learners, inwhich the important parameters associated with learner, LO and domain knowledge areformulated as a unified mathematical model and an enhance improved ant colony optimization(EACO) algorithm is utilized to identify the optimal personalized learning path.

2. Problem Formulation
The personalized learning path recommendation problem (PLPRP) is represented by a threetuple consisting of a learner model, LO model, and knowledge graph. Figure 1 presents a
description of the problem formulation.

147

Figure 1. Description of the problem formulation for the PLPRP.
Here, the PLPRP is considered as a constrained optimization problem. The unified
mathematical model is expressed as follows:
𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 𝐹 = 𝑤𝑚𝑎𝑡𝑐ℎ 𝐹𝑚𝑎𝑡𝑐ℎ + 𝑤𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑖𝑡𝑦 𝐹𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑖𝑡𝑦(1)
Subject to
𝐾𝑃_𝑠𝑒𝑡(𝐺′) ⊆ 𝐾𝑃_𝑠𝑒𝑡(𝑃)(2)(3)(𝐾𝑃𝑖 , 𝐾𝑃𝑗 ) ∈ 𝐸, ∀𝐾𝑃𝑖 , 𝐾𝑃𝑗 ∈ 𝐾𝑃𝑠𝑒𝑡 (𝑃), 1 ≤ 𝑖 < 𝑗 ≤ 𝐾𝑃_𝑛𝑢𝑚𝑏𝑒𝑟(𝑃)where 𝐹𝑚𝑎𝑡𝑐ℎ assesses the matching degree between the learner and LOs based on
Learner-to-LO matching rules and 𝐹𝑐𝑜𝑛𝑡𝑖𝑛𝑢𝑖𝑡𝑦 evaluates the path's continuity based on thedependency relations between KPs. Constraint (2) is employed to ensure the path's
completeness, while constraint (3) is utilized to guarantee its legality.

3. A Bio-Inspired Method for Solving the PLPRP
To solve the model of PLPRP, a bio-inspired algorithm, i.e., an enhanced ant colony
optimization algorithm (EACO) is proposed.
⚫
Problem representation. The problem space is transformed into a KP space and LO
space. Once an ant selects a node, the KP and LO space undergo dynamic updates.
When the KP space becomes empty, a feasible learning path solution is constructed.
⚫
Heuristic information. The matching degree between the learner and LO is leveraged todesign the new heuristic information.
⚫
Pheromone initialization. Following prior work (Yu et al., 2018), the value of pheromone
is initialized by a greedy nearest-neighbor algorithm.
⚫
Dynamic candidate set strategy. To generate feasible solutions of learning path, adynamic candidate set strategy is presented, which constructs the candidate set for eachelement based on the dependency relations.
⚫
Selection of the initial node. The heuristic information and the roulette selection rule areutilized to determine the initial positions of the ants.
⚫
Selection of the next node. The selection of the next node is determined using thepseudorandom-proportional rule (Ma et al., 2021).
⚫
Pheromone Update. After an ant finishes constructing a solution, the pheromone levels
on the edges visited by the ant in the obtained solution are locally updated. Once all theants have completed their solutions, the pheromone trails are evaporated, and only thebest-so-far ant is permitted to release pheromone on all the edges along its traveled path.

4. Computational Experiments
We generate multiple simulation datasets to perform computational experiments and assess
the effectiveness of our proposed algorithm. The datasets are defined based on the number
of KPs and LOs. The specific settings are as follows: (10, 30), (15, 45), (25, 75), (40, 120),(60, 180), and (90, 270). For the purpose of comparison, three widely used algorithms, namely
random algorithm (RA), genetic algorithm (GA), and standard ant colony optimization

148

algorithm (SACO), are employed. Considering the inherent stochastic nature of thesealgorithms, each of them is independently executed 20 times for every dataset.
Table 1 provides a summary of the computational results, presenting the average
objective function value over 20 runs (labeled as Obj) and the standard deviation of theobjective function values across different trial runs (labeled as Std). The values of Obj obtainedby RA are always the highest, so its performance is the poorest. This indicates that learners
encounter challenges when attempting to independently explore suitable learning paths. GA,
SACO, and EACO obtain better solutions than RA, demonstrating that employing heuristic
algorithms can effectively guide the search process toward more promising areas. In all
scenarios, EACO demonstrates the highest level of performance, and its superiority becomesincreasingly prominent as the problem dimension expands. In comparison to other algorithms,
RA exhibits significantly larger values of Std, which imply its limited stability. On the other hand,
IACO consistently achieves the smallest Std values across all scenarios, highlighting its
remarkable robustness.
Table 1. Performance of the Proposed Algorithm and Other Competitors on Six Datasets
Dataset
1
2
3
4
5
6

RA
Obj
1.25301.16261.2734
1.3364
1.6351
1.7216GA
Std0.17450.15890.17280.14970.12620.1375Obj
0.53390.5141
0.96490.98881.3335
1.1698Std
0.05140.05850.04260.02240.04740.0159

SACO
Obj
Std
0.3430 0.0188
0.3106 0.0199
0.7035 0.0231
0.6915 0.0210
1.0830 0.0199
1.1670 0.0342

EACO
Obj
Std
0.32500.00000.22500.00000.21700.00240.19660.00650.13080.00220.21120.00595. Conclusions
In this work, we propose a bio-inspired algorithm to address the personalized learning path
recommendation problem (PLPRP). The experimental results indicate that compared withother competitors, the proposed algorithm achieves better performance and robustness when
dealing with the PLPRP across varying sizes and levels of complexity.

Acknowledgements
This work has been supported by the National Natural Science Foundation of China (Grant
No: 62277006), the Beijing Natural Science Foundation (Grant No: 9222019), the Science andTechnology Research Youth Fund Project of Higher Education Institutions of Hebei Province(Grant No: QN2018261) and the Student Research Fund of the Faculty of Education at Beijing
Normal University (Grant No: 2212216).

References
Al-Fraihat, D., Joy, M., Masa’deh, R., & Sinclair, J. (2020). Evaluating E-learning systems success: An
empirical study. Computers in Human Behavior, 102, 67–86.
Meng, L., Zhang, W., Chu, Y., & Zhang, M. (2021). LD–LP generation of personalized learning pathbased on learning diagnosis. IEEE Transactions on Learning Technologies, 14(1), 122-128.
Nabizadeh, A. H., Leal, J. P., Rafsanjani, H. N., & Shah, R. R. (2020). Learning path personalization
and recommendation methods: A survey of the state-of-the-art. Expert Systems with Applications,
159, 113596.
Yu, X., Chen, W. N., Gu, T., Yuan, H., Zhang, H., & Zhang, J. (2018). ACO-A*: Ant colony optimization
plus A* for 3-D traveling in environments with dense obstacles. IEEE Transactions on Evolutionary
Computation, 23(4), 617-631.
Ma, W., Zhou, X., Zhu, H., Li, L., & Jiao, L. (2021). A two-stage hybrid ant colony optimization for highdimensional feature selection. Pattern Recognition, 116, 107933.

149

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Developing Effective Educational Chatbots
with ChatGPT prompts: Insights from
Preliminary Tests in a Case Study on
Social Media Literacy
Cansu KOYUTURKa+, Mona YAVARIa+, Emily THEOPHILOUb, Sathya BURSICa, Gregor
DONABAUERa,c , Alessia TELARIa, Alessia TESTAa,
Raffaele BOIANOd, Alessandro GABBIADINIa, Davinia HERNANDEZ-LEOb* ,Martin RUSKOVe** & Dimitri OGNIBENEa,f***a
Università of Milano Bicocca, Milan, Italy b Universitat Pompeu Fabra, Barcelona, Spainc
University of Regensburg, Regensburg, Germany dPolitecnico di Milano, Milan, Italye
Università of Milano, Milan, Italy f University of Essex, Colchester, United Kingdom*davinia.hernandez-leo@upf.edu, **martin.ruskov@unimi.it, ***Lead PI dimitri.ognibene@unimib.it+Same contribution
Abstract: Educational chatbots come with a promise of interactive and personalized
learning experiences, yet their development has been limited by the restricted freeinteraction capabilities of available platforms and the difficulty of encoding knowledge
in a suitable format. Recent advances in language learning models with zero-shotlearning capabilities, such as ChatGPT, suggest a new possibility for developing
educational chatbots using a prompt-based approach. We present a case study with asimple system that enables mixed-turn interactions and discuss the insights andpreliminary guidelines obtained from initial tests. We examine ChatGPT's ability to
pursue natural educational conversations, adapt the educational activity to users'
characteristics, such as culture, age, and level of education, and its ability to use
diverse educational strategies and conversational styles. Although the results areencouraging, challenges are posed by the highly structured form of responses by
ChatGPT, as well as their variability, which can lead to an unexpected switch of thechatbot's role from a teacher to a therapist. We provide some initial guidelines toaddress these issues and to facilitate the development of effective educational
chatbots.
Keywords: ChatGPT, Educational chatbots, Educational prompts

1. Introduction
Educational chatbots have gained increasing attention in recent years for their potential toprovide interactive and personalized educational activities with reduced teacher intervention.
However, their development has been mostly focused on chatbot-driven conversation flow
due to the complexity of mixed-initiative and user-driven approaches, the restrictedinteraction capabilities of available platforms, the lack of adequate training sets (Pérez et al.,2020), and the substantial efforts required to encode the necessary knowledge in a suitableformat, notwithstanding the wide availability of topic-specific information.
Recent advances in Large Language Model (LLM) systems with their zero-shotlearning capabilities (Wei et al.,2021), also named prompt-based approaches (Xu et al.,2023), to perform diverse tasks in different domains without training or fine-tuning suggest a
new way to develop educational chatbots. Chatbots based on Reinforcement Learning from
Human Feedback (RLHF), such as ChatGPT, have been extended to achieve more reliable
and safe interactions (Ouyang et al., 2022)

150

While showing big promises, LLM-based chatbots and their potential are not yet fully
understood (Mahowald et al., 2023; Borji,2023) and present different sources of vulnerability(Xu et al. 2023). The skills necessary for natural educational conversations, such as playing
a specific role, maintaining a conversation style (Lu et al.,2023), and assessing students’conditions (Ullman, 2023), are still challenging even for state-of-the-art systems.
We propose an investigation framework to study ChatGPT’s ability to produce mixedturn educational conversations where the students can steer the conversation toward
desired topics as well as cover multiple related learning objectives, tailor educational
activities to user characteristics such as culture and age and use different educational
strategies and conversational styles. Building on Jeon and Lee (2023), in this paper, we
provide suggestions for possible teacher orchestration and interrogation strategies and study
whether they lead to successful interlocution and content provision. We explore different
prompting strategies to test LLM capabilities in educational conversations with a particularfocus on reliability, interactivity, and coverage. We report results from tests performed by theauthors.

2. Methodology
To frame interactive educational chats with ChatGPT, we focused on the "space of learning"(Marton and Tsui, 2004). We explored the educational exchange during lessons, especiallythe challenge of imperfect communication between teachers and students. Adding thechatbot as a teacher mediator (Fig. 1), we interpreted the learning interaction using an
adaptation of Laurillard's Conversational Framework (2009), considering the distinctions
between conceptions and expressions for each role (teacher, student, and chatbotmediator).

Figure 1. The educational framework: The teacher's intention to convey learning objectives isexpressed through generated prompts in the technical framework. The chatbot's stochastic model
determines the communication directed to the student. As the student interprets the exchanged
messages, their understanding takes shape. Consequently, the actual experience of the messages
differs from the teacher's intention, the chatbot's representation, and the student's comprehension.
To establish a learning space and facilitate a directed educational exchange we haveformulated six blocks of educational objectives to be integrated into the prompt. Each block
targets a critical area of social media literacy (Ognibene et al., 2023;Theophilou et al., 2023).
The assessment of the interaction quality between ChatGPT and users was evaluated basedon four dimensions: correctness, consistency, conversation fluency, and semantic coverage.

3. Results and Conclusions
We report: I. the insights that emerged during our search for prompts that resulted in
effective educational conversations, and II. a classification of the issues we encountered. For
details see Koyuturk et al. (2023). Our study showed that ChatGPT with the correct prompt
could play the role of the teacher in short interactive educational conversations covering
multiple learning objectives related to social media literacy. We found that it is useful andfunctional to explain in the prompt every foreseeable aspect of the interaction one by one.
We aimed to clearly define the structure of interaction and the role of the chatbot. By testingwith eight personas with different cultural backgrounds and ages, we verified that ChatGPT
could also adapt to students’ characteristics. However, relatively small variations of theprompt lead to a range of issues (see table 1).

151

Table 1. Classification of observed issues in failed natural educational interactions.
1. Unnatural Response Format
1. Bullet style
2. Essay Style
2. Interactivity
1. No questions
2. Survey style question presentation
3. Fixed sequence
3. Limited coverage

4. Unreliable behaviors
1. Behavior switching
1. Therapist Eliza style
2. Encyclopedic Q/A
3. Programmer assistant
2. Personal questions
3. Introduction of code-like variable placeholder
4. Imaginary students’ answers

ChatGPT appears to be fine-tuned for question-answering, resulting in responses
that may not align with the desired style or exhibit essential conversational elements. Initial
prompt attempts resulted in unnatural response formats. Despite instructions to adopt a
conversational style and avoid bullet points or essay formats, the chatbot frequently
deviated. Turn-taking and questions were either lacking or excessive and the ability to cover
multiple learning objectives and related topics was limited. Conversations often continued
only when users asked relevant questions about previously unaddressed topics.
Interesting erratic and undesirable behaviors were observed. Requests to make theeducational conversation interactive sometimes resulted in the failure to effectively maintainthe teacher role. One of the most surprising and frequent behaviors was the tendency toautonomously cover both sides of the educational conversation. Our work shows thatstudying and extending the LLM’s capabilities to consistently play roles is an essential line ofresearch.

Acknowledgements
Spanish AEI (PID2020-112584RB-C33, MDM-2015-0502), Volkswagen S. (ref. 95 566 and
9B145) and Catalan SGR (00930). DHL (Serra Húnter) acknowledges the support of ICREA
Academia.

References
Borji, A. (2023). A categorical archive of ChatGPT failures. arXiv:2302.03494.
Jeon, J., Lee, S. (2023). Large language models in education: A focus on the complementary
relationship between human teachers and ChatGPT.https://doi.org/10.1007/s10639-023-11834-1
Koyuturk, C., ... & Ognibene, D. (2023). Developing Effective Educational Chatbots with ChatGPT
prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (withappendix). arXiv:2306.10645
Laurillard, D. (2009). The pedagogical challenges to collaborative technologies. Int. J. of CSCL, 4, 520.
Lu, A., Zhang, H., Zhang, Y., Wang, X., & Yang, D. (2023). Bounding the Capabilities of Large
Language Models in Open Text Generation with Prompt Constraints. arXiv:2302.09185.
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023).
Dissociating language and thought in large language models: A cognitive perspective.
arXiv:2301.06627.
Marton, F., Tsui, A. B., Chik, P. P., Ko, P. Y., & Lo, M. L. (2004). Classroom discourse and the spaceof learning. Routledge.
Ognibene, D., ... & Eimler, S. (2023). Challenging social media threats using collective well-beingaware recommendation algorithms and an educational virtual companion. Frontiers in AI, 5.
Ouyang, L., ... Schulman, J. (2022). Training language models to follow instructions with human
feedback. In Adv. in Neural Inf. Proc. Sys.35,27730-27744.
Pérez, J.Q., Daradoumis, T., & Puig, J.M.M. (2020). Rediscovering the use of chatbots in education: A
systematic literature review. Comp. Appl. in Eng. Edu., 28(6), 1549–1565.
Theophilou, E., … & Eimler, S. (2023). Empirically Investigating Virtual Learning Companions toEnhance Social Media Literacy. 345–360. HelMETO https://doi.org/10.1007/978-3-031-298004_27
Ullman, T (2023). Large language models fail on trivial alteration to theory-of-mind task.
arXiv:2302.08399
Wei, J., ... & Le, Q. V. (2021). Finetuned language models are zero-shot learners. arXiv:2109.01652.
Xu, L., Chen, Y., Cui, G., Gao, H., & Liu, Z. (2022). Exploring the universal vulnerability of promptbased learning paradigm. arXiv:2204.05239

152

CSCL

153

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Tackling Unserious Raters in Peer
Evaluation: Behavior Analysis and Early
Detection with Learner Model
Changhao LIANGa*, Izumi HORIKOSHIb, Rwitajit MAJUMDARb & Hiroaki OGATAba
Graduate School of Informatics, Kyoto University, Japanb
Academic Center for Computing and Media Studies, Kyoto University, Japan* liang.changhao.84c@st.kyoto-u.ac.jp
Abstract: Peer evaluation of individual or group work is often adopted in team-basedlearning design. However, some raters may not take the evaluation process seriously
and exhibit behaviors such as using the same score, rushing through evaluations, or
not evaluating during the presentation. This study investigates the issue of unserious
peer evaluation in group presentations, focusing on their behavior patterns. Usingevaluation behavior analysis indicators, we identified unserious raters who exhibited
low reliability in the peer evaluation process. Further, we conducted a preliminary
analysis to detect unserious raters based on learner model data available before thepeer evaluation process. This information can assist teachers in providing personalized
prompts and interventions prior to the peer evaluation process, thus enhancing theevaluation quality of these students with timely prompts to them.
Keywords: Peer evaluation, Evaluation behavior analysis (EBA), Team-basedlearning (TBA), Data-driven study, Learner model

1. Introduction
Evaluation is an essential aspect in collaborative learning, but teachers may struggle toproperly evaluate each student (Amarasinghe et al., 2021). Peer evaluation offers formative
feedback that encourages reflection and overcomes the limitations of traditional evaluation(Ohland et al., 2012). It has become widely adopted in online settings where student-centeredlearning is prevalent and can enhance both learning and interpersonal skills (Kasch et al.,2021). However, some raters may not take the evaluation process seriously, as Horikoshi andTamura (2021) discovered. Such evaluations may involve using the same score, rushing
through evaluations, or not evaluating during the presentation. These low-quality ratings canmake peer evaluation results less reliable and lower the learning outcome.
Nevertheless, the reliability of unserious peer raters can be improved by properinterventions (Van Zundert et al., 2010). Current studies have made attempts to calibrate
scores based on student engagement and previous peformance (Piech et al., 2013), or train
evaluation skills during the peer evaluation process (Gorham et al., 2023). These approaches
can be too late to nudge timely interventions to the ongoing evaluation activity. Fewer work
addresses predicting problem raters early before the assessment to facilitate possible
interventions to improve their evaluation behaviors.
This study investigates the issue of unserious peer evaluation in group presentations,focusing on their behavior patterns. Using behavioral indicators, we identified unserious raters
who exhibited low reliability in the peer evaluation process. Further, we conducted apreliminary analysis to examine how the learner model data from their learning logs and theirprior peer evaluation behaviors can be used for early detection. This information can assist
teachers in providing personalized prompts and interventions prior to the peer evaluation
process, thus enhancing the evaluation quality of these students in a timely manner.

154

2. Research Background
2.1 Peer evaluation in Team-Based Learning (TBL)
In peer evaluation, students provide ratings and feedback on each other's work, which isformative and can promote their performance in subsequent tasks (Ohland et al., 2012;
Gorham et al., 2023). Research has shown that peer evaluation encourages students to think
deeply and critically about their own work and contributes to the development of "internal
feedback" skills, where learners reflect on and regulate their own learning processes (Nicol etal., 2014; To & Panadero, 2019).
Team-Based Learning (TBL) is an educational strategy that involves multiple roundsof group work with peer evaluation. It was first introduced in medical education (Michaelsen etal., 2002). During each round of TBL, students start by exploring the learning topic individually
before working in teams to complete tasks (Parmelee et al., 2012). Group presentations andpeer evaluation conclude each round, where students assess the products or outcomes of
their peers' learning experiences and reflect themselves as a formative process (Topping,1998). Moreover, under the data-driven environment, previous rounds' learning log data
enables targeted interventions by teachers (Johnson, 2017).

2.2 Evaluation Behavior Analysis (EBA)
The process of peer evaluation generates behavior indicators that record key information,such as the identity of the evaluator, the timing of the evaluation, the items assessed, andthe corresponding scores (Horikoshi & Tamura, 2021). The behavior indicators stem from"paradata" in the web survey research field, which refers to the log data generated during
the evaluation process and is related to the quality of survey responses (Couper & Kreuter,2013). For instance, shorter response times are associated with a "lack of motivation toanswer accurately caused by continuous survey" (Yan & Tourangeau, 2008), and individuals
who answer quickly as "speeders" can lead to poor responses (Zhang & Conrad, 2014).
The web survey research and peer evaluation research share the goal of measuring
inappropriate behaviors in digital evaluation platforms. Therefore, to effectively analyze andvisualize the quality of peer evaluation based on behaviors, the Evaluation Behavior Analysis(EBA) method has been developed. It involves extracting data from peer evaluations andutilizing it to gain insights into students' evaluation behaviors. Using the EBA method,
instructors can identify patterns and trends in the evaluation behavior of students. Horikoshi
et al. (2022) have defined feature variables that capture the key aspects of evaluation
behavior, which are presented in Table 1.
Table 1. Definition of feature variables of evaluation behaviors (Horikoshi et al., 2022)
Behavior Indicator
Evaluation Time (ET)Mean of the Timestamp(tM)
SD of the Timestamp(tSD)
Click Count (CC)
Mean of the Score (sM)

Definition
Time span from clicking thefirst evaluation item to thelast item.
Average elapsed time since
the start of the presentation.
Standard deviation of thetimestamps for all
evaluations.
Total number of times theevaluation items wereclicked.
Average score for all theevaluation items scored by
the reviewer

155

Proposed constructs
Speed: how much time therater spent on the evaluation
Timeliness: whether the rater
evaluated immediately after
the presentation
Coherence: whether the rater
evaluated evenly throughout orwithin a short time
Certainty: how many changes
the rater madeLeniency: rater tendency toassign higher or lower scoresSD of the Score (sSD)

Standard deviation for thescores of all evaluation items
from the rater.

Straightlining: whether therater used only similar scores(Kim et al., 2019)

Compared to the conventional perspective of peer evaluation quality, which primarilyemphasizes scores and compliance with others (Cho & Schunn, 2007; Fukazawa, 2010), the
EBA indicators focus on the process of the peer evaluation. These evaluation behavior
indicators go beyond mere consistency and provide insights into various aspects of peer
evaluation performance (see Proposed construct in Table 1). Such insights can inform
instructional design and enable targeted feedback. By identifying strengths and weaknessesin specific behavior indicators, EBA allows for the recognition of areas that need intervention,
thus promoting the development of peer evaluation skills in line with the goal of formative
assessment and learning enhancement.

2.3 Data-driven peer evaluation with learner model
Peer evaluation systems offer learners a scaffold to evaluate their group members and receive
real-time feedback with reduced bias, enhanced individualism and privacy protection (Ismail
et al., 2016; Cleynen et al., 2020). In online evaluation systems, both the evaluation outputs
and the evaluation processes of raters can be traced, providing valuable data for learning
analytics applications as part of the learner model attribute. The concept of the "learner model"
encompasses domain-specific and domain-independent information, quantified as learning
evidence that varies according to the learning context (Boticki et al., 2019). These indicators
can derive from learning behaviors recorded on learning management platforms (LMS) suchas e-book reading logs, academic scores, previous experiences in group work, and otherrelevant data. In the context of TBL, the learner model can be dynamic and continuously
updated with the accumulation of data from each round. To support this process, Group
Learning Orchestration Based on Evidence (GLOBE) (Liang et al., 2021) was proposed as an
infrastructure that provides data-driven support for group work based on learner model data.
Peer evaluation plays a significant role in GLOBE, serving as a module for collecting peer
ratings and feedback (Liang et al., 2022a), while also contributing to the modeling of effective
group work and task experiences (Janssen & Kirschner, 2020). By synchronizing theevaluation data with other collaboration attributes from the prior phase, the learner model canbe utilized for subsequent rounds of TBL.
The data-driven perspective has been adopted to assess the quality of peer evaluation
in individual tasks. For instance, Piech et al. (2013) developed tuned models of rating reliability
based on students' previous performance in individual design assignments. Besides, there arestudies focusing on written reviews for writing artifacts. Cho & Schunn (2007) considered
consistency with others to model reviewers' capabilities, while Patchan et al. (2016) extracted
features from review texts, such as sentimental tendencies and comment types, using
semantic analysis to build a regression model. Regarding peer evaluation in group work, Lianget al. (2022a) demonstrated that the accumulated learner model, incorporating data on group
work and task experiences, can estimate the consistency of peer evaluation using GLOBE.
However, for iterative TBL with multiple rounds of group work, the detection of evaluation
behaviors on rating scores has yet to be extensively investigated.

3. Method
In this study, we conduct two analyses to answer the research questions. First, we examined
the behavior patterns of unserious raters based on evaluation behavior indicators using
clustering analysis. Then, to investigate the potential of using the learner model data fromlearning logs for early detection, a preliminary analysis of classification was conducted. The
research questions are as follows.
z
RQ1: What are the peer evaluation behavior patterns?

156

z

RQ2: How can the learner model data be used to early detect unserious raters?

3.1 Participant and context
The data of this study comes from a course of a Japanese university with a 4-week experiment.
The course is for students beyond sophomore in computer science, with 35 students enrolled
this year. The experiment employed an adapted TBL and jigsaw design, which is shown in
Figure 1.

Figure 1. Workflow of the class.
In the first week of the experiment, a lecture on a new topic was delivered and
BookRoll, an e-book reading tool that allows instructors to upload learning materials before
each class and enables students to engage in various activities during their reading (Ogata etal., 2015), was introduced. Out-of-class activities included reviewing lectures on BookRoll,participating in forum discussions, and completing assignments to summarize them. Startingfrom the second week, in-class activities began with group sharing of the previous week's
assignments. Each student presented the outcome from their forum discussion group in a
jigsaw group. Both the forum discussion groups and jigsaw groups were created by the groupformation system of GLOBE (Liang et al., 2021) based on each student's reading engagement
in BookRoll. In each jigsaw group, the audience provided peer ratings on the individual
presentation through the peer evaluation system. The jigsaw group then became the forum
discussion group for the following week. Following this, a lecture on the topic of the second
week was delivered. This workflow was repeated twice in the first three weeks, and as an
assignment in the third week, students created a presentation to the whole class, summarizing
what they had learned so far and presented it in the final week's class. The behavior patternanalysis in this study is based on the peer evaluation of this final presentation.

3.2 Data collection and preprocessing
In order to evaluate the final group presentation, students were instructed to assign a scoreon a 5-star scale to each group in the peer evaluation system (Liang et al., 2022a). The rubric
was displayed at the top of the rating section in the peer evaluation system for reference (seeFigure 2). The system also recorded a log of the timestamp and rating score each time a rating
button was clicked by a student. To ensure privacy, the identity of each student wasanonymized from the log. Using the clicking logs, six evaluation behavior indicators introducedin Table 1 were calculated. These indicators are used for visualization and clustering.
To detect unserious behavior prior to the peer evaluation, data from the learner model
was collected. In this study, the following learner model data was available before the final
peer evaluation of presentations:z
Reading engagement (RE), which includes reading time, operation times, completionrate, and the number of red markers, yellow markers, and memos on the e-book platform
BookRoll (Ogata et al., 2015). A comprehensive coefficient was calculated by averaging
the percentage rank of the aforementioned indicators to represent reading engagement.

157

z

z

Forum engagement (FE), which consolidates the number of forum posts and charactersin the out-class forum discussion. The percentage rank of the former indicators was
consolidated to represent the forum engagement.
Prior evaluation behavior indicators, which refer to indicators collected during the peer
evaluation of individual presentations in jigsaw groups in the second (round 1) and third
week (round 2). The six indicators introduced in Table 1 were collected for the first tworounds as the input indicators for classification.

Figure 2. Peer evaluation system with rubrics.
As some of the prior evaluation behavior indicators were found to be highly correlated
and estimating the same facet, as also mentioned in Horikoshi et al. (2022), we performed
dimension reduction through factor analysis. Based on the factor analysis, we combined ET
and tSD as “time feature (TF)” (explaining 99.01% of variance for round 1 and 98.63% forround 2), and sM and sSD as “scoring feature (SF)” (explaining 84.91% of variance for round1 and 91.97% for round 2). Additionally, since the extent of polarization in tM was deemed
important in the pattern, we derived a new indicator (tDEV) from tM, which represents the zscore of tM and describes the deviation of rating time from the mean. CC is treated as anindependent indicator due to its low correlation with other features. We used eight priorbehavior indicators (four for each round) for the classification modeling.

3.3 Data analysis
To answer RQ1, we performed a clustering analysis to differentiate unserious raters from theparticipants. This analysis entailed clustering the students according to their evaluation
behavior indicators, which were obtained from the final round of peer evaluation (for grouppresentation). We utilized the K-means method to obtain two distinct clusters, with the highest
silhouette score. Subsequently, we examined the behavior patterns of the students byanalyzing the distributions of each evaluation behavior indicator within each cluster.
For RQ2, we approached it as a binary classification problem to determine if the rateris unserious in evaluating the final group presentation. To accomplish this, we tested fivecommonly used machine learning classification models for numerical data and evaluated their158

performance using the Area Under Curve (AUC) (Fawcett, 2006), with values ranging from 0
to 1. Furthermore, we conducted a feature ablation analysis (Gabrilovich & Markovitch, 2004)
based on the information gain (IG) of ten input indicators as discussed in the previous section,
to figure out the predictive indicators for the classification.

4. Result
4.1 Behavior patterns clustering
Figure 3 illustrates the distribution of EBA indicators for each cluster. It is evident that students
in cluster C1 possess longer ET, more CC, and give a wider range of scores with lower sM
and higher sSD. Although tM does not show a significant difference between the two clusters,
the dispersion differs. Raters in C1 participated in peer evaluations during the presentation,and their distribution of timestamps appears to be more normalized. On the other hand,
students in cluster C2 have shorter ET, fewer CC, polarized tM, and smaller tSD. Regarding
scores, they tend to provide full marks, indicated by high sM and minimal sSD.

Figure 3. Distribution of EBA indicators of the two clusters.
Table 2. Statistical analysis of EBA indicators for clustering
ET
tM
tSD
CC
sM
sSD

Cluster
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2
C1
C2

N
21
14
21
14
21
13
21
14
21
14
21
13

Mean
41.7169.658
58.87050.77517.137
5.173
6.762
4.071
3.941
4.777
0.746
0.113

***p < .001.

159

SD
11.25910.7488.416
19.494
5.670
5.401
2.791
1.492
0.508
0.378
0.370
0.218

p
< .001***
0.077< .001***< .001***< .001***< .001***

4.2 Early detection of unserious raters
Figure 4 presents a performance comparison of various classification methods when using
the top N input indicators ranked by IG, and Table 2 listed these indicators in the order of their
IG in the classification modeling. Our analysis suggests that neural network and logistic
regression models outperform other methods when utilizing the top five to six input indicatorswith high information gains. The AUC scores were 0.738 for the 5-feature condition (Neural
Network) and 0.731 for the 6-feature condition (Logistic Regression).
As for predictive indicators, we observed that the deviation rating timestamp for round2, indicating a straightlining pattern, had the highest IG. Additionally, SF for both rounds
exhibited high information gains. Interestingly, all four prior behavior indicators for round 2
ranked in the top six indicators of the classification model. We also observed a significant
difference between the two groups in SF for round 1 and TF for round 2. Meanwhile, thereading behaviors of the two groups that occurred before the assessment started. The RE
feature also provided valuable information for distinguishing between different classes in aclassification, underscoring the importance of integrating learning model data in predictivemodeling. Conversely, the tDEV, CC, and TF of round 1 had low IG, which could be attributed
to the unfamiliarity with the system in the first round as students needed time to get
accustomed to it.

Figure 4. Prediction accuracy of classification based on learner model
Table 2. Input indicators for the classification modelling ranked by information gain
Rank
Indicator
1tDEV-2
2
SF-2
3
RE
4
SF-1
5
CC-2
6
TF-2
7
tDEV-1
8
FE
9
CC-1
10
TF-1
*p < .05, **p < .01.

Information Gain (IG)
0.226
0.211
0.205
0.178
0.154
0.077
0.071
0.055
0.049
0.031

160

t
0.974
0.865
0.971
3.251**
1.976
2.498*
0.974
1.264
0.773
0.397

5. Discussion
The findings of this study emphasize the significance of integrating learning model data in peer
evaluation of TBL. By using EBA indicators, we can analyze the time and scoring features ofpeer evaluation as the presentation progresses. These indicators can reveal behavior patterns
suggested by Horikoshi & Tamura (2021) such as modifying the evaluation, spending time on
the evaluation, or evaluating all evaluation items earlier or many evaluation items later. The
clustering corroborates these patterns and identifies characteristics of unserious raters.
To provide a clearer understanding of the behavior pattern, Figure 5 displays plots ofthe evaluation behavior of typical raters in the two clusters, indicating the timestamps, scores,
and rating intervals. The x-axis represents the elapsed time from the start of the first grouppresentation, and the y-axis denotes the candidate number of peer ratings. It can be observedthat typical students in C1 tend to rate each candidate group across the group presentation
sessions with even intervals. Moreover, they use different scores with noticeable variations.
In contrast, typical students in C2 exhibit a straightlining and speedy pattern (Zhang & Conrad,2014; Kim et al., 2019), completing their rating very quickly either at the beginning or the endof the session. In summary, C1 raters spend more time evaluating their peers, give a diverse
range of scores with less agreement among themselves, and exhibit a more even distributionof timestamps when giving their evaluations. C2 raters, on the other hand, spend less timeevaluating their peers, give higher scores with less variance, and show a polarized distributionof timestamps for their evaluations. These differences suggest that C1 raters demonstrate
more thoughtful and critical evaluations, while C2 raters appear to be more lenient and lessengaged in the evaluation process.

Figure 5. Visualization evaluation behavior of typical raters in two clusters.
Moreover, this study presents the potential of using learner model data collected fromall phases of TBL in previous rounds to early predict unserious raters. Our analysis shows thatscoring features in each round of TBL play a significant role in the detection model. Timefeatures, which describe the time distribution and frequency of the ratings, can also be
predictive when TBL is conducted over multiple rounds and raters become familiar with thesystem. Furthermore, the engagement of students in individual reading activities can serve asa predictor of unserious raters, while their forum engagement appears to be less relevant. This
discrepancy may be attributed to forum posts being compulsory and formatted as part of thecourse grade, resulting in minimal variation among learners. In summary, the prediction modelis expected to empower instructors to provide remedial instructions or give automatic nudges
to these at-risk students, improving the reliability of peer evaluation as a formative assessment
in TBL. These prompts can be delivered through group awareness tools (Strauß & Rummel,2021) and email interventions (Damgaard & Nielsen, 2018).
The study also contributes to TBL design by introducing the potential of data-driven
scaffolding with multiple rounds of group learning, where learning log data from previous
rounds can be utilized for various learning analytics purposes. One example of data-driven

161

support is provided for the detection of unserious raters in peer evaluation, which is intended
to improve the quality of peer assessment. Furthermore, continuous data support has broader
applications in group learning. The learning logs of previous activities can provide data forcreating groups (Liang et al., 2022b) and calibrating peer rating scores (Piech et al., 2013).
The accumulated data can also be useful for data visualization platforms for reflecting onteaching interventions (Kuromiya et al., 2020).
However, there are several limitations to this study. The sample size of learners
observed was relatively small, which might limit the generalizability of the findings. Moreover,it should be noticed that the current predictive model's AUC did not achieve a high level, andthe model needs to be validated using a different student population. Besides behavior
indicators, we plan to incorporate the consistency of the ratings, including the agreement with
instructor-assigned grades and average student-assigned grades (Fukazawa, 2010), into theprior evaluation behavior indicators. Further, considering more predictors in the model, suchas learning outcomes, collaborative skills, and personality variables (Piech et al., 2013;
Sánchez et al., 2021), could also enhance its effectiveness. Qualitative observations and selfreports can offer valuable insights into the reasons behind unserious patterns, and exploring
how the presented EBA estimated from logs connects to the observations is another promising
topic. Lastly, since this research only involved one trial of a group presentation, conducting
additional studies with more rounds of TBL and peer-evaluated group presentations isanticipated to address remaining issues and enhance the robustness of the findings.

6. Conclusion
In conclusion, this study discusses the issue of unserious raters in peer evaluation of grouplearning. We propose a method to describe unserious peer raters by detecting trends basedon the clustering of EBA indicators. The results reveal typical behavior patterns of unserious
raters: straightlining, speeding, and giving all full marks. Next, a preliminary evaluation isconducted for classifiers that can identify groups of unserious raters. The results revealed
typical time and scoring features associated with these raters, as well as predictive indicatorsfor early detection. Overall, these findings have implications for improving the effectivenessand reliability of peer evaluation in group learning contexts. Further investigation is required
to explore the actual quality of ratings and validate the classification model.

Acknowledgements
This research was supported by the following grants: JSPS KAKENHI 20K20131,
20H01722, 22H03902, NEDO JPNP20006, JPNP18013, and JST JPMJSP2110.

References
Amarasinghe, I., Hernández-Leo, D., & Ulrich Hoppe, H. (2021). Deconstructing orchestration load:
comparing teacher support through mirroring and guiding. International Journal of ComputerSupported Collaborative Learning, 16(3), 307-338.
Cho, K., & Schunn, C. D. (2007). Scaffolded writing and rewriting in the discipline: A web-based
reciprocal peer review system. Computers & Education, 48(3), 409-426.
Damgaard, M. T., & Nielsen, H. S. (2018). Nudging in education. Economics of Education Review, 64,
313-342.
Fawcett, T. (2006). An introduction to ROC analysis. Pattern recognition letters, 27(8), 861-874.
Fukazawa, M. (2010). Validity of peer assessment of speech performance. ARELE: Annual Review ofEnglish Language Education in Japan, 21, 181-190.
Gabrilovich, E., & Markovitch, S. (2004, July). Text categorization with many redundant features: Using
aggressive feature selection to make SVMs competitive with C4. 5. In Proceedings of the twentyfirst international conference on Machine learning (p. 41).

162

Goolsarran, N., Hamo, C. E., & Lu, W. H. (2020). Using the jigsaw technique to teach patient
safety. Medical education online, 25(1), 1710325.
Gorham, T., Majumdar, R., & Ogata, H. (2023). Analyzing learner profiles in a microlearning app fortraining language learning peer feedback skills. Journal of Computers in Education, 1-26.
Horikoshi, I., Liang, C., Majumdar, R., & Ogata, H. (2022). Applicability and reproducibility of peer
evaluation behavior analysis across systems and activity contexts. In 30th International
Conference on Computers in Education Conference Proceedings (Vol. 1, pp. 335-345).
Horikoshi, I., & Tamura, Y. (2021). How do students evaluate each other during peer assessments? Ananalysis using “evaluation behavior” log data. Educational technology research, 43(1), 3-21.
Janssen, J., & Kirschner, P. A. (2020). Applying collaborative cognitive load theory to computersupported collaborative learning: Towards a research agenda. Educational Technology Researchand Development, 68(2), 783-805.
Johnson, L. D. (2017). Exploring cloud computing tools to enhance team-based problem solving forchallenging behavior. Topics in Early Childhood Special Education, 37(3), 176-188.
Kim, Y., Dykema, J., Stevenson, J., Black, P., & Moberg, D. P. (2019). Straightlining: Overview ofmeasurement, comparison of indicators, and effects in mail–web mixed-mode surveys. Social
Science Computer Review, 37(2), 214-233.
Kuromiya, H., Majumdar, R., & Ogata, H. (2020). Fostering evidence-based education with learning
analytics. Educational Technology & Society, 23(4), 14-29.
Liang, C., Gorham, T., Horikoshi, I., Majumdar, R., & Ogata, H. (2022a). Estimating peer evaluation
potential by utilizing learner model during group work. In Collaboration Technologies and Social
Computing: 28th International Conference, CollabTech 2022 (pp. 287-294).
Liang, C., Majumdar, R., Horikoshi, I., Flanagan, B. & Ogata, H. (2022b). Exploring predictive indicatorsof reading-based online group work for group formation assistance. In 30th International
Conference on Computers in Education Conference Proceedings (Vol. 1, pp. 642-647).
Liang, C., Majumdar, R. & Ogata, H. (2021). Learning log-based automatic group formation: systemdesign and classroom implementation study. Research and Practice in Technology Enhanced
Learning, 16(1), 1–22.
Michaelsen, L. K., Knight, A. B., & Fink, L. D. (Eds.). (2002). Team-based learning: A transformativeuse of small groups. Greenwood publishing group.
Ogata, H., Yin, C., Oi, M., Okubo, F., Shimada, A., Kojima, K., & Yamada, M. (2015). E-Book-basedlearning analytics in university education. In International conference on computer in education(ICCE 2015) (pp. 401-406).
Ohland, M. W., Loughry, M. L., Woehr, D. J., Bullard, L. G., Felder, R. M., Finelli, C. J., ... &Schmucker, D. G. (2012). The comprehensive assessment of team member effectiveness:
Development of a behaviorally anchored rating scale for self-and peer evaluation. Academy ofManagement Learning & Education, 11(4), 609-630.
Parmelee, D., Michaelsen, L. K., Cook, S., & Hudes, P. D. (2012). Team-based learning: a practicalguide: AMEE guide no. 65. Medical teacher, 34(5), e275-e287.
Patchan, M. M., Schunn, C. D., & Correnti, R. J. (2016). The nature of feedback: How peer feedback
features affect students’ implementation rate and quality of revisions. Journal of Educational
Psychology, 108(8), 1098.
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013). Tuned models of peer assessment
in MOOCs. arXiv preprint arXiv:1307.2579.
Sánchez, O. R., Ordonez, C. A. C., Duque, M. A. R., & Pinto, I. I. B. S. (2021). Homogeneous groupformation in collaborative learning scenarios: An approach based on personality traits and genetic
algorithms. IEEE Transactions on Learning Technologies, 14(4), 486-499.
Strauß, S., & Rummel, N. (2021). Promoting regulation of equal participation in online collaboration by
combining a group awareness tool and adaptive prompts. But does it even matter?. International
Journal of Computer-Supported Collaborative Learning, 16, 67-104.
To, J., & Panadero, E. (2019). Peer assessment effects on the self-assessment process of first-year
undergraduates. Assessment & Evaluation in Higher Education, 44(6), 920-932.
Van Zundert, M., Sluijsmans, D., & Van Merriënboer, J. (2010). Effective peer assessment processes:
Research findings and future directions. Learning and instruction, 20(4), 270-279.
Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience andquestion complexity on web survey response times. Applied Cognitive Psychology: The Official
Journal of the Society for Applied Research in Memory and Cognition, 22(1), 51-68.
Zhang, C., & Conrad, F. (2014, July). Speeding in web surveys: The tendency to answer very fast andits association with straightlining. In Survey research methods (Vol. 8, No. 2, pp. 127-135).

163

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Future City: A Simulation for Making SDGs
Action Decisions
Pin-Chen CHEN*, Yu-Hao LU, Ju-Ling SHIHġ
Graduate Institute of Network Learning Technology, National Central University, Taiwanġ
*daisy12296021@gmail.comġ
Abstract: The Sustainable Development Goals (SDGs) deliver a concept aimed ataddressing social, economic, and environmental challenges that require collaborative
efforts from everyone. The actions often involve conflicts of interest between
businesses, governments, social organizations, and citizens. This study develops an
inquiry-based learning system <Future City> aiming to illustrate SDGs issues. Students
play different roles and simulate future social scenarios based on their positions andaction decisions. This assists students in understanding the concepts of sustainability.
Keywords: SDGs, Simulation, Action Decisions

1. Introduction
To enhance students' awareness of sustainable development, the Taiwanese government has
integrated the United Nations' proposed SDGs into education which involves not only videos
and reading materials but also activities that encourage student participation and reflection.
The effectiveness of these efforts goes beyond mere promotion; they are primarily focused on
education. As a result, this system builds upon the <City Auncel> developed by Lu, Shih, &Hong (2022), and further develops an issue simulation decision system designed to engage
students in various societal roles. Students work within the interests of their respective roles
while also considering the common welfare. This entails finding a balance between ecologicalenvironments and socio-economic development, achieved by resolving conflicts of interest
through effective negotiations. Consequently, this study delves into the impact of SDGs
simulations on learners' understanding of the SDGs and their behavioral decisions.

2. Literature Review
2.1 SDGs-related simulation
The United Nations the SDGs to mitigate climate change and promote sustainable
development on a global scale (Rosati & Faria, 2019). However, establishing partnerships
among businesses, governments, and civil society can be challenging if they have competing
goals (Brinkerhoff & Brinkerhoff, 2011).
Computer simulation is a tool used to approximate real-world results and aid in futureplanning. For example, Collste (2017) utilized the iSDG dynamic model to simulate the annual
effects of solar energy investment and policy interventions on the SDGs of Good Health andWell-Being, Quality Education, and Affordable and Clean Energy. The study investigated thecausal relationship between electricity acquisition, education, and life expectancy, and theresults showed that improved electricity supply was associated with increased educational
attainment and life expectancy. Another simulation system, iplan (Ruis et al., 2023), is basedon urban development actions and uses a complex computational model to simulate land use
scenarios, helping players explore and solve land use problems and cultivate students to makemeaningful actions to address social and environmental issues.
This study focuses the development of a simulation system based on the survival crisisof the Formosan Leopard Cat, which is related to sustainable cities and communities, climate
action, and life on land. In the simulation, students assume different roles, including164

government officials, conservationists, developers, hunters, and farmers, to explore issues
related to urban development and ecological conservation.

2.2 Action Decision
In the simulation, students had to make developmental decisions. During the process, thecurrent social environmental situation is taken into consideration. Students collect information,
evaluate the uncertainty of action consequences, and analyze the pros and cons of various
actions before making the best next possible decision. Different decision models emerge
depending on the tasks at hand, and the decision maker possesses analytical characteristics
considering the variables that appear in the field. Students evaluate the probability of futurestates and target schemes with the focus on the decision event (Klein & Calderwood,1991).
In this study, each player’s decision-making process will be analyzed in regard of the inquirylearning and issue-discussion.

3. System Architecture
The simulation system supports the inquiry process in three stages: Inquiry, Action, and
Negotiation, that are cycled for three rounds (Figure 1).

Figure 1. Simulation process flow chart
The first stage, Inquiry, requires students to select a region for exploration, identify
problems relating to SDGs issues, and provide explanations. The second stage, Action,
students choose a development action, and the system simulates the environmental changes
caused by the action. The third stage, Negotiation, requires students to resolve conflicts ofinterests arise from the developmental actions done by various roles in future cities. All
behaviors in the simulation are synchronized, presented, and recorded in the database.

4. Simulation Design
Roles in the simulation has respective independent values and missions that the collective
action would affect the common geographical values (Table 1).
Table 1. Independent and common missions

Government
Conservationists
Developers
Hunters
Farmers

Missions
Independent
Public opinion>60, Environment > 60,
Public opinion > 60,ġPublic category >=1.
Environmental protection>=2,Environment > 60
Buildings >=2, Money > 1000.
Production>50, Money > 300.
Production>50, Money > 300.

commonDevelopment>60
Environment >60.ġ

There are 20 actions players can choose from, including public, conservation,
construction, hunting, and agriculture (Figure 2). All actions cause corresponding numericalvalue changes. For instance, building an industrial area upstream of a river or in a general165

plain area make different impacts on water pollution. Thus, it is important that the player
investigates the relevant data of an area before taking action. Action labels are added to themap with the identification of respective roles (Figure 3). Players can inspect the impact of
their actions on the environment and adjust their future action decisions.

Figure 2. Action option diagram.

Figure 3. Future City Simulation.

5. Methods
This study plans to use pre-test and post-test as well as questionnaires to understand
students’ attitudes, awareness, and knowledge of SDGs. The collected system data will bestatistically analyzed to understand whether their understanding of the concept andperceptions of sustainable development goals has improved.

6. Expected Results
Utilizing an inquiry-based learning approach, this research empowers students to delve into
real-world data, ensuring that they possess a substantial background knowledge before
making behavioral decisions. The study aims to incorporate SDGs issues through role-play
simulations, investigating students' understanding and attitudes towards addressing SDGs
problems. It also examines their ability to access relevant information and their decisionmaking process in response to simulated future social and environmental changes. The
negotiation processes they engage in while resolving conflicts of interest among different
groups will be analyzed. By scrutinizing the disparities between actual data and simulated data
resulting from behavioral decisions, users can reflect on the environmental impact they havegenerated, which aids in their comprehension of SDGs issues. The simulation system isanticipated to promote the understanding of SDGs issues through an entertaining approachand encourage players to allocate more attention to the topic of sustainable development.

References
Brinkerhoff, D. W., & Brinkerhoff, J. M. (2011). Public–private partnerships: Perspectives on
purposes, publicness, and good governance. Public administration and development,
31(1), 2-14.
Collste, D., Pedercini, M., & Cornell, S. E. (2017). Policy coherence to achieve the SDGs:
using integrated simulation models to assess effective policies. Sustainability science, 12,921-931.
Klein, G. A., & Calderwood, R. (1991). Decision models: Some lessons from the field. IEEE
Transactions on systems, man, and cybernetics, 21(5), 1018-1026.
Lu, Y.H., Shih, J.L., & Hong, G.D. (2022). Issue-based Guided Inquiry Model with Real
Socioscientific Open Data <City Auncel>. The Main Proceedings of the 30th International
Conference on Computers in Education (536-541), Malaysia: Asia-Pacific Society on
Computers in Education.
Rosati, F., & Faria, L. G. (2019). Addressing the SDGs in sustainability reports: The
relationship with institutional factors. Journal of Cleaner Production, 215, 1312-1326.
Ruis, A. R., Barford, C., Brohinsky, J., Tan, Y., Bougie, M., Cai, Z., Lark T. J., & Shaffer,
D.W.(2023) Iplan: A Platform for Constructing Localized. Reduced-Form Models of LandUse Impacts. Retrieved from https://ssrn.com/abstract=4358320

166" !   !" " "   #" !
#" !"$  #" ! #"                    
    !     
#,-/)#*/+$*$+-)/'+**"+))0*'!/'+*#!&*+(+%'#.
*'1#-.'//+),#0 -,'*
#,-/)#*/+$+),0/#-!'#*!#*"#!&*+(+%3*'1#-.'/3+$) -'"%#*'/#"'*%"+)

#,-/)#*/+$.3!&+ '+(+%3*"#/&+"+(+%3+$#(/&!'#*!#.
*./'/0/#+$#0-+.!'#*!#*'1#-.'// 0/4*+)"# -!#(+*,'*
B6G8 7:6G9HA:NJE; :9J






  &A>6A4BB8E4<DB2D;0AA4;0G0C8>=&#'8B0C427=8@D4>5C4=DB438=2;8=820;
B4CC8=6B 5>A A43D28=6 ?7HB820; BH<?C><B >5 0=G84CH 0=3 8<?A>E8=6 F4;;148=6 )78B
?0?4A ?A4B4=CB 0 2><?0A8B>= >5 ?BH27>;>6820; 0=3 ?7HB8>;>6820; 45542CB >5 0= D;CA0
1A845E834>6D8343&#'02C8E8CHD&#'C70C70B144=030?C435>A2;0BBA>><B4CC8=6B
0=30C8<4<0C27432>=CA>;02C8E8CH0=3BCD34=C>?8=8>=B>5D&#'C>60D648CB
0224?C018;8CH5>ADB48=5>A<0;43D20C8>=0;B4CC8=6B>ACHD=34A6A03D0C4BCD34=CB 
H40AB5A><0(?0=8B7D=8E4AB8CH?0AC828?0C438=C74BCD3H8=F7827<40BDA4B>5BC0C4
0=G84CH (C0C4)A08C =G84CH =E4=C>AH () 05542C 5542C8E4 (;834A ( 0=3
?0A0BH<?0C74C82=4AE>DBBHBC4<&$(02C8E8CH8=34G43E80C74740ACA0C4 E0A8018;8CH
?0A0<4C4A A>>C <40= B@D0A4 >5 BD224BB8E4 38554A4=24B '#(( F4A4 2>=3D2C43
'4BD;CBB7>FC70C'#((F0BB86=85820=C;H78674A3DA8=6D&#'C70=F74=<40BDA43
145>A4)74B0<445542CBF4A4=>C5>D=3F8C7338C8>=0;;H?0AC828?0=CBA0C43C74
34B86=>5D&#'<>A450E>A01;H0=30224?C01;45>ADB48=5>A<0;43D20C8>=0;B4CC8=6B
C70=(86=85820=C270=64B8=05542C0=3BC0C40=G84CH0=3BDBC08=43?7HB8>;>6820;
45542CBF4A4=>C34C42C43BD664BC8=6C70CC741A4E8CH>5C7402C8E8CH<0H30<?4=8CB
45542CB)7458=38=6BA4;0C43C>C74?7HB8>;>6820;45542CB0=3BCD34=C>?8=8>=BBD??>ACC74=>C8>=C70CD&#'?>BB4BB4B?A><8B8=6270A02C4A8BC82B5>ADB48=B8CD0C43C40278=6
0=3?A02C824>5B4;5A46D;0C>AHC427=8@D4B08<430CA43D28=6?7HB8>;>6820;0A>DB0;>=
0=02DC410B8B4=248C20=142>=B834A430=0??A>?A80C402C8E8CHC>8=C46A0C48=C>
C427=>;>684B C70C BD??>AC C40274AB 8= 2A40C8=6 >??>ACD=8C84B 5>A B4;5A46D;0C8>= 8= C742;0BBA>><0=36D838=6BCD34=CBC7A>D674E834=2410B43B4;5A46D;0C8>=?A02C824B

  (4;5A46D;0C8>= &7HB8>;>6H (CA4BB 5542C &A>6A4BB8E4 <DB2D;0A
A4;0G0C8>=  
U"AD76AANI=:B:CI6A=:6AI=D;JC>K:GH>INHIJ9:CIH>H6C>BEDGI6CIEJ7A>8=:6AI=>HHJ:V6CI?:H#/( E  6C98DHI :;;:8I>K:HJHI6>C67A:6C9H86A67A:>CI:GK:CI>DCH6G:C::9:9 
HNHI:B6I>8 G:K>:L D; G:K>:L A:K:A :K>9:C8: DC HJEEDGI>C< JC>K:GH>IN HIJ9:CI B:CI6A =:6AI=G:EDGIHDC6L>9:G6C<:D;>CI:GK:CI>DCH -DB:D;I=:>CI:GK:CI>DCHL>I=:K>9:C8:D;EDH>I>K:
:;;:8IH DC HIJ9:CI B:CI6A =:6AI= >C8AJ9: H@>AA IG6>C>C< EGD<G6BH L>I= HJE:GK>H:9 EG68I>8:
HIG:HH B6C6<:B:CI G:A6M6I>DC IG6>C>C< B>C9;JAC:HH 76H:9 >CI:GK:CI>DCH >CI:GK:CI>DCH
9:A>K:G:9K>6I:8=CDAD<N6C9H:II>C<H 76H:9>CI:GK:CI>DCHL=>8=6>BID:C=6C8:6869:B>8
I:68=>C<6C9A:6GC>C<1DGHA:N#/(  -IG:HH >H 6 9G>K:G D; B:CI6A =:6AI= 8=6AA:C<:H  /C>K:GH>IN HIJ9:CIH :ME:G>:C8: =><=
A:K:AH D; HIG:HH L=>8= 6;;:8IH I=:>G =:6AI= 6869:B>8 E:G;DGB6C8: 6C9 ADC<:G I:GB:BEADN67>A>IN+6H8D:#/(    .=:G:>H6C::9ID:9J86I:HIJ9:CIH67DJII=::;;:8IHD;
HIG:HH6C9ID>CIGD9J8:I=:BID:;;>8>:CI8DE>C<HIG6I:<>:H7DI=IDHJEEDGIHIJ9:CI6869:B>8

167

E:G;DGB6C8:6C9L:AA 7:>C<0D<:A-8=L67:  ,:A6I:9AN:BDI>DC6AH:A; G:<JA6I>DC
9:;>C:96HI=:67>A>INID>C;AJ:C8:DJGDLC:BDI>DCHDGI=DH:D;DI=:GH>H6C>BEDGI6CIH:A;
B6C6<:B:CIH@>AA $I>CKDAK:HG:8D<C>O>C<DC:WH:BDI>DC6AHI6I:H:A:8I>C<6C9>BEA:B:CI>C<
6G:<JA6IDGNHIG6I:<NIDBD9>;N>I6C9BDC>IDG>C<;DGHJ88:HH>C68=>:K>C<I=:G:<JA6IDGN<D6A
(8,6:  "GDHH     -:A; G:<JA6I>DC HIG6I:<>:H HJ8= 6H B>C9;JA 7G:6I=>C< EGD<G:HH>K:
BJH8JA6G G:A6M6I>DC +(, 6C9 G:6EEG6>H6A 86C 9:8G:6H: HNBEIDBH D; 6CM>:IN 6C9 9DLC
G:<JA6I:C:<6I>K::BDI>DCH1DGHA:N#/( -8JAI#/(  
J>A9>C<JEDCI=:>9:6I=6I;DGB6A:9J86I>DC6AH:II>C<H86CEGDK>9:HIJ9:CIHL>I=H6;:
6C9H>IJ6I:9DEEDGIJC>I>:H;DGA:6GC>C<6C9EG68I>8>C<H:A; G:<JA6I>DCI=>HLDG@8DCIG>7JI:HID
:;;DGIHI=6IJH:I:8=CDAD<NID>C8G:6H::9J86IDG688:HHIDG:HDJG8:H6C9I:8=C>FJ:HI=6I9D
CDIG:FJ>G:HE:8>6A>O:9IG6>C>C<IDI:68=> : I=:N86C7:9:A>K:G:97N6CDC :ME:GI9DCDI
I6@:6ADC<E:G>D9D;I>B:ID9:BDCHIG6I:> : I=:N86C7:>CI:<G6I:9>CID6G:<JA6G8A6HH
6C986C=:AE:9J86IDGH7:II:GEGDK>9:<GDJEHD;HIJ9:CIHDEEDGIJC>I>:HID>BEGDK:67>A>I>:HID
H:A; G:<JA6I: ADC<I=:H:A>C:H69><>I6AI:8=CDAD<N;DGI:68=>C<6C9A:6GC>C<L6H9:K:ADE:9
IDHJEEDGII:68=:GH>C8G:6I>C<DEEDGIJC>I>:H;DGH:A; G:<JA6I>DC>CI=:8A6HHGDDB6C9<J>9>C<
HIJ9:CIHI=GDJ<=:K>9:C8: 76H:9H:A; G:<JA6I>DCEG68I>8:H .=:6EEA>86I>DCA6HH(DD9EE>CKDAK:HHIJ9:CIHG:EDGI>C<I=:>G8JGG:CI:BDI>DC6AHI6I:6C976H:9DC6C6<<G:<6I:HI6I:D;
I=: 8A6HH I=: I:68=:G H:A:8IH 6C9 GJCH 6C 68I>K>IN ID 6AI:G I=: :BDI>DC6A HI6I: D; I=: 8A6HH
:6G9HA:N#/(  A6HH(DD9EE6>BHID=:AEHIJ9:CIH7:8DB:6L6G:D;I=:>GDLC
:BDI>DCH6C9:ME:G>:C8:68I>DCHI=:N86CI6@:IDG:<JA6I:I=:B6C96AHDID7:8DB:6L6G:
D; I=: :BDI>DC6A HI6I:H D; I=:>G E::GH 6C9 I=: DK:G6AA 8A6HH T 6H I=: 8A6HHGDDB :BDI>DC6A
8A>B6I:86C6;;:8I8A6HHE:G;DGB6C8:G68@:II#/( .D>C8G:6H:688:HH>7>A>INJAIG6
7G>:;ID B>CK>9:D <J>9:968I>K>I>:H=6K:7::C>CI:<G6I:9>CIDI=:6EEA>86I>DC *7G69DK>Y:I 6A     ;DJC9 :K>9:C8: I=6I 6C JAIG6 7G>:;  B>C K>9:D <J>9:9 9::E 7G:6I=>C<>CI:GK:CI>DC:;;:8I>K:ANG:9J8:98=>A9G:CWHE=NH>DAD<>86A6GDJH6A6C9L6H6C688:HH>7A:IDDA
I=6I8DJA97:JH:9>C=DB:H8=DDADGDCA>C:H:II>C<H 8I>K>I>:H6K6>A67A:>CA6HH(DD9EE
>C8AJ9: :K>9:C8: 76H:9 I:8=C>FJ:H HJ8= 6H B>C9;JAC:HH 6C9 7G:6I=LDG@  #DL:K:G I=:
EHN8=DAD<>86A 6C9 E=NH>DAD<>86A :;;:8IH D; JAIG6 7G>:; K6G>6I>DCH D; I=:H: :K>9:C8: 76H:9
I:8=C>FJ:H>HJC8A:6G .=>HLDG@6>BHID>CK:HI><6I:I=::;;:8IHD;6CJAIG6 7G>:;K>9:D <J>9:9
EG68I>8:D;EGD<G:HH>K:BJH8JA6GG:A6M6I>DCJ+(, +(,>H6HIG:HHB6C6<:B:CII:8=C>FJ:D;I:CJH:9>C8A>C>86AH:II>C<HI=6I>CKDAK:HI=:
G:E:6I:9 8DCIG68I>C< 6C9 G:A6M>C< D; 9>;;:G:CI BJH8A: <GDJEH  $I =6H 6AHD 7::C JH:9 ;DG
G:9J8>C<E=NH>86AHNBEIDBHD;6CM>:IN>CI=:<:C:G6AEDEJA6I>DC&A6C>C 2D76H#/( >C8=>A9G:C06<CDA>#/( 6C96BDC<JC>K:GH>INHIJ9:CIH"6AA:<D "PB:O#/(  +(,LDG@H7NEGDBDI>C<69::EHI6I:D;G:A6M6I>DC68I>K6I>C<I=:E6G6HNBE6I=:I>8HNHI:B
-6@6@>76G6 #/ (  9:8G:6H>C< 8DGI>HDA A:K:AH "6AA:<D "PB:O #/ (   6C9
9:8G:6H>C<7G6>C68I>K>IN&D76N6H=>&D>I676H=>  INE>86A+(,>CI:GK:CI>DC8DCH>HIH
D; BJAI>EA:  B>C H:HH>DCH 68GDHH   96NH 1>A8ONZH@6 #/ (    -:K:G6A HIJ9>:H D;
+(,=6K:7::CGJC>CKDAK>C<JC>K:GH>INHIJ9:CIH -8=>:CA:6C9/C<:G  HIJ9>:96 
96N+(,8DJGH:;DGJC>K:GH>INHIJ9:CIH>CL=>8=E6GI>8>E6CIHL:G:IDEG68I>8:+(,:68=96N +6GI>8>E6CIHG:EDGI:9ADL:GA:K:AHD;6GDJH6ABDG:EDH>I>K:K6A:C8:=><=:GG:A6M6I>DCA:K:AH
6;I:GI=:>G96>AN+(,EG68I>8:6C9ADL:GHIG:HHA:K:AH6II=::C9D;I=:8DJGH: +6LADL%DC:H G6C6H>C<A: ID B>C+(,EG68I>8:6C98DBE6G:9I=::;;:8IHID6CD IG:6IB:CI
8DCIGDA  .=: 6JI=DGH ;DJC9 I=6I +(, EG68I>8: ADL:G:9 A:K:AH D; =:6GI G6I: HI6I: 6CM>:IN
E:G8:>K:9HIG:HH6C9H6A>K6GN8DGI>HDA .=:6JI=DGHDA7>:G,JH= D;6HIJ9NDC6
H>C<A: B>CAN>C<9DLC+(,EG68I>8:L>I==><=HIG:HHJC>K:GH>INHIJ9:CIH8DC8AJ9:9I=6I
+(,86C9:8G:6H:8D<C>I>K:6CM>:IN>C8G:6H:B:CI6A6C9E=NH>86AG:A6M6I>DC6C9:C=6C8:
G:HIDG6I>K: E=NH>DAD<>86A ;JC8I>DC>C<  .=: A6II:G >H G:EG:H:CI:9 7N 6C >C8G:6H: >C
E6G6HNBE6I=:I>8 C:GKDJH HNHI:B +)- 68I>K>IN L=>8= >C IJGC >H >C9>86I:9 7N 8=6C<:H >C=:6GI G6I:K6G>67>A>IN#,0 DA7>:G6C9,JH= E  LG>I:I=6I#,0G:;:GHID8=6C<:H
>CI=:I>B:>CI:GK6AH7:IL::C8DCH:8JI>K:=:6GI7:6IHL>I=<G:6I:G;AJ8IJ6I>DC>C9>86I>C<+)-
68I>K>INL=:G:6HADL:G;AJ8IJ6I>DC>C9>86I:HHNBE6I=:I>8C:GKDJHHNHI:B68I>K>IN .=:A6II:G
HIJ9>:HHJ<<:HI6H>C<A:677G:K>6I:9EG68I>8:D;+(,86C=6K:EDH>I>K:EHN8=DAD<>86A6C9
E=NH>DAD<>86A:;;:8IHDCJC>K:GH>INHIJ9:CIH 
A><C>C< L>I= H:II>C<H 76H:9 >CI:GK:CI>DCH I=6I D;;:G H@>AA IG6>C>C< G:A6I:9 ID HIG:HH
B6C6<:B:CIL>I=I=:HJEEDGID;I:8=CDAD<NI=>HEG:H:CIHIJ9N>CK:HI><6I:HL=:I=:GJ+(,>H

168

688:EI67A:;DGI=:H>IJ6I:9I:68=>C<6C9EG68I>8:D;H:A; G:<JA6I>DC>C;DGB6A:9J86I>DC .D9D
HD I=: EHN8=DAD<>86A 6C9 E=NH>DAD<>86A :;;:8IH D; J+(, DC HIJ9:CI 6CM>:IN 6;;:8I 6C9
E6G6HNBE6I=:I>8C:GKDJHHNHI:B+)-68I>K>IN>C9:M:9K>6I=:#,0E6G6B:I:GGDDIB:6C
HFJ6G:D;HJ88:HH>K:9>;;:G:C8:H,(--6G:6HH:HH:9>C8DBE6G>HDCID6I>B: B6I8=:9
8DCIGDA 68I>K>IN   ,(-- >H I=DJ<=I ID G:;A:8I 86G9>68 K6<6A 68I>K>IN L=>8= 9:E>8IH +)-
8DCIG>7JI>DCID86G9>68G:<JA6I>DC'67DG9:#/(  ,(-->HL>9:ANJH:9>C:BDI>DC
G:<JA6I>DC HIJ9>:H 6AO6GDII> #/ (   (DG:DK:G HIJ9:CI DE>C>DCH D; J+(, 6G: 6AHD
EG:H:CI:9ID6HH:HHHIJ9:CI688:EI6C8:D;I=:68I>K>IN #:C8:I=:G:6G:I=G::=NEDI=:H:H
7:>C<>CK:HI><6I:9>CI=>HE6E:G # J+(,L>AAA:69IDBDG:7:C:;>8>6AE=NH>DAD<>86A:;;:8IH
I=6C6I>B: B6I8=:98DCIGDA68I>K>IN6H:K>9:C8:97N6<G:6I:G>C8G:6H:>C#,0,(--;GDB 6 8DBE6G>HDC D; 76H:A>C: 6C9 68I>K>IN E:G;DGB6C8: E:G>D9H +6LADL  %DC:H  )>8@:A#/( DA7>:G,JH=  .=:9>;;:G:C8:>C#,0G:;A:8IH=><=:GEDHI EG:+)-
68I>K>IN #J+(,L>AAA:69IDBDG:7:C:;>8>6AEHN8=DAD<>86A:;;:8IHI=6C6H:K>9:C8:9
7NADL:GEG: EDHIG6I>C<HD;6GDJH6A6C9=><=:GK6A:C8:-8=>:CA:/C<:G  6C9ADL:G
EG: EDHIG6I>C<HD;HI6I:6CM>:IN+6LADL%DC:H DA7>:G,JH=  #/C>K:GH>IN
HIJ9:CIHL>AAG6I:J+(,6HBDG:688:EI67A:;DGJH:>C;DGB6A:9J86I>DC6AH:II>C<HI=6C6I>B:
B6I8=:98DCIGDA68I>K>INI=6I>CKDAK:HL6I8=>C<6K>9:D9D8JB:CI6GN " $%' #.=:EGDID8DA>H68DC8:EIJ6AG:EA>86I>DCD;I=:LDG@7N'67DG9::I6A  I=6I8DBE6G:9
I=::;;:8IHD;K6G>6I>DCHD;6E68:9 7G:6I=>C<68I>K>INJH>C<E=NH>DAD<>86A#,0,(--6C9
EHN8=DAD<>86A B:6HJG:H 6CM>:IN 6GDJH6A 6C9 K6A:C8:  +:G>D9H D; G:HI G:68I>K>IN 6C9
G:8DK:GN;DAADL>C<I=:,HD;#,0'67DG9:#/( JC9:GA>:I=:HIGJ8IJG:D;I=:EGDID8DA ,:HI >H I=: 76H:A>C: G:68I>K>IN >H I=: 8=6C<: 7:IL::C 76H:A>C: 6C9 I=: :K:CI > :  H:A;
G:<JA6IDGN68I>K>IN6C9G:8DK:GN>HI=:8=6C<:7:IL::CI=::K:CI6C96EDHI :K:CIB:6HJG: (D9>;>86I>DCHIDI=:G:EA>86I:9EGDID8DA>C8AJ9:I=:JH:D;606C>AA66H:A>C:0IDG:EA68:
I=:G:HI>C<;DG8:9G:A6M6I>DC> : 6HI=:G:8DK:GNI6H@ !DG0E6GI>8>E6CIHE:G;DGB6I6H@
G:FJ>G>C<HJHI6>C:96II:CI>DC7JIB>C>B6A8D<C>I>K:AD69%:CC>C<H#/( ID7:II:GG:;A:8I
6HIJ9:CIHW8A6HHGDDB:ME:G>:C8: !DGI=>HHIJ9NE6GI>8>E6CIHL:G:6H@:9ID8DJCII=:CJB7:G
D; I>B:H 9>;;:G:CI 8DADGH 6EE:6G:9 DC 6 H8G::C  .=: 8DADG 8=6C<:9 :K:GN  H 6C9 I=: I6H@
A6HI:9B>CJI:H !JGI=:GK>9:DHL:G:JH:9ID<J>9:E6GI>8>E6CIHI=GDJ<=I=:H:A; G:<JA6I>DC
68I>K>I>:H .=>H;DAADLHI=:6EEGD68=D;*7G69DK>Y:I6A   IDB>B>8=DL:9J86IDGHLDJA9
JH:B6I:G>6AHI=6I=6K:7::CEGD9J8:9>C6H86A67A:B6CC:G> : H=DL>C<HIJ9:CIH6K>9:D
696EI:9;DG8A6HHGDDBJH:  "$!$#
.=:G:L:G: E6GI>8>E6CIH;:B6A: B6A:;GDB IDN:6GHDA9   R  +HN8=DE=NH>DAD<>86A96I6;GDBE6GI>8>E6CIHL:G::M8AJ9:99J:IDI:8=C>86A>HHJ:HG:A6I:9ID
:A:8IGD9:BDK:B:CIDCH><C6A>CI:<G>IN6C9B>HJC9:GHI6C9>C<D;H:A; G:EDGIB:6HJG:H  .=:;>C6AH6BEA:;DGI=:EHN8=DE=NH>DAD<>86A96I6L6HE6GI>8>E6CIH ;:B6A: B6A:
;GDB IDN:6GHDA9  R  AAE6GI>8>E6CIHL:G:8DBE:CH6I:9 :JGDH;DG
I=:>GE6GI>8>E6I>DC6C9<6K:LG>II:C>C;DGB:98DCH:CI  $"##%"#
$CI=>HL>I=>C HJ7?:8I9:H><CE6GI>8>E6CIH?D>C:9ILD =DJGA67DG6IDGNH:HH>DCH .=:H:HH>DCH;DAADL:9 I=: EGDID8DA H=DLC >C !><JG:  >C L=>8= ILD H:A; G:<JA6I>DC 68I>K>I>:H J+(, DG
6AI:GC6I>K:H6C9DC:8DCIGDA68I>K>INL:G:E:G;DGB:9  68=68I>K>INA6HI:9B>C6C9I=:G:L6H
6 B>C7G:6@7:IL::CI=:E:G;DGB6C8:D;68I>K>I>:H .D8DCIGDA;DG86GGNDK:G:;;:8IH7:IL::C

169

68I>K>I>:HI=:H:HH>DCHIDD@EA68:DCH:E6G6I:96NH .=:8JGG:CIHIJ9NEG:H:CIH68DBE6G>HDCD;DC:D;I=:H:A; G:<JA6I>DC68I>K>I>:HJ+(,6C96C:JIG6A8DCIGDA68I>K>IN 

'%0-#.=: ME:G>B:CI6A+GDID8DA (/- -'#$-+%-#..'1#0.!0(-#(2/'+* !/'1'/30
 EJ7A>8AN 6K6>A67A: K>9:D =IIEH NDJIJ 7: .9;F7 (< I=6I L6H 8G:6I:9 >C 6
BJAI>9>H8>EA>C6GN JGDE:6C EGD?:8I L6H JH:9 ID <J>9: E6GI>8>E6CIH I=GDJ<= I=:  B>C J+(,68I>K>IN  $C I=: K>9:D 6C 6C>B6I:9 8=6G68I:G A:69H K>:L:GH I=GDJ<= 6 8N8A: D; I:CH>C< 6C9
G:A:6H>C<BJH8A:HHI6GI>C<;GDBI=:;::II=><=H6C9=6C9HIDI=:H=DJA9:GH6C9;68: .=:
K>9:DL6H:9>I:9IDDB>II=:>CIGD9J8I>DCID+(,6C9DCAN>C8AJ9:96 B>C8A>EID<J>9:I=:
E:G;DGB6C8:D;+(,  +*/-+( !/'1'/3  B>C8A>E;GDB69D8JB:CI6GNU7:CI:J:G!DGH8=JC<V4,:H:6G8=9K:CIJG:H5DCDJI:G
HE68: L6H H=DLC ID E6GI>8>E6CIH 6H I=: 8DCIGDA 68I>K>IN  .=>H K>9:D L6H ;DJC9 ID =6K:
:BDI>DC6AAN C:JIG6A G6I>C<H 6C9 L6H JH:9 >C 6 EG:K>DJH HIJ9N DC I=: 6JIDCDB>8 :;;:8IH D;
7G:6I=>C<:M:G8>H:H2DJ#/(   " %"
JG>C<I=:L:A8DB:E=6H:E6GI>8>E6CIHL:G:>CIGD9J8:9IDI=:H:FJ:C8:D;68I>K>I>:HH::!><  .G>6AI6H@H6C968I>K>I>:HL:G:GJCDC+HN8=D+NK   6C9H=DLCIDE6GI>8>E6CIHDC
6BDC>IDG HE6GI>8>E6CIH8DBEA:I:96HJGK:N8DAA:8I>C<9:BD<G6E=>8>C;DGB6I>DC6C996I6
G:A6I:9IDEDHH>7A:8DC;DJC9H'67DG9:#/( H:CHDGHL:G:EA68:9DCI=:BH:CHDG
6II68=B:CI E=6H:  JG>C< I=: ;6B>A>6G>O6I>DC E=6H: I=: H:A; G:EDGI H86A:H -.$ - -G:HI>C<I6H@0I6H@6C9H:A; G:<JA6I>DC68I>K>I>:HL:G:>CIGD9J8:96C9E6GI>8>E6CIHL:G:67A:
ID6H@FJ:HI>DCHID8A6G>;NI6H@H !DGI=:G:HI>C<I6H@E6GI>8>E6CIHL:G:>CHIGJ8I:9IDADD@6I6
7A68@H8G::CL>I=6;>M:9L=>I:8GDHH6I>IH8:CI:G+6G8=B:CI#/(  +GD<G:HH>DCD;I=:
IG>6A;DGI=:H:A; G:EDGIFJ:HI>DCC6>G:HL6H8DCIGDAA:97NI=:E6GI>8>E6CIL=:G:6HI=:G:HI>C<
I6H@0I6H@6C968I>K>I>:H69K6C8:96JIDB6I>86AAN76H:9DCEG: H:II>B:H .=:IG>6A:C9:9
L>I=69:7G>:;>C<E=6H:>CL=>8=E6GI>8>E6CIH8DBEA:I:96HJGK:ND;I=:>GK>:LHD;I=:68I>K>IN
I=: H:CHDGH L:G: G:BDK:9 6C9 9:7G>:;>C< D88JGG:9  I=>86A 6EEGDK6A L6H G:8:>K:9 ;DG I=:696EI:9EGDID8DA  #,#*"#*/-' (#.
JGG:CIHI6I:D;6CM>:INL6HB:6HJG:9JH>C<6 >I:BH=DGI;DGBD;I=:-I6I: .G6>ICM>:IN
$CK:CIDGN -.$ - (6GI:6J  :@@:G   -IJ9:CI 6;;:8I L6H B:6HJG:9 JH>C< I=:;;:8I>K:-A>9:G-L=>8=8DCH>HIHD;68DCI>CJDJHH86A:L>I=>8DC>B6<:HDC:>I=:G:C9ID
B:6HJG:A:K:AHD;:BDI>DC6A6GDJH6AU#DL68I>K:9DNDJ;::AV6C9:BDI>DC6AK6A:C8:U1=6I
>H NDJG BDD9V :I:AA6  0:GH8=JG:    #,0 L6H 8DBEJI:9 ;GDB I=: G6L:A:8IGD86G9>D<G6B  " H><C6A 68FJ>G:9 ;GDB 6 L:6G67A: ;DG 86EIJG>C< E=NH>DAD<>86A 96I6
9JG>C<I=:E:G;DGB6C8:D;;JAA 7D9NBDK:B:CIH-6N>H#/(  .=:L:6G67A:G:8DG9HI=:

170

"H><C6AK>6:A:8IGD9:HDCI=:8=:HIDCI=:H=DJA9:GH6C9 EA68:9DCI=:C:8@ 6I6L6H68FJ>G:96I6H6BEA>C<G6I:D; #O C:K:CI 76H:9HNC8=GDC>O6I>DC6EEGD68=
L6H;DAADL:9IDHNC8=GDC>O:96I6HDJG8:H6CC68=#/(   +./-'(0-1#3

+6GI>8>E6CIH8DBEA:I:96"DD<A:;DGBHJGK:NL>I=8ADH:9 :C9:96C9DE:C :C9:9FJ:HI>DCH
6II=::C9D;:68=A67DG6IDGNH:HH>DC .=>HE6E:GG:EDGIH6C6C6ANH>HD;I=:8ADH:9FJ:HI>DC
6 ED>CI'>@:GIH86A: HIGDC<AN9>H6<G::HIGDC<AN6<G::I=6I<6I=:G:9HIJ9:CIDE>C>DCH
DC >I:BH;DG:68=68I>K>INH::.67A: AHD6C6C6ANH>HD;6CDE:C :C9:9FJ:HI>DCG:A6I:9
IDI=:+(,68I>K>INI=6I6H@:9E6GI>8>E6CIHIDHE:8>;N>;I=:G:L6H6CNI=>C<EG:K:CI>C<I=:B;GDBG:A6M>C<9JG>C<I=:68I>K>IN>HEG:H:CI:9  $ '##.=:JH:D;E=NH>DAD<>86A#,06C9H:A; G:EDGIB:6HJG:H--.$ -;DAADLHI=:6EEGD68=
D; '67DG9: #/ (    " H><C6A 96I6 L:G: >BEDGI:9 >CID &J7>DH K:GH>DC   6C9
6C6ANO:9B6CJ6AAN;DG6GI>;68IH !:6IJG:HL:G::MIG68I:9>CI>B:6C9;G:FJ:C8N9DB6>CH6C9
CDCA>C:6G >C9>8:H >C8AJ9:9 >C &J7>DH  =:8@H ;DG CDGB6A>IN 6C9 DJIA>:GH L:G: 9DC:  .=:,(--96I6L:G:CDC CDGB6AAN9>HIG>7JI:9HD6AD< IG6CH;DGB6I>DCL6H6EEA>:9AC,(--
8DCH>HI:CIL>I=#,0G:H:6G8='67DG9:#/(  !DG-K6A:C8:6C96GDJH6A96I6L:G:
BDHIAN CDGB6AAN 9>HIG>7JI:9  #DL:K:G  DJIA>:G L6H G:BDK:9 ;GDB K6A:C8: G6I>C<H  -.$ -
96I6L:G:CDC CDGB6AAN9>HIG>7JI:9HDAD< IG6CH;DGB6I>DCAC-.$ -L6H6EEA>:9 H:G>:HD;
G:E:6I:9 B:6HJG:H )*0 L>I= "G::C=DJH:T":>HH:G 8DGG:8I>DC L:G: 8DC9J8I:9 L>I=
8DC9>I>DCH J+(, KH   6C9 I>B: +,  /,$)" +*-. H:I 6H >C9:E:C9:CI K6G>67A:H #,096I6;DGI=:;JAA B>C9JG6I>DCD;:68=E:G>D9+, /,$)"+*-.L6HJH:9 06A:C8:
6GDJH6AA:K:AD;6CM>:INL:G:H:A; G:EDGI9:E:C9:CIK6G>67A:H6C9AC,(--L6H6C#,0
9:E:C9:CI K6G>67A:  9?JHIB:CIH ;DG BJAI>EA: 8DBE6G>HDCH L:G: 9DC: L>I= DC;:GGDC>
8DGG:8I>DC :H8G>EI>K:HI6I>HI>8HL:G:6AHDJH:9;DGI=:HJGK:N96I6 E6>G:9 H6BEA:HI I:HI
L6HJH:9ID9:I:GB>C:L=:I=:GI=:G:L6H6HI6I>HI>86AANH><C>;>86CIB:6C9>;;:G:C8:>CK>:LH
IDL6G968I>K>I>:H -I6I>HI>86A6C6ANH:HL:G:8DBEJI:9JH>C<-+--K    
:H8G>EI>K: HI6I>HI>8H 6G: EG:H:CI:9 >C !><   ;DG $C,(-- .67A:  ;DG - 6GDJH6A 6C9
K6A:C8: 6C9 $C-.$ -  $C 6AA ;><JG:H  G:EG:H:CIH , S  G:EG:H:CIH , S 
G:EG:H:CIH,S  '#   $# %

!"# $      G:E:6I:9 B:6HJG:H )*0 8DC;>GB:9 I=6I I=:G: L6H CD DK:G6AA
H><C>;>86CI9>;;:G:C8:7:IL::CI=:ILD8DC9>I>DCH  ,   #DL:K:GI=:G:
L6H 6 H><C>;>86CI >CI:G68I>DC :;;:8I ;DG 8DC9>I>DC X I>B:        , +6>GL>H: 8DBE6G>HDCH G:K:6A:9 =><=:G A:K:AH D; $C,(-- 9JG>C< I=: J+(, 8DC9>I>DC
8DBE6G:9 ID I=: A:K:AH G:<>HI:G:9 9JG>C< I=:  ,   (DG:DK:G CD HI6I>HI>86AAN
H><C>;>86CI 9>;;:G:C8:H 7:IL::C 8DC9>I>DCH >C $C,(-- A:K:AH 9JG>C< +,  ,   6C9+*-., L:G:;DJC9 $CI:G:HI>C<ANE6>GL>H:8DBE6G>HDCH>CJ+(,8DC9>I>DCH=DL:9
H><C>;>86CIAN =><=:G A:K:AH >C $C,(-- 9JG>C< I=: H:HH>DC 8DBE6G:9 ID 76H:A>C: A:K:AH
B:6HJG:9 EG>DG ,   ):K:GI=:A:HH CD H><C>;>86CI 9>;;:G:C8:H >C $C,(-- A:K:AH
8DBE6G>C< +,  6C9 +*-. B:6HJG:H >C J+(, L:G: D7H:GK:9 ,     .=:H: G:HJAIH
HJ<<:HII=6IJ+(,>C8G:6H:H$C,(--A:K:AH8DBE6G:9IDI=:L=>A:I=:68I>K>IN>H7:>C<
E:G;DGB:9=DL:K:GI=>H>C8G:6H:9D:HCDI6EE:6GIDH=DLADC< A6HI>C<:;;:8IH *JG;>C9>C<H9>;;:G;GDBI=DH:D;)>8@:A:I6A  6HI=:6JI=DGH;DJC9I=6I,(--
K6AJ:H L:G: =><=:G EDHI 68I>K>IN ;DAADL>C< 6  B>C +(, EG68I>8: >C 6 8A>C>86A EDEJA6I>DC 

171

#DL:K:GI=:E6II:GCD;G:HJAIH;DJC9>CDJGHIJ9N6G:H>B>A6GIDI=:DC:HD7H:GK:9>C2DJ:I
6A    L=:G: I=: >C8G:6H: >C $C,(-- L6H 6AHD ;DJC9 9JG>C< 6 7G>:;   B>C K>9:D<J>9:9E68:9 7G:6I=>C<68I>K>IN7JICDIB6>CI6>C:96;I:GI=:>CI:GK:CI>DC *I=:GHIJ9>:H9>9
CDIH=DL8=6C<:H>CE=NH>DAD<>86AB:6HJG:HHJ8=6H=:6GIG6I:#! #,0DGG:HE>G6I>DC6;I:G6 7G>:; +(, IG6>C>C< -B>I=  )DGB6C    *JG G:HJAIH 8DC;>GB 6 H><C>;>86CI >C8G:6H: >C+)-68I>K>IN6HHD8>6I:9L>I=I=:G:A6M6I>DCG:HEDCH:9JG>C<J+(,E:G;DGB6C8:#D;;B6C#/(  ):K:GI=:A:HH86G9>68K6<6A68I>K>ING:8DK:G:9ID76H:A>C:6;I:GEG68I>8: EDHH>7A:
G:6HDC8DJA97:I=6IE6GI>8>E6CIHBDK:>BB:9>6I:AN6;I:GI=:68I>K>INID6CHL:GI=:H:A; G:EDGIHJGK:NH86JH>C<+)-68I>K>INID9:8G:6H: BDG:A>@:AN:MEA6C6I>DC>HI=6II=:>CI:GK:CI>DC
I>B: L6H IDD 7G>:; ID =6K: 6 ADC<:G A6HI>C< >BE68I DC +)- 68I>K>IN  (DG:DK:G BJH8A:
G:A6M6I>DC>C<:C:G6AB><=I=6K:<G:6I:GBJH8JA6G:;;:8IH6C9HB6AA:G6JIDCDB>8:;;:8IHI=6C
DI=:GG:A6M6I>DCB:I=D9HHJ8=6H;>C<:GI:BE:G6IJG:7>D;::9768@6C9 DG6JID<:C>8IG6>C>C<':=G:G#/(  

'%0-#:H8G>EI>K:-I6I>HI>8HAC,(--  #'   $# %

!"# $  ,:<6G9>C<:BDI>DC6A6GDJH6AI=:G:L6HCDH><C>;>86CIB6>C:;;:8ID;8DC9>I>DC  E  CD>CI:G68I>DC:;;:8I;DG8DC9>I>DCXI>B:    ,    1=>A: J+(, /,$)" 6C9 +*-. K6AJ:H L:G: ADL:G I=6C +,  A:K:AH I=:9>;;:G:C8:HL:G:CDIH><C>;>86CIL=:G:6H+*-.K6AJ:HL:G:H><C>;>86CIANADL:GI=6C+, 
A:K:AH ,   AHD 6 H><C>;>86CI G:9J8I>DC >C H:A; G6I>C<H D; :BDI>DC6A 6GDJH6A ;GDB/,$)">BB:9>6I:AN6;I:GI=::C9D;I=:68I>K>INE:G;DGB6C8:ID+*-.6;I:G6 B>CG:HI
E:G>D9L6HD7H:GK:9>C7DI=J+(,, 6C9,
 ,:<6G9>C<:BDI>DC6AK6A:C8:I=:G:L6HCDH><C>;>86CIB6>C:;;:8ID;8DC9>I>DC  ,   .=:G:
L6H 6 H><C>;>86CI >CI:G68I>DC :;;:8I ;DG 8DC9>I>DC X I>B:         ,  +6>GL>H: 8DBE6G>HDCH G:K:6A:9 I=6I +,  K6A:C8: A:K:AH 9>;;:G:9 7:IL::C 8DC9>I>DCH ,  )DH><C>;>86CI9>;;:G:C8:>CK6A:C8:A:K:AHL6H;DJC9>C/,$)", 6C9+*-.
, 
 A:K:AH  ">K:C I=: ADL:G K6A:C8: 76H:A>C: +,  >C J+(, G:HJAIH HJ<<:HI I=6I
J+(, =69 6 BDG: EDH>I>K: :;;:8I DC K6A:C8:  $C DI=:G LDG9H J+(, B6N =6K: 7GDJ<=I
E6GI>8>E6CIHID6=><=:GK6A:C8:HI6I:;GDB+, ID/,$)", L=>A:LDG@:9>C
I=:DEEDH>I:9>G:8I>DCL>I=C:<6I>K::;;:8IHDCK6A:C8:;GDB+, ID/,$)",  6C9/,$)" ID +*-. ,   .=:  L6H 6 CDC HI>BJA6I>C< 68I>K>IN I=6I 8DCH>HI:9 D;
L6I8=>C<6K>9:D6C9G:B6>C>C<E6HH>K: .=>H8DJA97:I=:G:6HDCE6GI>8>E6CIA:K:AHD;6GDJH6A
9:8G:6H:9 6C9 I=:N HL>I8=:9 ID 6 BDG: C:<6I>K: K6A:C8: HI6I: > :  >C8G:6H:9 7DG:9DB 

172

1=>A:8=6C<:H;GDB76H:A>C:+, ID>BB:9>6I:AN6;I:G68I>K>INE:G;DGB6C8:/,$)"9>9
HJ<<:HI=><=:G6GDJH6A6C9BDG:EDH>I>K:6;;:8I>CI=:J+(,8DC9>I>DCI=:H:8=6C<:HL:G:
CDI H><C>;>86CI  .=JH DJG ;>C9>C<H 9>;;:G ;GDB I=DH: D; -8=>:CA: 6C9 /C<:G    >C L=>8=
JC>K:GH>INHIJ9:CIHG:EDGI:9H><C>;>86CIANADL:G6GDJH6A6C9BDG:EDH>I>K:6;;:8I6;I:G96>AN
B>C+(,:M:G8>H:H EDHH>7A::MEA6C6I>DC>HI=6IE6GI>8>E6CI;6B>A>6G>IN>CE:G;DGB>C<+(,6C968I>K>IN9JG6I>DC6G:>BEDGI6CI;68IDGHG:A6I:9ID<:C:G6I>C<8=6C<:H>C6;;:8I>K:HI6I:H    " # .=:G:L6HCDH><C>;>86CIB6>C:;;:8ID;8DC9>I>DC 
, CD>CI:G68I>DC:;;:8I;DG8DC9>I>DCXI>B:   ,  
+6>GL>H:8DBE6G>HDCHG:K:6A:9I=6IDCANJ+(,H=DL:96H><C>;>86CIG:9J8I>DC>CH:A; G6I>C<H
D;HI6I:6CM>:IN;GDB+, ID+*-.,  -><C>;>86CIG:9J8I>DCH>CHI6I:6CM>:IN;GDB/,$)"ID+*-.L:G:;DJC9>C7DI=J+(,, 6C9, .=JHA:K:AHD;
6CM>:IN 9>9 CDI 9>;;:G ;GDB 76H:A>C: ID I=: B:6HJG: >BB:9>6I:AN 6;I:G J+(, E:G;DGB6C8:
/,$)" 7JI DCAN ;GDB 76H:A>C: ID I=: B:6HJG: I6@:C 6;I:G 6  B>C G:HI E:G>D9 +*-. .=:H:;>C9>C<H9>;;:GE6GI>6AAN;GDBI=DH:D;+6LADL%DC:H >CL=>8=I=:N8DBE6G:96
H>C<A:  B>C +(, EG68I>8: ID 6 8DCIGDA 6C9 ;DJC9 H><C>;>86CIAN G:9J8:9 EDHI 68I>K>IN H:A;
G6I>C<H D; HI6I: 6CM>:IN  .=JH >I H::BH I=: 9JG6I>DC D; EG68I>8: >H 6C >BEDGI6CI ;68IDG
8DCIG>7JI>C<IDI=:EHN8=DAD<>86A:;;:8IHD;+(, $CHJB#L6HDCANE6GI>6AANHJEEDGI:96HI=:G:L6H6H><C>;>86CI9:8G:6H:>CA:K:AHD;
6CM>:IN8DBE6G:9ID76H:A>C:B>CJI:H6;I:GJ+(,IG6>C>C< #DL:K:GJ+(,9>9CDIBD9>;N
K6A:C8:6C9>IG:9J8:9A:K:AHD;6GDJH6AH>B>A6GANID6HB:6HJG:97N- AHD>IH=DJA97:
CDI:9 I=6I I=: A:K:A D; 6CM>:IN B:6HJG:9  B>CJI:H 6;I:G 6C9 CDI >BB:9>6I:AN 6;I:G J+(,IG6>C>C<L6HADL:GI=6C76H:A>C:L=>A:68I>K6I>DCD;+)-6H>C9>86I:97N8=6C<:H>C$C,(--
L6H D7H:GK:9 9JG>C< I=: IG6>C>C< 6C9 CDI 6;I:G  C :MEA6C6I>DC ;DG I=>H E=NH>DAD<>86A 6C9
EHN8=DAD<>86A9:HNC8=GDC>O6I>DC8DJA97:I=6II=::BDI>DC6A:ME:G>:C8:D;6CM>:INH::BHID
7:9:E:C9:CIDC7DI=6JIDCDB>8C:GKDJHHNHI:B;::9768@6C9I=:8D<C>I>K:>CI:GEG:I6I>DCD;
I=: HI>BJAJH I=6I >C9J8:9 I=:H: 7D9>AN H><C6AH )JBB:CB66 #/ (    $C DI=:G LDG9H
:BDI>DC6AHI6I:H6G>H:;GDBE=NH>DAD<>86A8=6C<:H6C9>CI:GD8:EI>K:H><C6A>C<D88JGH7:;DG:
I=:G:>H:BDI>DC6A6L6G:C:HHG>I8=A:N"6G;>C@:A  
.67A: #.!-',/'1#//'./'!.01. &A4
DA8=6&>BC

A>DB0;
D&#'
 P   P  P P  P  P +0;4=24
D&#'
 P  P   P   P  
 P  
 P  ;=()(D&#' P   P  
 P   P  
 P  
 P    $%$! #$" $ #$"%"&'
:H8G>EI>K:HI6I>HI>8H86C7:;DJC9>C.67A: 8GDHH>I:BHG:A6I:9IDI=:688:EI67>A>IND;I=:
68I>K>IN ;DG JH: >C ;DGB6A :9J86I>DC6A H:II>C<H >I:BH     G6I>C<H ;DG J+(, L:G: 6AA67DK:DJID;6C9H><C>;>86CIAN=><=:GI=6CHJ<<:HI>C<I=6IE6GI>8>E6CIHLDJA9688:EI
J+(,;DGJH:>C;DGB6A:9J86I>DC6AH:II>C<H !JGI=:GBDG:E6GI>8>E6CIH;DJC9J+(,:6HNID
E:G;DGBL>I=B>CJI:H7:>C<G6I:9=><=AN6H6C6EEGDEG>6I:9JG6I>DC ,6I>C<H;DG>I:BH 
:;;:8IH DC 6GDJH6A 6C9 BDD9 6G: >C A>C: L>I= I=: EHN8=DE=NH>DAD<>86A G:HJAIH EG:K>DJHANG:EDGI:9 .=JH#>H8DC;>GB:96HE6GI>8>E6CIHG6I:9J+(,6HBDG:;6KDG67A:6C9688:EI67A:
;DGJH:>C;DGB6A:9J86I>DC6AH:II>C<HI=6C EG:K>DJHHIJ9N7N*WDCC:AAJCA6E 
6AHD ;DJC9 =><= 688:EI6C8: G6I:H D; 9><>I6A 6J9>D <J>9:9 +(, >CI:GK:CI>DCH 6BDC<
H8=DDAI:68=:GH .=JHJ+(,=6HI=:EDI:CI>6AID7:6C688:EI:968I>K>IN7N7DI=HIJ9:CIH6C9
I:68=:GH AI=DJ<=6;DAADLJEHIJ9NL>I=JC>K:GH>IN:9J86IDGHLDJA9=:AE8DC;>GBI=>H $CG:HEDCH:IDI=:DE:CFJ:HI>DC  D;E6GI>8>E6CIH8DBB:CI:9I=6II=:N=69
9>;;>8JAIN G:A6M>C< 9JG>C< I=: 68I>K>IN  .=:B6I>8 8D9>C< ">77H   D; I=: G:6HDCH <>K:C
<:C:G6I:98D9:H ,:HEDCH:HG:;:GG:9ID6A68@D;8DB;DGIL>I=I=:68I>K>INA68@D;8DB;DGI
L>I=I=::CK>GDCB:CIEGD7A:BHL>I=I=:C6IJG:D;I=:68I>K>INHJ8=6H>I7:>C<IDDADC<IDD
>HDA6I>C< 6C9 CDI G:A6M>C< ID 8DCIG68I BJH8A:H  6C9 GJB>C6I>C< L=>A: DG =6K>C< <:C:G6A
9>H8DB;DGIL>I=G:A6M6I>DC68I>K>I>:HL=>8=EG:K:CIHG:A6M6I>DC H6C:M6BEA:D;I=:A6II:G

173

DC:E6GI>8>E6CILGDI:U$I=6H6AL6NH7::C9>;;>8JAI;DGB:I=>HH>8G:A6M>C<68I>K>I>:H7:86JH:
$9DCI;::A8DB;DGI67A: V.=:G:6HDC;DGI=:A6II:GG:HEDCH:H8DJA97:I=6I>C>I>6AE:G;DGB6C8:H
D; 76H>8 INE:H D; G:A6M6I>DC 68I>K>I>:H HJ8= 6H +(, DG B>C9;JA 7G:6I=>C< G:FJ>G: :;;DGI 6C9
6II:CI>DC .=>H86CB6@:I=:68I>K>I>:HA:HHG:A6M>C<6C9 DG:K:C>C8G:6H:HIG:HHA:K:AH 1>I=
EG68I>8: A:HH :;;DGI 6C9 6II:CI>DC 6G: G:FJ>G:9 6C9 <G:6I:G G:A6M6I>DC 7:C:;>IH 6G: G:6A>O:9 .=>H >H @CDLC 6H I=: E6G69DM >C G:A6M6I>DC IG6>C>C< #:>9:  DG@DK:8  6C9 >H BDG:
EG:K6A:CI>CE:DEA:L>I==><=:G76H6AA:K:AHD;6CM>:INDG6CM>:IN9>HDG9:GH&>B):LB6C
  .=>HE6G69DMB6N6AHD7:6;68IDGI=6I>C;AJ:C8:9I=:H:A; G:EDGIH;DG6;;:8I6C96CM>:IN .67A: #.!-',/'1#//'./'!.$+-/&#+./-'(0-1#3(+.#"*"#"0#./'+* 
D&#')7402C8E8CH8B40BH5>A<4C>?4A5>A<   )7402C8E8CH20;<B<43>F=  )7402C8E8CH4=4A68I4B<4   )7402C8E8CH8<?A>E4B<H<>>3   )743DA0C8>=>5C7402C8E8CH8B0??A>?A80C4
   F>D;3;8:4C>3>C78B02C8E8CH0B?0AC>502;0BB   F>D;3544;4<10AA0BB43?4A5>A<8=6C78B02C8E8CH8=02;0BB  )78B02C8E8CHF>D;374;?<414<>A4>?4=F8C7>C74AB   )78B8B0=0??A>?A80C41A40:02C8E8CHC>DB48=0=>=;8=42;0BB    
Q8=0=8=?4AB>=2;0BB 
QF74=BCD3H>=<H>F=   
                      ?E0;D4


 

 


 



  






 





   

.=:G:6G:H:K:G6AA>B>I6I>DCHIDI=>HHIJ9NI=6I86C7:JH:96HHJ<<:HI>DCH;DG;JIJG:LDG@  .=: HIJ9N L6H 8DC9J8I:9 >C 6 8DCIGDAA:9 :ME:G>B:CI6A H:II>C< L>I= >C9>K>9J6A E6GI>8>E6CIH
G6I=:GI=6C>C:K:GN96NH:II>C<HD;E6GI>8>E6CIHL=>8=A>B>IHI=:<:C:G6A>O67>A>IND;G:HJAIHID
G:6A A>;:H:II>C<H :HE>I:I=:H6BEA:H>O:7:>C<H>B>A6GIDDI=:GE=NH>DAD<>86AHIJ9>:HD;
H:A; G:<JA6I>DC68I>K>I>:H2DJ#/( )>8@:A#/( +6LADL%DC:H 6A6G<:G
H6BEA:H=DJA97:8DCH>9:G:9ID688DJCI;DGEDI:CI>6AI:8=C>86A>HHJ:H86JH>C<I=:ADHHD;96I6 1=>A:L6H>CI:C9:9ID7:6C:JIG6A68I>K>IN6C9L6HJH:96HHJ8=7N2DJ:I6A  
I=>HHIJ9N;DJC9I=6I>IEGD9J8:9H><C>;>86CI:;;:8IHDC6;;:8I6C9+)-68I>K>IN >;;:G:CI8DCIGDA
68I>K>I>:HI=6I6G:BDG:68I>K: C:JIG6A-B>I=)DGB6C DG76H:9DCA:6K>C<;G::I>B:ID
I=:E6GI>8>E6CIGD8@:G"GDO:AA:  H=DJA97:8DCH>9:G:9 
:NDC9699G:HH>C<I=:A>B>I6I>DCH;JGI=:G>CK:HI><6I>DCHH=DJA9:MEADG:L=:I=:GI=:
:;;:8IHD;J+(,9>;;:G7N<:C9:G:I=C>8>IN6C96BDC<9>H69K6CI6<:9HIJ9:CIH 99>I>DC6AAN
IDB6@:J+(,BDG:>C8AJH>K:;JIJG:LDG@>HC::9:9ID>9:CI>;N=DLIDHJEEDGII=DH:CDI67A:
ID G:A6M L=:C E:G;DGB>C< I=: 68I>K>IN HJ8= 6H 7N 6AI:G>C< <J>96C8: D;;:G:9 DG EGDK>9>C<699>I>DC6AH86;;DA9>C< !>C6AANB>8GDA:6GC>C<L=>8=;D8JH:HDC9:A>K:G>C<HB6AA8=JC@HD;H@>AAH
76H:9?JHI >C I>B:A:6GC>C<3=6C<1:HI  H=DJA97::MEADG:96H6L6ND;9:K:ADE>C<
HIJ9:CIH:A; G:<JA6I>DC8DBE:I:C8:H !DG:M6BEA:JAIG6 7G>:;H:A; G:<JA6I>DC68I>K>I>:H86C7:
I6J<=I6C9EG68I>8:9>CHIG:HH;JA6869:B>8H>IJ6I>DCHHJ8=6HEG>DGID:M6BH6C9EG:H:CI6I>DCH
7JI;DGBE6GID;6ADC<:G8JBJA6I>K:IG6>C>C<8DJGH:DCH:A; G:<JA6I>DCH@>AAH   

+GD<G:HH>K: BJH8JA6G G:A6M6I>DC >H 6C :K>9:C8: 76H:9 >CI:GK:CI>DC ;DG G:9J8>C< E=NH>86AHNBEIDBH D; 6CM>:IN 6C9 >BEGDK>C< L:AA 7:>C<  .=: 8JGG:CI HIJ9N ;>C9>C<H HJ<<:HI I=6I 6C
JAIG6 7G>:;K>9:D<J>9:9+(,68I>K>INEDHH:HH:HEGDB>H>C<8=6G68I:G>HI>8H;DGJH:>CH>IJ6I:9
I:68=>C<6C9EG68I>8:D;H:A; G:<JA6IDGNI:8=C>FJ:H6>B:96IG:9J8>C<E=NH>DAD<>86A6GDJH6A6
HNBEIDBD;6CM>:IN6C9HIG:HHDC6C68JI:76H>H>CJC>K:GH>INHIJ9:CIH *K:G6AAI=>HLDG@
6>BHIDHJEEDGIHIJ9:CIH>C9:K:ADE>C<8G>I>86AA>;:H@>AAHG:H>A>:C8:H:A; B6C6<:B:CIB:I6
8D<C>I>DC 7N >9:CI>;N>C< :K>9:C8: 76H:9 H:A; G:<JA6I>DC 68I>K>I>:H 6EEGDEG>6I: ;DG ;DGB6A

174

:9J86I>DC6A H:II>C<H 6C9 HJEEDGI>C< :9J86IDG JH: D; I=:H: 68I>K>I>:H  .=: A6II:G 86C 7:
68=>:K:9I=GDJ<=I:8=CDAD<>:HHJ8=6HA6HH(DD9EEI=6I=:AEI:68=:GH8G:6I:8A6HHGDDBDEEDGIJC>I>:H;DG<J>9>C<HIJ9:CIHI=GDJ<=:K>9:C8: 76H:9H:A; G:<JA6I>DCEG68I>8:H !

.=>H LDG@ =6H 7::C E6GI>6AAN ;JC9:9 7N +DBE:J !67G6 /C>K:GH>INWH +A6C:I6GN 1:AA 7:>C<$C>I>6I>K: +'1    ( -E6C>H= (>C>HIGN D; -8>:C8: 6C9 $CCDK6I>DC +$ ,  <G6CI:9 7N ($$)  $  
 :E6GIB:CI D; ,:H:6G8=
6C9 /C>K:GH>I>:H D; I=: "DK:GCB:CI D; 6I6ADC>6 -",   6C9 JGDE:6C /C>DC ):MI":C:G6I>DC / 

0;I0A>CC8(80BB>=8>;><1>824A8#'0A3802E060;2>=CA>;0B0<0A:4A>5
4<>C8>=A46D;0C8>=8=740;C7H03D;CBA4E84F,1.1*,&$.49&+1.1*9 
0==027<5C%"D:>F82I&(4?C4<14ADC><0C82E4=C0B43(H=27A>=8I0C8>=>5
#D;C8<>30;0C0(CA40<B5A><,40A01;40=3<184=C(4=B>AB=631 ??  
0=C94B D=C-(C48= 
&D1;8240;C7??A>0274BC>&A><>C8=6*=8E4AB8CH
(CD34=CBT#4=C0;40;C7;>10;&4AB?42C8E4633(0549&+,$539(21354   
40A3B;4H#+D9>E82#&>AC4A>)A4BB4AA0#4A=J=34I"4>
;0BB#>>30??0
2;0BBA>><>A274BCA0C8>=C>>;5>A834=C85H8=60=38=5;D4=28=6BCD34=C<>>3B=!3$04)13/,0*
($30,0*8,5+($0,0*)6.!(&+01.1*,(4 5+6312($010)(3(0&(10!(&+01.1*90+$0&('($30,0*!(.)5!+((5+(3.$0'4 (25(/%(3 ;31&((',0*4 ??  (?A8=64A=C4A=0C8>=0;&D1;8B78=6
4C4;;0+4AB27DA4&)7405542C8E4B;834A3868C0;B4;50BB4BB<4=CB20;45>AC74
<40BDA4<4=C>57D<0=4<>C8>=B.1 0( 4 
A02:4CC#'4H4B#''8E4AB(;14ACB>=$(0;>E4H&;0BBA>><
4<>C8>=0;2;8<0C4C40274A0558;80C8>=0=3BCD34=C2>=3D2C!+(1630$.1).$44311/05(3$&5,10  
A8C27;4H0A58=:4;($=C4A>24?C8>=0=34<>C8>=633(052,0,10,049&+1.1*9
 
>;184A"'DB7)
558202H>5011A4E80C43?A>6A4BB8E4<DB2;4A4;0G0C8>=8=07867
BCA4BB2>;;464B0<?;405(30$5,10$.1630$.1) 53(44$0$*(/(05  
0;;46>N<4I 0;0=I0("40;";>?8B 0A2L0#K=34I %;8E0&KA4I ><K=427
)>AC>B0 N<4I0;;46>#(8<>=4;;8#DM>I '8E4A00A0E020 #5542C8E4=4BB>5<DB82C74A0?H0=3?A>6A4BB8E4<DB2;4A4;0G0C8>=8=A43D28=6BCA4BB145>A4
4G0<B0=38<?A>E8=602034<82?4A5>A<0=248=$DAB8=6BCD34=CBA0=3><8I43CA80;634('6&$5,10!1'$9 811B')74<0C822>38=60=320C46>A8I8=60$.9:,0*6$.,5$5,7($5$  
>55<0= ,4=B>=A=B&(C08=1A>>:""0=3B14A6".>D=6 8;;  '43D243BH<?0C74C82=4AE>DBBHBC4<A4B?>=B8E8CH0BB>280C43F8C7C74A4;0G0C8>=
A4B?>=B4 &,(0&(       
4834 >A:>E42)  '4;0G0C8>=8=3D2430=G84CH?0A03>G820;0=G84CH4=70=24<4=C
3D4C>A4;0G0C8>=CA08=8=61630$.1)1046.5,0*$0'.,0,&$.49&+1.1*9    !8<$4F<0=#)74?0A03>G>5A4;0G0C8>=CA08=8=6'4;0G0C8>=8=3D2430=G84CH0=3
<4380C8>=45542CB>5=460C8E42>=CA0BCB4=B8C8E8CH8=64=4A0;8I430=G84CH38B>A34A0=3<09>A
34?A4BB8E438B>A34A1630$.1)))(&5,7(,413'(34    !>10H0B78(!>8C010B78!
5542CB>5?A>6A4BB8E4<DB2;4A4;0G0C8>=>=24A41A0;02C8E8CH
=5#'8=E4BC860C8>=1/2.(/(05$39!+(3$2,(4,0(',&,0(   
4==8=6B '!0<0A2:)(C4F0AC33H# >7=B>=&  ;C4A=0C420A38>E0B2D;0A
10B4;8=40BB4BB<4=CC427=8@D4B+0=8;;0>AA4BC8=610B4;8=449&+12+94,1.1*9  "01>A34(;;4=#(>A64B*B:A0#/0<<8C$.>D#>B0=6)#>B;4H
>BB4E8;;4
&BH27>?7HB8>;>6820;45542CB>5B;>F?02431A40C78=60CB8G2H2;4B?4A
<8=DC4F8C7>AF8C7>DC740ACA0C4E0A8018;8CH18>5443102:49&+12+94,1.1*9  4  "01>A34(#>B;4H)70H4A 
40ACA0C4E0A8018;8CH0=320A3802E060;C>=48=
?BH27>?7HB8>;>6820;A4B40A27RA42><<4=30C8>=B5>A4G?4A8<4=C?;0==8=630C00=0;HB8B0=3
30C0A4?>AC8=63105,(34,049&+1.1*9 

175

"47A4A&#0AA'(0A6D=0A09,>>;5>;:'"  (CA4BB<0=064<4=CC427=8@D4B0A4
C74H0;;4@D8E0;4=C>A3>C74H70E4B?42858245542CB,1)(('%$&-$0' (.)3(*6.$5,10   #0AC40D)#4::4A  )7434E4;>?<4=C>50B8G8C4<B7>AC5>A<>5C74BC0C4B20;4>5
C74(?84;14A64A(C0C4S)A08C=G84CH=E4=C>AH()3,5,4+1630$.1).,0,&$.49&+1.1*9 
 #2'04!A>BB  <>C8>=A46D;0C8>=/15,10  R
$82:4;!4CC;4A#D47;10274A#"07<0==)A8CC!0AC024:'027;4A'>C74A$
664A'>C74A,!">4F)
5542C>5?A>6A4BB8E4<DB2;4A4;0G0C8>=8=
03>;4B24=C54<0;41A>=2780;0BC7<0?0C84=CB0A0=3><8I433>D1;41;8=32>=CA>;;43BCD3H
1630$.1)49&+141/$5,&(4($3&+    $D<<4=<00";4A40=0A8'84C0=4= !
1',.9/$241)(/15,10431&((',0*41)5+($5,10$.&$'(/91) &,(0&(4   %1A03>E8Y (D;8:# A<BCA>=60AC4A
)0:8=6054F344?1A40C7BB86=85820=C;H
A43D24B278;3A4=B?7HB8>;>6820;0A>DB0;8=4E4AH30HB4CC8=6B'4BD;CB>50?A4A468BC4A43E834>
8=C4AE4=C8>=(7(.12/(05$.49&+1%,1.1*9 4
%T>==4;;&(D=;0?"")40274A0224?C018;8CH>5?A>6A4BB8E4<DB2;4A4;0G0C8>=8=C74
2;0BBA>><5>AC74CA40C<4=C>5C4BC0=G84CH1630$.1)49&+1.1*,454$0'1604(..134,0 &+11.4
    &0A27<4=C,>7;414A','48=4A<0= >=4B"&BH27>?7HB8>;>6820;10B4;8=4
<4C7>3B0=3DB064=160'$5,1041)6*/(05('1*0,5,10(631(3*101/,&4$0'2(3$5,10$.
(6314&,(0&(5+05(30$5,10$.10)(3(0&( (.'$4$351)05(30$5,10$. !131051$0$'$6.9  31&((',0*4$35 ??    (?A8=64A
=C4A=0C8>=0;&D1;8B78=6
&0B2>4#4CA82:(&0A:4A
)748<?02C>5BCA4BB>=BCD34=CB8=B42>=30AH
B27>>;0=378674A43D20C8>=05(30$5,10$.1630$.1)'1.(4&(0&($0'#165+  
&0F;>F" >=4B)748<?02C>5011A4E80C43?A>6A4BB8E4<DB2;4A4;0G0C8>=>=
B0;8E0AH2>AC8B>;0=3B0;8E0AH8<<D=>6;>1D;8=B622.,('49&+12+94,1.1*9$0'
,1)(('%$&-  (0:0:810A0#)0:4D278(0H0=>   5542C>5A4;0G0C8>=CA08=8=6>=20A3802
?0A0BH<?0C74C82C>=449&+12+94,1.1*9  (0H8B'0<8A4I'&0A4B$#8G43A40;8CH>A"%60<4?;0H>BC4A8=6B>280;
8=C4A02C8>=8=278;3A4=F8C7DC8B<",356$.($.,59   (2784=;4*=64A
%?4=;014;?;0241>CA40C<4=CC>8<?A>E4A4;0G0C8>=CA08=8=645542CB8=
740;C7H?BH27>;>6HBCD34=CB0A0=3><8I432>=CA>;;43CA80; &,(05,),&(21354  (2D;C#!=>3C'(F0ACI 'A868380A8A8')78=:8=60=3544;8=6
8=38E83D0;38554A4=24B8=7018CD0;4<>C8>=A46D;0C8>=0=3BCA4BBA4;0C43<>>30A40BB>280C43F8C7
?A45A>=C0;4G42DC8E42>=CA>;.,0,&$.49&+1.1*,&$. &,(0&(     (<8C7!$>A<0= A845A4;0G0C8>=CA08=8=68B=>CBD558284=CC>0;C4AC>;4A0=24C>
4G?4A8<4=C0;?08=8==>E824B1 0(4  +06=>;8"4CC8=8<>A44#0B8(#4BB4A8
'4;0G0C8>=6D83438<064AH
A43D24B?4A8>?4A0C8E40=G84CH0=3?08=8=278;3A4=0A0=3><8I43BCD3H6312($01630$.1)
(',$53,&4  R +>64;((27F014""40A=8=60=3<4<>AHD=34ABCA4BB8<?;820C8>=B5>AC742;0BBA>>< &,(0&(1)($30,0* 
,8;2IHZB:0WHB0:'03><B:0&>32I0AB:0X>F02:0#/09C >A=>FB:8#
(:>=842I=H&E0;D0C8>=>5C7445542C8E4=4BB>5A4;0G0C8>=8=;>F4A8=6C74;4E4;>5
0=G84CH8=H>D=603D;CB0?8;>CBCD3H05&&62('07,310($.5+  ,>AB;4H &4==8=6C>=>A2>A0='(D??>AC8=6<4=C0;740;C70=3F4;;148=6>5
D=8E4AB8CH0=32>;;464BCD34=CBBHBC4<0C82A4E84F>5A4E84F;4E4;4E834=24>58=C4AE4=C8>=B1 0( 4  .>D#"01>A34((0;E>CC8/0<<8C$#>B;4H>BB4E8;;4
=5;D4=24>50
B8=6;4B;>F?02431A40C78=6B4BB8>=>=20A3802E060;02C8E8CH8=0C7;4C4B05(30$5,10$.1630$.1)
(05$.($.5+$0''',&5,10     
/70=6 ,4BC'
4B86=8=6#82A>;40A=8=6=BCAD2C8>=5>A&A>54BB8>=0;4E4;>?<4=C
)7A>D670><?4C4=2H0B43??A>027!(&+!3(0'4  
 

176

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Development and Evaluation of the
Platform for Online High-Level CooperativeGames
Feng-Lung LIUa, Geng-De HONGb*, Ju-Ling SHIHc & George Ghinead
abcGraduate Institute of Network Learning Technology, National Central University, Taiwan
dDepartment of Computer Science, Brunel University London, UK*hgengde@gmail.com
Abstract: The design of educational games plays an increasingly important role inmodern education. By combining learning with games, it can attract students' attention
and making learning more interesting. This study describes the development of aplatform designed specifically for the online high-level cooperative game in which
students are encouraged to achieve their own goals as well as common goals to reach
win-win situation. Technology Acceptance Model (TAM) was used to analyze students'
overall perceptions of the platform. The results show that the design of the gameplatform had high perceived usefulness and perceived ease of use, and players also
showed positive attitudes and usage intentions. Therefore, a good game platform
should have interactivity, educational value, problem-solving settings, and usability toprovide the best gaming experience and have a positive impact on learning.
Keywords: Game-Based Learning, Gaming Platform, System Development, HighLevel Cooperative Games, Technology Acceptance Model

1. Introduction
1.1 Research goal
In traditional learning methods, students usually need to spend a lot of time and effort
learning knowledge and skills, which can lead to learning fatigue and loss of interest. Today,
some games are gradually integrating educational resources and elements to enhance
students' learning effectiveness and interest. According to Petri and von Wangenheim(2017), incorporating interactive, exploratory, and problem-solving elements into games canenhance learning outcomes by boosting students' engagement, and motivation; resulting in
better learning outcomes.
The design of educational game platforms can contribute to students’ experience ofthe pleasure of success and sense of achievement in games. This, in turn, can enhancetheir learning motivation and interest. Therefore, this study involved the development of an
online high-level cooperative game platform. The platform provides various game modules
for resource sharing, trading, bargaining, and issue discussions among players. After thegame, reflections and feedbacks were received using the TAM questionnaire to investigate
students' learning attitudes and technology acceptance. Evaluation results help systemdesigners to develop more effective online game systems and enable continuous
improvements of the game system.
1.2 Research questions
This study aims to understand students' experiences, acceptance, and willingness to use theonline high-level cooperative game system using the Technology Acceptance Model (TAM).
The research question of this study is to investigate students' perceived usefulness,
perceived ease of use, attitudes toward use, willingness to use, and overall feedback
experience of using the game system.

177

2. Related work:
2.1 Game-Based Learning
Digital games have emerged as effective learning tools that can assist students in acquiringknowledge while playing. Gee (2003) argues that digital games can enhance students'
problem-solving, analytical, and creative abilities, and even help them develop skills beyond
the classroom. Originally conceived as entertainment, digital games have become widelyused as a learning tool, thanks to the prevalence of 3C technology products. Digital games
provide an entertaining environment that increases student engagement and motivation(Annetta et al., 2009). Laine and Lindberg (2020) have demonstrated that digital games
possess powerful motivational factors that can spark learners' initiative. By providing an
engaging and interactive setting that challenges and entertains, digital games facilitate
learning and cultivate students' problem-solving, analytical, and creative skills. Given theirsignificant motivational benefits, digital games have become a popular tool for learning andare anticipated to remain a crucial component of education.
2.2 Technology Acceptance Model (TAM)
The Technology Acceptance Model (TAM) was initially proposed by Davis Bagozzi, andWarshaw (1989) to investigate the factors influencing individuals' willingness to use newtechnologyį TAM comprises of two fundamental elements that substantially influence auser's inclination towards adopting new technology and shaping their behavioral attitudes.
Perceived usefulness pertains to the extent to which users deem that utilizing a system canboost work efficiency. Perceived ease of use denotes the user's subjective perception of thesystem's complexity or simplicity, which is closely associated with their level of information
technology proficiency. In order to further explore the acceptance of other technologies, it isnecessary to consider the attitude toward using, which includes the user's perception offactors such as interface environment and system quality, and how they influence the user's
behavior. Behavioral intention is influenced by multiple variables and ultimately reflects theuser's actual operational behavior. The TAM has been widely used in research on user
acceptance of technology and is considered an established model for understanding
technology acceptance (Gefen, Karahanna, & Straub, 2003). Examples of its application
include e-learning systems (Nga et al., 2007), online courses (Landry et al., 2006), websites(Koufaris, 2002), and others. In addition, TAM can be further expanded by incorporatingexternal variables and combining with other theories. Different external variables may berelated to system, user, and task characteristics, which can be investigated in research.

3. Online High-Level Cooperative Games Platform Design
This study combines game-based learning and issue-based learning to develop a high-level
collaborative game-based learning system. The system incorporates historical crisis events,
and through game mechanics, players can solve game problems through trading andnegotiation. The system design is based on the Great Voyage game developed by Hong,
Shih, and Lu (2022), allowing players to negotiate and divide tasks among groups to resolve
historical crises. Students are divided into five groups to role-play European countries:
England, France, Portugal, Spain, and the Netherlands, each with their own tasks. In thegame, players go through four stages of gaming process in every round. The first stage isinternal affairs, in which countries generate strategic plans for crisis resolution. The second
stage is diplomacy, every country negotiates to resolve conflicts. The third stage isdeclaration, in which country representatives state their stands and propose solutions to thecrisis event. The final stage is settlement, in which the Pope assesses the situation of eachcountry and makes judgments. The game ends when all events are effectively resolved.
In respond to this game process, this study designed four functional modules in thesystem. Provide players with a good gaming experience, this system with simple user

178

interface, clear game information, and obvious text and icons to guide students to complete
game tasks smoothly and foster their learning motivation. The design of each module isintroduced and explained below:(1) Information Model (Figure 1):
The Information Model interface not only displays relevant information about theplayer's own country in the game, but also allows for comparison through the intelligence
information screen after each crisis is resolved. By comparing the total assets and trends
shown in the initial information and current status, players can obtain valuable information
about their own country and make wiser decisions to achieve victory. This transparency of
information can help players better understand the situation of their own country, optimize
their resource allocation, and effectively respond to crisis challenges. This feature enhances
the fun and challenge of the game, while also giving players a sense of self-improvement.
(2) Negotiation Model (Figure 2):
In the Negotiation Model interface, players can freely choose different strategy
behaviors, moods, message recipients, and discussion topics to better meet their needs andnegotiation goals. This allows players to participate more comprehensively in negotiation
topics and dialogue processes, fosters cooperation and communication, and increases
player involvement in resolving crises. During the negotiation process, text messages can bedisplayed in real-time, allowing players to conveniently understand the negotiation process
and results; the different colors of the dialogue box can distinguish messages from different
countries, further enhancing the visual effects and user experience. This interface design
enhances the freedom and fun of player participation in negotiation and dialogue, while also
strengthening the realism and interactivity of the negotiation process, enabling players tobetter experience and learn negotiation skills and strategies.

Figure 1. Information model interface

Figure 2. Negotiation model interface(3) Trading Model (Figure 3):
The Trading Model provides multiple options and trading methods, allowing players
to easily select different trading modes, resource items, commodity categories, trading
partners, trading items, and trading commodities, thereby increasing the flexibility andoptions of resource trading for players. This interface design not only enhances the freedom
and fun of players' participation in resource trading but also strengthens their decisionmaking ability and strategic thinking in resource trading. Players can flexibly choose different
trading methods and commodity categories based on their own needs and goals, in order toachieve maximum economic benefits and game objectives, helping players to understandthe essence and rules of resource trading.
(4) Auction Mode (Figure 4):
The Auction Model interface provides players to participate freely in product auctions
and transactions, and to list their own products, conduct auctions, and choose to buy spices
and other goods, as well as query transaction records after completion. Through these
operations, players can effectively understand the general process of the transaction andthus enhance their understanding and comprehension of the transaction. At the same time, it
can also help players improve their trading skills and strategic abilities, thereby gaining more
economic benefits in the game. More importantly, this platform allows players to gain more
freedom and fun in participating in auctions and transactions, while also enhancing theirunderstanding and mastery of trading skills and strategies in the game.

179

Figure 4. Auction model interface

Figure 3. Trading model interface

4. Research design
4.1 Research Framework
This study recruited fifteen Taiwanese graduate players to participate, who were randomly
assigned into five groups. The research procedure, commenced with an explanation of thegame rules and the system's operation before the game's commencement. The gaming
activities were then documented in the game system. During the game (Figure 5), all players
engage in online domestic discussions and diplomatic negotiations to solve problems andachieve the assigned tasks. The system supports students to tackle multiple crisis events
which are supported by the system with various functional modules. Subsequently, uponcompletion of the game, the students were guided to complete a feedback questionnaire
concerning the game system.

Figure 5. Players' gameplay process
4.2 Research Tools
The questionnaire design of this study is based on the TAM proposed by Davis et al., In theTAM, the external variables of perceived usefulness and perceived ease of use are
explored. This study focuses on "computer self-efficacy" as the main factor to examine
whether players have the ability to use information systems to complete tasks.
Therefore, the game content was taken into consideration; appropriate design andadjustments were made with the results of a questionnaire. The questionnaire was divided
into five dimensions, including computer self-efficacy, perceived usefulness, perceived easeof use, attitude toward using, and behavioral intention to use. Each dimension includes fivequestions to understand players' acceptance and satisfaction with the system platform.
Further evaluation of the system performance was done in terms of technology acceptance.

5. Results
Players' perceptions using online high-level cooperative games platform was analyzed withTAM (Table 1). According to the data results, it can be observed that the average value ofeach dimension is higher than the medium score of 3.00 of the five-point likert scale. Thesehigh ratings reflect the students' positive attitude towards the game, and their willingness tousing it in the future. They believe that the game can make it easier for them to learn.

180

Table 1. TAM evaluation to the game platform
Dimension
Mean
S.D.
Self-Efficacy for Computer
Perceived usefulness

3.73

1.02

3.76

0.73

Perceived Ease of Use

3.32

0.84

Attitude towards Use

3.92

0.71

Behavioral intention

3.81

0.67

In the players' self-efficacy for computer dimension, the overall mean score is 3.73
(SD=1.02). The various sub-dimensions have average scores ranging from 4.33 to 3.13.
Among them, the highest score is for Q1 "I often use computer devices to aid in my learning"(m=4.33), indicating that players who frequently use computer devices to assist learning.
However, the lower scores are seen in Q2 "I often use computer devices to play online
games" (m=3.73), suggesting that players use computer devices less often to play online
games. This shows that players have a higher acceptance of using computers to play online
games, indicating that they are better able to cope with the challenges and problems invarious computer games, and have confidence and ability in using the platform.
In the players' perceived usefulness for computer dimension, the overall mean scoreis 3.6 (SD=0.73), with the mean values of sub-dimensions ranging from 3.93 to 3.53. The
values indicate that most players have a high level of agreement with the "high-level
collaborative online game platform". The highest score is seen in Q10 "Overall, using a 'highlevel collaborative online game platform' is beneficial for me" (m=3.93), suggesting that
players find the platform highly beneficial. However, in Q8 "The 'high-level collaborative
online game platform' saves me a lot of time and effort when I am learning" (m=3.53), thelower score indicates that players perceive the platform to be limited in terms of time-saving
and effort-saving during learning. The overall result shows that players still perceive thegame system as assisting them in achieving their gaming or learning goals.
In the players' perceived ease of use dimension, the overall mean score is 3.32
(SD=0.84), and the mean values of each sub-dimension range from 3.47 to 3.20. The
highest score is seen in Q15 "Overall, I find the 'high-level collaborative online gameplatform' to be easy to use" (m=3.47), indicating that players find the platform easy tooperate, with clear and concise functions that allow for easy learning and quicker mastery of
the system's features. However, the lower score is observed in Q13 "I am proficient in using
the 'high-level collaborative online game platform'" (m=3.20), suggesting that players may
find it difficult to become familiar with the platform's use due to the complexity of the game's
mechanisms. Overall, players perceive the game platform as easy to use, with clear andconcise functions that enable them to learn and master the system quickly.
In the players' attitude towards use dimension, the overall mean score is 3.92
(SD=0.71), and the average scores of each sub-dimension range from 4.07 to 3.67,
indicating that players are highly satisfied with the overall experience of the system. The
highest scores are observed in Q18 "I think that the 'high-level collaborative online gameplatform' is an attractive learning tool" and Q20 "Overall, I think that using the 'high-level
collaborative online game platform' is a good idea" (m=4.07), suggesting that players believe
the game promotes learning and cooperation. Players find the platform easy to use andappreciate its practicality, which helps them achieve game objectives and effective learning.
In the players' behavioral intention to use dimension, the overall average score is
3.81 (SD=0.67), with average scores ranging from 3.87 to 3.80 in each sub-dimension,indicating a high willingness to use the system among the participants. Among them, in Q24“I expect to continue using the "high-level collaborative online game platform" in the future”(m= 3.87) is the highest score for the expectation that they will continue to use. This
indicates that learners have a high willingness to use the system, are willing to spend timeand effort using the game for learning, and are satisfied with its good experience andfunctionality, providing an excellent learning environment.

181

6. Conclusion
This study developed an online gaming platform with high-level cooperation, providing
various system functional modules to assist players in effective communication,
collaboration, conflict resolution, issue discussion, and reflection during the game. After thegame, the TAM model was used to evaluate the game, and the questionnaire design wasappropriately adjusted based on the game content to understand players' technology
acceptance and satisfaction with the system platform. The results showed that players had
positive evaluations of the various system functional modules, which helped them achieve
their learning goals and provided a good gaming experience. It also stimulated their learninginterest and motivation, met their gaming needs and expectations.
According to research findings, a good gaming platform should have interactivity,
educational value, problem setting, and ease of use to provide players with the best gaming
experience and play a positive role in learning. Interactivity promotes player interaction andcommunication with other players and enhances team collaboration awareness. Setting
challenges and problems in the game allows students to learn through problem-solving andfurther stimulates their learning motivation and interest. Ease of use is also essential; simple
operation and a good user experience allow players to easily immerse themselves in thegame. By designing games in this way, it can not only improve students' problem-solving
skills and creativity but also promote communication and interaction among students,
enhancing their social skills and teamwork spirit.
The ongoing system development seeks to tailor the platform for global educational
endeavors, with the goal of investigating how participants from diverse cultural backgrounds
interpret and engage with the various in-game roles. Anticipating cross-cultural comparisons
is part of this endeavor.

Acknowledgements
This study is supported in part by the National Science and Technology Council (previously
known as Ministry of Science and Technology) of Taiwan, under MOST 108-2511-H-008 016 -MY4 and 112-2918-I-008-003

References
Annetta, L. A., Minogue, J., Holmes, S. Y., & Cheng, M. T. (2009). Investigating the impact of video
games on high school students’ engagement and learning about genetics. Computers &Education, 53(1), 74-85.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A
comparison of two theoretical models. Management Science, 35(8), 982-1003.
Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers inEntertainment (CIE), 1(1), 20-20.
Gefen, D., Karahanna, E., & Straub, D. W. (2003). Trust and TAM in online shopping: An integrated
model. MIS Quarterly, 51-90.
Hong, G. D., Shih, J.L., & Lu, Y.H. (2022). High-level Cooperative Behavior Model of Online Summit
Games. The Main Proceedings of the 30th International Conference on Computers in Education(530-535), Malaysia: Asia-Pacific Society on Computers in Education.
Koufaris, M. (2002). Applying the technology acceptance model and flow theory to online consumer
behavior. Information Systems Research, 13(2), 205-223.
Laine, T. H., & Lindberg, R. S. (2020). Designing engaging games for education: A systematic
literature review on game motivators and design principles. IEEE Transactions on Learning
Technologies, 13(4), 804-821.
Landry, B. J., Griffeth, R., & Hartman, S. (2006). Measuring student perceptions of blackboard using
the technology acceptance model. Decision Sciences Journal of Innovative Education, 4(1), 8799.
Ngai, E. W., Poon, J. K. L., & Chan, Y. H. (2007). Empirical examination of the adoption of WebCT
using TAM. Computers & Education, 48(2), 250-267.
Petri, G., & von Wangenheim, C. G. (2017). How games for computing education are evaluated? Asystematic literature review. Computers & Education, 107, 68-90.

182

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Thematic Analysis Exploring Flexibility
in Programming-based Mathematical
Problem Solving
Huiyan YEa, Oi-Lam NGa* & Zhihao CUIa
Faculty of Education, The Chinese University of Hong Kong, China* oilamn@cuhk.edu.hkaAbstract: Research on computational thinking (CT) have received significant attention
in recent years, but how CT and mathematical thinking (MT) work together to influence
problem solver’s computational strategies and competency has not been illuminated.
Taking the method of thematic analysis combined with a conceptual framework offlexibility in programming-based mathematical problem solving, this study aims to shedlight on the interplay between CT and MT. We identify two kinds of flexibility when aproblem solver engages in programming-based mathematical problem-solving:translation flexibility and integrated flexibility. We hope that this research will contributeto a deeper understanding of the interplay of CT and MT as a basis for conceptualizing
an interdisciplinary form of STEM education and thinking on task design, pedagogy,
and assessment approaches of CT-based mathematics instruction.
Keywords: flexibility, computational thinking, mathematical thinking, interplay

1. Introduction
Computational thinking (CT) is regarded as a fundamental skill required in daily life, and akind of analytical thinking that shares common characteristics with mathematical thinking(MT) in many ways (Wing, 2011). Thus, researchers have investigated ways to integrate CT
into mathematics teaching and learning (e.g., Ng & Cui, 2021). However, one of the key
challenges is to understand how CT and MT work together to influence students’ strategies
when engaging in mathematical problem-solving in computational contexts. Previous studies
have identified discrepancies between CT and MT, which make mathematical problemsolving in computational contexts more challenging for students (Cui & Ng, 2021). Therefore,
understanding the complex relationship of CT and MT is vitally important if CT is to bemeaningfully integrated into mathematics education.
While the breadth of research has advanced the empirical basis of understanding what
CT-based mathematics instruction entails, very little is currently known about the processesof how CT and MT specifically work together in this context. Meanwhile, noting the diversityof solution strategies in the CT-based mathematical problem-solving, research on flexibility
in mathematical problem-solving may provide possible directions for our attempts to explore
the interplay between CT and MT. Flexibility refers to “the ability to change according toparticular circumstances” (Star & Seifert, 2006, p. 281), which relies on the problem solver’s
understanding of the knowledge relevant to the problem situation. If the problem context is
extended to a CT-based mathematical problem, not only would multiple programming
solution strategies be possible, but the problem solvers would also require knowledge ofprogramming and mathematics, as well as to adapt this knowledge according to theprogramming environment used and mathematical ideas involved respectively within the
problem context. In this study, we are interested in delving into this very interplay between
CT and MT from perspective of flexibility. Therefore, the study is guided by the researchquestion: What types of flexibility are exhibited during CT-based mathematical problemsolving?

183

2. Theoretical 澶ackground濅濁濄 濖瀂瀀瀃瀈瀇濴瀇濼瀂瀁濴濿澳濧濻濼瀁濾濼瀁濺澳濴瀁濷澳濠濴瀇濻濸瀀濴瀇濼濶濴濿澳濧濻濼瀁濾濼瀁濺澳In this study, we define CT as “the thought processes involved in formulating problems andtheir solutions so that the solutions are represented in a form that can be effectively carriedout by an information-processing agent” (Wing, 2011, p. 20), and MT not as “thinking about
the subject matter of mathematics but a style of thinking that is a function of particular
operations, processes, and dynamics recognizably mathematical” (Burton, 1984, p. 35). In
other words, both CT and MT refer to the cognitive structures that govern an individual’s
programming and mathematical activities respectively, as well as the anticipation of theresults of those activities. CT is thought to share a similar approach to MT in problem solving(Wing, 2011). Researchers also attempted to find out some of the empirical connections
between CT and MT, with some reckoning that CT and MT are mutually supportive or haveoverlapping features. For example, Ng and Cui (2021) argued that mathematical exploration
can serve as a site that allows students to engage in CT in meaningful ways. On the other
hand, the presence of CT and MT may occur in mismatching ways. As Rich and colleagues(2018) point out, mathematics focuses on students’ flexible arithmetic operations andarithmetic strategies, while CT focuses on sequence so that the computer can recognize andexecute. While these studies point to an inter-relationship between CT and MT (e.g., CT
support MT, or vice versa, or the two are constraining), the conclusions are relatively broad
and lack a detailed perspective on how such facilitative or hindering relationships of CT and
MT arise and develop. That is, there requires more evidence to capture the presence or
expression of CT and MT in a clear way and how they affect students’ thinking processes
when they engage in CT-based mathematical problem solving.

濅濁濅 濙濿濸瀋濼濵濼濿濼瀇瀌澳濼瀁澳濣瀅瀂濵濿濸瀀澳濦瀂濿瀉濼瀁濺澳
Flexibility is considered to be a key competency in problem solving. Even if research
focusing on flexibility has been studied extensively in different domains of mathematics,
there has been little research on interdisciplinary flexibility, that is, how flexibility manifest
across different subject domains. In particular, the flexibility of working across CT and MT in
the context of CT-based mathematical problem-solving not yet been empirically studied andanalysed. In this study, we adopt flexibility as “knowledge of multiple strategies and theability to select the most appropriate strategy for a given problem and problem-solving
circumstances” (Star et al., 2022, p. 2) so that we can observe how flexibility aredemonstrated through various tasks in our designed CT-based mathematical problems. This
investigation can potentially refine our understanding of how problem solvers makeconnections between computational and mathematical concepts. In particular, we pay
attention to the tasks in which problem solvers are situated to translate between
mathematical and programming language or integrate mathematics concepts andcomputational concepts while making meaningful connections of CT and MT. When
analysing these tasks, we recognise that programming itself is a tool-mediated activity;
therefore, the instrumentation of different programming tools would necessitate different
kinds of CT-MT translations and connections to occur. For this reason, we choose toaddress one specific programming language, Scratch.

3. 濁濙濨濜濣濘濣濠濣濛濭濆濁濄 濖瀂瀁瀇濸瀋瀇澳濴瀁濷澳濣濴瀅瀇濼濶濼瀃濴瀁瀇澳澳This study is part of a larger design-based research addressing the research objectivesof designing a series of tasks for supporting computationally enhanced mathematicalproblem-solving in school mathematics classrooms, and of studying student learning184

trajectories in terms of developing their CT, MT, and problem-solving competence in thedesigned tasks. The participants in this study were from two programming-based
mathematical problem-solving programs. One was a student program, which was designbased research (DBR) undertaken with three iterative cycles of implementations withseventh to ninth grade (age 12 to 15) students (n = 95) from three secondary schools in
Hong Kong. Another one was a teacher program, which adopted task-based semi-structured
interview to collect data. Eight in-service mathematics teachers from different schools in
Mainland China, participated in a five-week online professional development workshop in
Scratch programming. The two programs were conducted independently upon obtaining
participants consent, with the student program being undertaken in 2021 and the teacher
program being conducted in 2022 via Zoom.
In this study, we adopt the method of thematic analysis to further interpret thetranslation/connection involved between CT and MT in the programming solutions to thechosen tasks through the lens of flexibility. As the purpose of this study was to explore theflexibility of learners’ strategies in solving programming-based mathematics problems,
although participants included two different groups (i.e., students and in-service mathematics
teachers), we believe our data still appropriate to answer the RQs, since the programming
solutions, as opposed to the participants (or problem solvers) were the focus of this study.

濆濁濅 濗濴瀇濴澳濖瀂濿濿濸濶瀇濼瀂瀁澳濴瀁濷澳濔瀁濴濿瀌瀆濼瀆澳
Each program contained an introduction game and several tasks related to four domains:
number, algebra, geometry, and data handling. A total of 18 tasks were studied in responseto the research question posed, in which students program contained 14 tasks (1introduction game, 5 in number and algebra, 4 in geometry, 4 in data handling) and teachers
program contained 4 tasks with one task in each topic. All tasks are programming-based
mathematical problems and are all open-ended problems with no fixed coding strategy. For
example, “Please write a code project that can be used to verify whether an entered number
is prime or not.” Coding-related solutions are all thought out and collaborated by participants(students or teachers). The researchers will observe participants’ problem-solving process
and guide the participants to reflect through task-based interviews at the appropriate time,
which aims to understand what they think and to help them solve the difficulties theyencounter during the programming process. Therefore, our data include (a) video and voicerecording of interactions and interview between researchers and participants; (b) collectionof field notes, screen capturing, developed artefacts and documents in both programs, which
help us to gain a deep understanding of the research questions.
As a qualitative analysis method to identify, analyze and report patterns of data (Braun& Clarke, 2006), thematic analysis is regarded as an effective approach for identifying
similarities and differences, yielding unforeseen insights and summarizing key features of alarge data set (Nowell et al., 2017). Therefore, we adopted thematic analysis with the aim to
identify how participants exhibited flexibility in their programming solutions for mathematicalproblems. Taking our designed tasks and participants programming solutions as the unit ofanalysis, we conducted latent thematic analysis, which developing the themes necessitatesinterpretive effort and the resulting analysis is not just descriptive, but is already theorized(Braun & Clarke, 2006). Our analysis was implemented following the six steps of doing
thematic analysis outlined by Braun and Clarke (2006).

4. 濆濙濧濩濠濨濧
By analyzing and generalizing participants’ problem-solving strategies in solving CT-based
mathematical problems, the results of the thematic analysis yielded two main kinds offlexibility (i.e., translation flexibility and integrated flexibility) and six sub-categories.
Translational flexibility refers to a cross-disciplinary, cross-problem context capability thatinvolves translation between mathematical and programming language, with the twoprocesses being independent. In contrast, integrated flexibility is an overall capability similar185to flexibility in a mathematical problem context (e.g., to solve an algebra equation; Star &Seifert, 2006). It is the flexibility to solve problems in a programming environment thatinvolves the use of knowledge from both disciplines, with the MT and CT realizations being
inseparable as a whole. Due to the length of the article, we will only cover integrated
flexibility in detail with a geometry example.
One of the most evident examples of integrated flexibility is the setup of subroutines,
often known as creating My Blocks in Scratch. As an important CT concept, subroutines canbe divided into two types. One is the parameterless subroutine, which calls an entire piece ofcode directly and repeatedly as a whole, without altering it before or after the subroutine
function is invoked, but rather given it a separate name for wrapping (Figure 1a-b). This typeof subroutine does not require programmer to make any adjustments to the codes. Insteadof executing the original code directly, another kind of subroutine with parameters requires
programmers to comprehend the section of codes in its entirety, identify the key attribute andnumerical relationships between them, and then modify the codes by abstracting parameters
to be a subroutine (Figure 1c). In this section, we will focus on subroutines with parameters
and the related integrated flexibility will be examined through a geometry drawing task.

(a)

(b)
(c)
(d)
Figure 1. Setting of My Blocks as Subroutine in Scratch

As we know, to draw a regular polygon in paper-and-pencil context, we first need toknow the polygon’s side length, the degree of interior angles, then to draw it with a ruler anda protractor. While to achieve this mathematical process in Scratch, the Pen function is used
to record the trajectory of a Sprite. For example, to draw a regular pentagon with side lengthof 50, the pen needs to [move 50 steps, turn clockwise 72 degrees] then repeat this process
five times (Figure 1a). The code at this point does not yet make use of subroutine, but if
programmers try to use My Block to simplify the programming process or design a programthat can change the size and shape of the polygon at any time, mathematical considerationis required for setting parameters in the subroutine (Figure 1c-d).
It is worth noting that the process of changing the original codes (Figure 1a) to a pieceof code with subroutine (Figure 1c-d; Figure 2a-b) requires the involvement of both MT and
CT. The interplay between CT and MT is evident in this integration, particularly in the
different strategies from Figure 1c-d to Figure 2a-b. For example, programmers can abstracta parameter “side length” in the My Block and replace all the numbers or variables in thesource code that represent the length of the sides to create a subroutine served to control
the size of a pentagon (Figure 1c). Here, we can see that such a change is mediated by both
CT and MT because one may have to figure out which variable represent the side length thatcan be a parameter in the subroutine. CT support programmers to create the subroutine,186

while MT helps determine the first parameter in the subroutine, i.e., the size of the polygon isdetermined by the length of the sides. However, when the programmer went on to optimize
the subroutine so that it can also change the shape of the polygon, it does not mean thatmathematical property “different polygons have different interior angles” makes it possible to
add a second parameter “exterior angle” directly to the My Block in Figure 1c. Otherwise,even if the second parameter “exterior angle” is added to the My Block (Figure 1d), it will not
be possible to draw any other regular polygon than a regular pentagon, because only withthe parameter “polygon 50 72” will the matching angle and number of sides can form an
enclosed polygon. This is because the number of sides and the degree of interior angles
determine the shape of the polygon, while the subroutine in Figure 1c only contains codes of[move 50 steps, turn clockwise 72 degrees] five times which does not have the function tochange number of sides that follow the change of degree of exterior angle. The key to solve
such problems is the effective coordination of CT and MT.

(a)(b)
Figure 2. Setting of My Blocks as Subroutine
Furthermore, although the subroutine in Figure 2a with three parameters (i.e.,“side_length”, “exterior_angle”, “number_of_sides”) can change the size and shape of theregular polygon, there actually exists a specific quantitative relationship between the degreeof exterior angle and the number of sides, i.e., degree of exterior angle = 360/number ofsides, so the My Block in Figure 2a can be optimised to a subroutine with only two
parameters affecting the shape (number_of_sides) and size (side_length) of the polygon in
Figure 2b. This reveals the synergy between CT and MT in the process of setting up
subroutines with parameters. Thus, we can observe that during the process of setting
subroutines, the key concepts of CT (subroutines, loops) and MT (mathematical properties)
constrain and facilitate each other: When problem solvers want to set up flexible subroutinesthat can change the shape and size of regular polygons, CT concepts such as subroutinesand loops play a key role in supporting the adaptation of the program, while quantitative
relationships in mathematics involved in MT responsible for decision making on parameters
setting in the subroutine, the choice of using which CT concepts and how they are used.
That is, the number of sides determine the number of loops; the parameters of thesubroutines are the attributes that control size and shapeȹthe length and the number ofsides, respectively. It is in the combined effect of CT and MT that provide opportunity for
programmers to abstract the key parameter that determines the shape of a regular polygon.
We have named such an integrative programming process as “integrated flexibility” in which
CT and MT co-exist and support each other.

5. Discussion
Overall, translation flexibility is a cross-disciplinary, cross-problem context competence,
while integrated flexibility is a holistic capability similar to flexibility in mathematical problem

187contexts (Star et al., 2022; Star & Seifert, 2006), that is, the general flexibility to solve
problems in a programming environment that includes knowledge from both mathematics
and programming disciplines. It involves a complex interaction between MT and CT which
different from the one-way conversion of translation flexibility. In general, integrated flexibility
is more often found in the process of selecting a CT technique when the problem solver hasa clear mathematical solution. It is vital to take into account which mathematical approach
and which CT technique are compatible. If the two do not match, adjustments must be madeto the mutual circumstance in order to develop an effective programming solution.
We believe such a conceptual framework has potential for understanding the mutual
cognitive development in CT and MT, including how mathematical concepts co-emerge with
programming processes in a problem-solving context, and the role of CT in such processes.
We understand that because programming is a tool-mediated activity, different programming
tools will be instrumented differently, necessitating various CT-MT translations andinteractions. It is hoped that this research will contribute to a deeper understanding of theinterplay of CT and MT as a basis for conceptualizing an interdisciplinary form of STEM
education and thinking on task design, pedagogy, and assessment approaches of CT-based
mathematics instruction. Moreover, the findings will offer theoretical contributions that informresearchers and mathematics educators in terms of student learning characteristics during
CT-based mathematics instruction. Finally, we have noticed that our data from two different
participant groups exhibit some comparable features of flexibility, which may point to acertain universality of problem solvers’ flexibility in programming-based mathematicalproblems. As a concluding remark of the paper, we pose several questions that arise fromour exploration—Given the existence of multiple solutions, are the prevailing programming
solutions simpler or more complex to solve the problem? Why are these approaches morecommon? What factors are relevant to them? How are the problem solvers’ flexibility to use
different types of translations affected by their CT and MT? We call for more research toconsider the different kinds of flexibility demonstrated in CT-based mathematical problem
solving to sketch the big picture of how CT and MT emerge in complementary or
mismatching ways.

References
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77–101.
Burton, L. (1984). Mathematical thinking: The Struggle for meaning. Journal for Research in
Mathematics Education, 15(1), 35–49. https://doi.org/10.2307/748986
Cui, Z., & Ng, O. (2021). The interplay between mathematical and computational thinking in primaryschool students’ mathematical problem-solving within a programming environment. Journal of
Educational Computing Research, 59(5), 988–1012.
Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet
the trustworthiness criteria. International Journal of Qualitative Methods, 16(1),1609406917733847.
Ng, O., & Cui, Z. (2021). Examining primary students’ mathematical problem-solving in a
programming context: Towards computationally enhanced mathematics education. ZDM –Mathematics Education, 53(4), 847–860.
Rich, K. M., Strickland, C., Binkowski, T. A., Moran, C., & Franklin, D. (2018). K-8 learning trajectories
derived from research literature: Sequence, repetition, conditionals. ACM Inroads, 9(1), 46–55.
Star, J. R., & Seifert, C. (2006). The development of flexibility in equation solving. Contemporary
Educational Psychology, 31(3), 280–300.
Star, J. R., Tuomela, D., Joglar-Prieto, N., Hästö, P., Palkki, R., Abánades, M. Á., Pejlare, J., Jiang, R.
H., Li, L., & Liu, R.-D. (2022). Exploring students’ procedural flexibility in three countries.
International Journal of STEM Education, 9(1), 4.
Wing, J. (2011). Research notebook: Computational thinking—What and why. The Link Magazine, 6,
20–23.
Ye, H., Liang, B., Ng, O., & Chai, C. S. (2023). Integration of computational thinking in K-12
mathematics education: A systematic review on CT-based mathematics instruction and student
learning. International Journal of STEM Education. Doi:10.1186/s40594-023-00396-w

188

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Epistemic Network Analysis to assesscollaborative engagement in Knowledge
Building discourse
Aloysius Onga*, Chew Lee Teoa, Alwyn Vwen Yen Leea & Guangji YuanaaNational Institute of Education, Nanyang Technological University, Singapore*aloysius.ong@nie.edu.sg
Abstract: Knowledge Building (KB) is an established learning sciences theory thatseeks to promote innovative ideas and idea improvement among students via
collaborative engagement in productive discourse. KB discourse supports students tomake constructive discourse moves such as questioning, explaining with evidence,adding new information and so on, to advance the collective inquiry. However, current
understanding on KB discourse remains limited to students’ online participation.
Although small group discussion is a common practice, there is little understanding on
the role of verbal discussions to support KB discourse. This paper attempts to address
this line of inquiry by assessing student engagement in KB discourse supported by bothonline and verbal discussions. Data is retrieved from a group of six students in a Grade
6 Social Studies class. The group participated in a 2.5hr lesson designed with
opportunities for discussions on the Knowledge Forum (online) and in small groups(verbal). Group talk was transcribed, and Knowledge Forum notes were coded for its
semantic level of contribution, with the codes being analysed for weighted connections
using Epistemic Network Analysis (ENA). The ENA analysis revealed clear differences
in both group and individual engagement between the online and verbal discourse.
Notably, students’ contributions on Knowledge Forum showed an apparent pattern ofstronger connections among codes of higher semantic levels, suggesting that students
were more cognitively engaged in the online discussion than their group verbal talk.
Implications for research and practice are discussed.
Keywords: Knowledge Building, Discourse Analysis, Epistemic Network Analysis

1. Introduction
Knowledge Building (KB) is an established learning sciences theory that seeks to promote
innovative ideas and idea improvement among students via collaborative engagement inproductive discourse. Knowledge Building brings students together to work as a community totackle real-world problems and creates a discursive environment that supports students todevelop a deeper collective understanding of the problem at hand (Scardamalia & Bereiter,
2006). When a community of students is engaged in Knowledge Building discourse to advancecollective understanding, they contribute important moves such as questioning, explaining
with evidence, adding new information and so on, in order to advance the collective inquiry.
The understanding of what and how students contribute to improve ideas and their
engagement in KB discourse have been of interest among researchers. In fact, to understand
student engagement in KB discourse, various frameworks have been proposed to characterise
student discourse moves such as theory building moves (Chuy et al., 2010) and idea
complexity (Zhang et al., 2007). Recent works suggest quantitative modelling techniques suchas Epistemic Network Analysis (ENA) to assess connections between student moves (Ma etal., 2019). Insights from such analyses provide a means to understand the quality of student
ideas from discussions as well as how they develop collective inquiry (build on ideas).
However, current understanding on KB discourse remains limited to students’ online
participation. Although small group discussion is a common practice, there is little
understanding on the role of verbal discussions to support KB discourse. This paper attempts

189to address this line of inquiry by assessing student engagement in KB discourse supported byboth online and verbal discussions. Specifically, we aim to understand the similarities anddifferences in the students’ discourse moves and connections between their ideas as they
participate in online and verbal discussions in KB lessons.

2. Literature
2.1 Knowledge Building discourse
Knowledge building discourse is characterised by the generation of novel and challenging
student ideas, questions, and different perspectives that strive to improve the collective
understanding of the problem at hand. It focuses on the generative and creative nature of aclassroom discussion, moving away from a more typical and traditional classroom discussionthat focuses on getting the single correct answer (Ong et al., 2021; Teo et al., 2022). The goalof Knowledge Building classrooms is beyond information sharing and divide and conquer toput a product together (Scardamalia & Bereiter, 2006). The essence of knowledge buildingclassroom lies in the Knowledge Building discourse. In KB discourse, students are encouraged
to constantly question and refine their ideas collectively, find their motivation to connect withthe big problem and help each other progress in the inquiry and knowledge. KB discourse isusually supported by the Knowledge Forum which is an online platform to facilitate students
in posting and building on ideas (Scardamalia & Bereiter, 2006). Figure 1 shows a view of theKnowledge forum where students interact and contribute to the discussion together. Eachnode (square box) represents a note (contains ideas) from a student, and the arrows show
the build-on from one node to another. However, it is also common for students to work insmall groups and discuss verbally to improve ideas. Figure 1 shows a group of four students
in a face-to-face setting with computer devices to allow students to engage in both online andverbal discussions.

Figure 1. Face-to-Face and Online discussion modes in Knowledge Building

2.2 Understanding student engagement in KB discourse
Several ways of understanding student contributions in KB discourse have been proposed in
the field. One notable area of focus is on the characterisation of students’ ideas or discourse
moves (e.g., Chuy et al., 2010; Zhang et al., 2007; 2022). Notably, Chuy and colleagues
proposed the theory building moves to code student discourse moves. Specifically, they
developed an empirically grounded list of six major ways of contributing to a productive
dialogue for KB discourse. These categories comprise: asking thought-provoking questions,190

theorizing, experimenting, working with evidence, creating syntheses and analogies, andsupporting discussion. From another study, Zhang and colleagues (2007) looked atelementary students’ (Grade 4) contributions in KB discourse from the quality of their individual
ideas. They devised the idea complexity scheme to categorise students’ ideas. Questions may
be categorised into explanation-seeking versus fact-seeking questions and single-area versus
cross-area questions. According to Zhang et al. (2007), students’ contributions can also beassessed based on the level of scientific sophistication (pre-scientific; hybrid; basicallyscientific; and scientific) and the level of epistemic complexity (unelaborated facts; elaborated
facts; unelaborated explanations; elaborated explanations).
Another area of focus is on the epistemic understanding via patterns of how students
build on each other’s ideas to progress knowledge (e.g., Lin & Chan, 2018; Ong et al., 2021).
For instance, Lin and Chan introduced the thread analysis to code the quality of student online
discourse. The authors found and categorised different patterns of how students build on eachother’s ideas into four epistemic levels of increasing collective knowledge advancement. The
basic level, Fragmented Discussion, showed no connection or a weak connection between
student ideas which do not contribute to knowledge advancement. The next level, Knowledge
Sharing, showed connections between student ideas mainly for exchanging information andmay not be essential to solving the problem. Knowledge Construction represented a higher
level where there is a building of connections to generate questions and explanations toconstruct theories. Finally, Knowledge Building represents the highest epistemic level where
there are coherent connections between ideas to bring about new understanding andprogressive development of students’ theories. Lin and Chan’s work highlighted theimportance and need to consider idea connection when assessing student engagement in KB
discourse, however their coding scheme remains restricted by few epistemic levels.
Recent works suggest the potential of modelling techniques to understand connections
between student discourse moves in Knowledge Building. For instance, Ma and colleagues(2019) explored the Epistemic Network Analysis (ENA) to assess Knowledge Building
discourse, specifically looking into “ways of contributing” to the discourse such as questions
and theories. The authors argued that there is potential for ENA to model contribution
dynamics during cycles of knowledge advancement. Ma et al.’s study demonstrated thepotential of ENA as a feasible technique to identify similarities and differences in discourse
patterns between groups. For example, one group may differ from another in terms of certain
connections such as theorizing and questioning moves. As explained by Ma and colleagues,
such network models can be used to understand complementary engagement in group
discussions and indications of collaborative engagement in both online and verbal discourse.
While there is an extensive development of techniques to assess student contributions
in KB discourse, current understanding on KB discourse remains limited to students’ online
participation. Existing studies tend to focus on student participation in online discussions
supported by the Knowledge Forum. There is little understanding on the role of verbal
discussions to support KB discourse, even though it is a common practice among teachers toplan for small group discussions to allow students to talk about their ideas verbally (see Ong
et al., 2021). Elsewhere beyond KB contexts, there are studies showing that verbal
discussions do not necessarily compliments nor supports online discourse. For example, arecent study by Bagheri and Zenouzagh (2021) highlighted that students engaged differently
in face-to-face and online conversations. The researchers investigated the discourse from agroup of 30 student participants and they found differences in verbal, paralinguistic andfunctional level between the students’ face-to-face and online conversations (computermediated communication). Their study revealed that student engagement in online discourse
was more productive compared to the physical discourse as (i) the students showed active
control of the talk and engaged in richer exchanges of ideas such as argumentation and (ii)students have more tendency to appeal to the teacher for help and displayed silences andpauses in the physical discourse (Bagheri & Zenouzagh, 2021). As student engagement inboth online and verbal discussion may impact on the quality of their collective inquiry in KB, it
is thus important for ongoing research to delve deeper on this understanding.
In this paper, we hope to build on this line of inquiry by assessing students’ ways ofcontributing in both online and verbal discussion to better understand their engagement during

191Knowledge Building discourse. We report on the interactional patterns in verbal and online
discussions from a group of students engaged in KB collective inquiry over a 2.5hr lesson.
Specifically, we aim to understand the similarities and differences in the students’ discourse
moves and connections between their ideas as they participate in online and verbal
discussions in KB lessons by using Epistemic Network Analysis. Our research question
therefore is “What connections of student contribution can we see during verbal and online
discussion using ENA?”

3. Research design and Analysis
3.1 Methods
To capture students’ verbal interaction during face-to-face discourse, we used a 360-degree
camera placed at the center of the group to video-record the group interaction as shown in
Figure 4. We flattened the video data to enable us to see the interaction of each participant in
the group. To ensure high quality of verbal data is captured, we used another audio recorderwith lapel microphones for every individual student to capture their verbal data (Figure 2). This
setup was part a multimodal recording setup that also collected students’ self-reports on theiremotions and physiological data from Empatica E4 wearables. However, we focused on thedescription of the setup on the video and audio recording.

Figure 2. Recording setup for f2f and online discussion in classroom

3.2 Lesson design and implementation
In this case study, we collaborated with a teacher (experienced in knowledge building) to
implement a lesson designed with Knowledge Building activities to support collective inquiry
and idea improvement. Table 1 shows the teacher’s planned lesson activities that engaged
students to conduct idea generation and idea building about a real-world problem (based onSocial Studies). Students will be tasked to play the roles of locals situated in three Southeast
Asian areas (Sarawak Rainforest, Mekong River, and Anak Krakatoa) and they will continue
to discuss the pros and cons of relocation and to provide their final stand with justifications.
Before the lesson, the students were pre-assigned to one of the three geographicalregions. Each group was tasked to discuss and find solutions to existing problems (habitat
loss, active volcano, and deforestation) that threaten the respective residents’ livelihoods andenvironments. Table 1 shows the flow of the activities that were planned to provide

192opportunities for student discussion both online and face-to-face. A total of 17 Primary Five
(Grade Five) students participated in this lesson that lasted for 2.5 hours.
Table 1. Activities from the teacher’s lesson plan
Teacher’s instructions and class activities
Lessonphase

Students’
activities

KF

Activating
Activityභ Students played the roles of locals in three No KF activities
Southeast Asian areas (Sarawak Rainforest, recorded
Mekong River, and Anak Krakatoa).
භ Students introduced to the KB task, which is a
Town Hall discussion anchored on the question:“Should our community stay or relocate from our
area?”

Idea
Generation

භ Students discussed the anchor question in theirrespective groups.
භ Students posted notes on KF stating their standand with supporting reasons.

Discuss and post
different
perspective (stand)with reasons on KF

Idea
Improvement– Build On

භ Students discussed and built on their peers’ ideason the Knowledge Forum using build-on scaffolds.
භ The teacher used learning analytics such as word
clouds and an in-built Scaffold Tracker to conduct
a meta-discussion and highlight value-adding
contributions.

Read and build ongroupmates’
viewpoints, such as
clarifying, inquiring,or proposing newelaborations.

Idea
Improvement– Rise Aboveභ Teacher introduced a Rise-Above scenario:
Students took on newer roles as community
representatives to potentially implement a solution
to the respective area’s issues.

Idea
Assessment
Reflection

Negotiate as agroup,
weighingpros and cons ofdifferent solutions
todevelopasolution.
භ Students consolidated their problems and Post synthesis ofsolutions that were then shared with the class for group ideas andpeer feedback forcomments and suggestions.
improvement.
භ Students responded to three reflection questions Post reflection onKF
on KF.

3.3 Analysis
In this study, we used the ENA Web Tool (Shaffer et al., 2009; Shaffer et al., 2016) to analyseboth Knowledge Forum data and verbal discussion data. ENA models the weighted structureof connections in discursive data (Shaffer & Ruis, 2017). The method was designed to makesense of connections between ideas and actions using a repertoire of network-based methods
that include visualizations and statistical modelling. For example, if a student group is
discussing a particular topic, they share important elements such as the areas of their interest,
their questions, information that they know, and so on. They may have a few conversations
during the lesson, and a key part of understanding their discourse is modelling how they think
about the connections between the different elements of their inquiry. ENA quantifies the cooccurrence of, for example, codes in discourse or elements of interaction in a chat andproduces a weighted network visualization that shows connections for each unit of analysis(e.g., individual speakers or sub-groups of speakers) (Shaffer & Ruis, 2017). This paper
reports the preliminary analysis with one student group (six students).

193

Based on 42 Knowledge Forum notes from the group, we coded for student
contribution based on five levels of semantic patterns (adapted Zhang et al., 2022). These
included casual talk (CT), fact-seeking question (Q1), explanation-seeking question (Q2),explanation with simple points or simple statement without elaboration (E1), and explanation
with elaborations or rise-aboves (E2). Table 2 provides the description and sample coding.
We use each student idea as the unit of analysis. This is typically in the form of a question or
an explanation (statements). Where a student note or utterance contains a question andexplanation, we coded for both respectively. We coded student contribution from their verbal
talk as well. In total, we captured approximately 1.5hr of talk time from the student group andthis data was transcribed (2416 utterances) and coded with the same coding scheme and unitof analysis. In total, we coded 51 codes from the KF notes and 2548 codes from the verbal
data.
We next applied Epistemic Network Analysis (ENA) to identify and quantify
connections between these codes for the group. We created the epistemic networks by firstly
using the discourse type (KF and verbal) as units for ENA to construct networks. The two
networks were aggregated using a binary summation to reflect the presence or absence ofthe co-occurrence of each pair of codes within a stanza window of 6 lines in the conversations.
The conversations are collections of lines that ENA uses to model connections between
concepts, and this was based on the student utterances or KF notes in our analysis. Finally,codes are considered concepts and its patterns of association are what we want to model. In
our case, we applied all the five semantic levels of the student contribution.
Table 2. Description and examples of semantic patterns of KF notes
Codes of semantic patterns Description/Examples
Casual Talk (CT)

Information or inputs generally irrelevant to the collective
inquiry, e.g., “I need the toilet.”Fact-seeking question (Q1)

Questions that elicit factual information, using when, where,
or who, e.g., “Where do you want to go?”Explanation-seeking
question (Q2)

Questions which inquire into relations between facts andelicit elaborations, such as why, how, and what, e.g., “Whyshould we be affected by other people living in the modern
world?”

Explanation with simple
points/Simple statement
without elaboration (E1)

Simple statement without elaboration, e.g., “You arecontradicting yourself. These are the exact reasons why weshould move.”

Explanation withelaborations/rise-aboves(E2)

Statement with elaborations to provide reasons,relationships, or synthesis of ideas, e.g., “We don't reallyhave a government. Without a government, there is no law,no order. And with a government, we suffer under adictatorship. People who are weaker will abuse power.”

4. Findings
Answering the research question “What connections of student contribution can we see duringface-to-face and online discussion using ENA?”, Figure 3 shows the epistemic network
between the group’s verbal talk and KF discussion throughout the entire lesson. The positionsof the network graph nodes are fixed, and the nodes correspond to the codes applied in theanalysis. ENA determined these positions using an optimization routine that minimises thedifference between the plotted points and their corresponding network centroids (Shaffer etal., 2016). The lines reflect the relative frequency of co-occurrence, or connection, between

194two codes. The weight of each code is represented by the size of the node in the network
graph. The results show that the group demonstrated a relatively unique profile between thetwo discourse types. Student contributions from their KF discussion shows centrality on
elaborated questions (node with highest weight) and strongest connections between this codewith explanations (simple and elaborated). Student contributions from their verbal discussion
however shows centrality on simple explanation and strongest connections with casual talk(thickest line) and questions (simple and elaborated) than elaborated explanations. This
preliminary analysis from this group suggest that students seem to engage in a deeper
discussion of their ideas when they interact on KF compared to their verbal interaction (despitethat they were given time to discuss verbally).

a)b)
Figure 3. Epistemic networks showing connection of codes (semantic levels) for (a) verbal
and (b) online discussion from the student group.
This finding was corroborated when we looked at the distribution of codes for the verbal
and online discussion for the group. As shown in Figure 4, there was also a considerable
difference in the distribution of codes between the two discourse modes. Student contributionsin their KF discussion showed highest occurrence of simple explanation followed byelaborated explanations and then elaborated questions and lower occurrences of casual talkand simple questions. Student contributions in verbal discussion also showed highest
occurrences of simple explanations but followed by casual talk and lower occurrences of thecodes of higher semantic levels (elaborated explanations and elaborated questions).
100%
80%
60%
40%
20%
0%
Verbal data (n=2548)

KF data (n=51)

CasualTalk

SimpleQuestion

ElaboratedQuestion

ElaboratedExplanation

SimpleExplanation

Figure 4. Comparison of codes (semantic levels) between verbal and online discussion fromthe student group.
However, we found interesting observations when we further explore the epistemic
network of individual students. As shown in Figure 5, 3 students in the group showed
connections between the codes for their online discussion for the group but their network
profile were not totally consistent. Specifically, two students, S1 and S5, showed strongest

195connections between elaborated question and elaborated explanation (thickest line), whilestudent S4 showed strongest connections between elaborated question and simple
explanation and question. This result suggests that S1 and S5 were consistently using higher
semantic levels like elaborated questions and explanations to deepen the collective inquiry.
Note that the ENA did not picked up connections for the other students (as these students
contributed very few notes).

S1)
S4)
S5)
Figure 5. Epistemic networks showing connection of codes (semantic levels) for online
discussion from individual students in the group.
Similarly, their individual network profile was also different in the verbal discussion.
Figure 6 shows connections between the codes for their verbal discussion. This time, students
S1 and S4 showed strongest connections between elaborated question and simple
explanation, whereas students S1 and S5 displayed strongest connections between
elaborated question and elaborated explanation (thicker line). The other three students S2,
S3, and S6 showed weaker connections between these codes but strongest connection
between simple explanation and casual talk. Thus, students S1, S4 and S5 seemed moreengaged in a deeper discussion of their ideas during their verbal discussion, which is aconsistent find with their contributions on KF discussion.

S1)

S4)

S2)

S3)

S5)

S6)

Figure 6. Epistemic networks showing connection of codes (semantic levels) for verbal
discussion from individual students in the group.

5. Discussion and conclusion
Our preliminary finding from the ENA analysis is consistent with those by Ma et al. (2019) as
we found that students deepened their collective inquiry as they engaged in Knowledge
Building discourse. As shown in our analysis, this group of students reflected a strong
connection between higher semantic levels of contributions elaborated questions andelaborated explanations when given time to discuss on KF. However, we also found different
patterns of their contributions during verbal discussion with the ENA analysis. Specifically, intheir group verbal talk, there was a strong presence of casual talk and connections between

196

simple explanations and questions which suggest that the collective inquiry was more
superficial during verbal discussions. This claim that students engage more productively in
online communication is consistent with the report from Bagheri and Zenouzagh (2021).
However, we think that such a result may also be due to the affordances of the online KF
platform such as the support of KB scaffolds and the chance to read (even multiple times) andconsider multiple peer ideas. Unlike KF, students tend to observe turn-taking during verbal
discussion, which may restrict them in responding to certain ideas and giving longer responses
such as elaborated explanations. Therefore, future work can consider looking into the impact
of different support features in both online and verbal discussion.
An interesting finding was the interactional patterns from individual students. The ENA
analyses suggested that students who were engaged in a deeper discussion of their ideas on
KF also contributed ideas of higher semantic levels during their verbal interaction. This finding
has an implication for practitioners particularly on the lesson design and facilitation. For
example, teachers can first plan for an online discussion to allow students to generate
questions or explanations of higher semantic levels and then allowing them to continuebuilding their ideas via small group discussions. As it is easy to extract student notes from theKF, teachers can also run the data using the ENA tool and use the analysis to discuss withstudents on how their ideas are connected and ways to improve the quality of their talk. Such
teacher facilitation has been shown to be beneficial for students to develop new inquiry (Zhu
& Lin, 2023). However, we recognise that our findings were based on a very small dataset andour claims warrant further investigation with more data.
Moving forward, we have collected new datasets, and additional analyses areunderway to compare findings from other groups and to validate the claims. In conclusion, thisstudy highlights that the understanding of student engagement in KB discourse is notstraightforward, as student interactions can occur within and between groups as well as via
online or verbal discussion. As we found that the type of and connections between student
discourse moves were less sophisticated in their verbal discussions, a key implication fromthis study is that students need more support during verbal talk to engage in deeper collective
inquiry and advancing the community knowledge. Our study also shows ENA as a promising
approach to assess Knowledge Building discourse in both online and verbal mode. ENA
affords modelling of both group and individual networks to enable the understanding ofdifferences in student engagement during KB discourse. Future studies can explore on theautomation of such feedback for teachers and students. For example, the real-time capturing
and coding of verbal data which is integrated with the ENA tool to generate feedback on the
discourse quality from the verbal discussion.

Acknowledgements
This study was funded by the Ministry of Education (MOE), Singapore, under the Education
Research Funding Programme (OER 19/19 TCL) and administered by National Institute of
Education (NIE), Nanyang Technological University, Singapore. The views expressed in thisstudy are the authors’ and do not necessarily represent the views of the host institution. The
authors thank the teacher and student participants involved in this study. The authors also
express their appreciation to Associate Professor Justin Dauwels and his team for expert
advice and support on the multimodal recording setup.

References
Bagheri, M., & Mohamadi Zenouzagh, Z. (2021). Comparative study of the effect of face-to-face andcomputer mediated conversation modalities on student engagement: speaking skill in focus. AsianPacific Journal of Second and Foreign Language Education, 6(1), 1-23
Chuy, M., Resendes, M., & Scardamalia, M. (2010, August). Ways of contributing to knowledge building
dialogue in science. Paper presented at the Knowledge Building Summer Institute, Toronto,
Canada.

197

Lin, F., & Chan, C. K. (2018). Promoting elementary students’ epistemology of science throughcomputer-supported knowledge-building discourse and epistemic reflection. International Journalof Science Education, 40(6), 668-687.
Ong, A., Lee Teo, C., Tan, S., & Song Kim, M. (2021). A knowledge building approach to primaryscience collaborative inquiry supported by learning analytics. Education 3-13, 49(3), 371-386.
Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K.
Sawyer (Ed.), Cambridge Handbook of the Learning Sciences (pp. 97-118). New York: Cambridge
University Press.
Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing thestructure of connections in cognitive, social, and interaction data. Journal of Learning
Analytics, 3(3), 9-45.
Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., ... & Mislevy, R. (2009).
Epistemic network analysis: A prototype for 21st-century assessment of learning. International
Journal of Learning and Media, 1(2), 33-53.
Shaffer, D., & Ruis, A. (2017). Epistemic network analysis: A worked example of theory-based learning
analytics. Handbook of learning analytics.
Teo, C. L., Ong, A., & Lee, V. Y. A. (2022, June). Exploring students’ epistemic emotions inknowledge building using multimodal data. In Weinberger, A., Chen, W., Hernandez-Leo,D., and Chen, B. (Eds.), Proceedings of the 15th Computer-Supported Collaborative
Learning (CSCL) 2022 (pp. 266-273). Hiroshima, Japan: International Society of the
Learning Sciences. https://2022.isls.org/proceedings/
Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Socio-cognitive dynamics ofknowledge building in the work of 9-and 10-year-olds. Educational Technology Research andDevelopment, 55, 117-145.
Zhang, J., Tian, Y., Yuan, G., & Tao, D. (2022). Epistemic agency for costructuring expansive
knowledge-building practices. Science Education.
Zhu, G., & Lin, F. (2023). Teachers scaffold student discourse and emotions in knowledge buildingclassrooms. Interactive Learning Environments, 1-18.
Ma, L., Matsuzawa, Y., & Scardamalia, M. (2019). Using epistemic network analysis to ExploreWays ofcontributing to knowledge building discourse. In Proceedings of the 13th international conferenceon computer-supported collaborative learning (pp. 881–882). Lyon, France: International Societyof the Learning Sciences.

198

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education)0!-.%#.%)#,/-.1*,.$%)!--) *)"'%.%)%-.*,%'/'.%+'!!2.- ,*(3!,&%)#.*"*/,!)  *).!).,*!--%)#
$!)#*)#/)/))3%)
%5-10%.$%0+-0+,-%1"60+#0-7)34-5:"%-8%0EG@@K>;@ @5FG9?3;> 5A?34

-.,. )@>;@7D736;@9;EA@7A8F:7EAGD57EFA35CG;D7=@AI>76974GF>73D@7DE?3K
7@5AG@F7D 5A@8>;5F;@9 F7JFE 6GD;@9 D736;@9  .:GE >73D@7DE @776 FA 6;EF;@9G;E: F:7
EAGD57TE FDGEFIADF:;@7EE 3@6 67F7D?;@7 I:;5: B7DEB75F;H7E FA 47>;7H7  "AI7H7D
BD7H;AGEEFG6;7E:3H78A5GE76?AD7A@F:7EAGD57F:3@A@F:75A@F7@F3@6?AD7A@
E5;7@57F:3@A@:;EFADK;@7K7FD35=;@98;7>6E .:7D78AD7F:7BGDBAE7A8F:;EEFG6K;EFA
;@H7EF;93F7 :AI D7367DE B7D57;H7 5A@8>;5F 3@6 BDA57EE EAGD57 3@6 5A@F7@F I:7@
D736;@9:;EFAD;53>?G>F;B>7F7JFEI;F:H3DK;@9679D77EA8FDGEFIADF:;@7EE  B3DF;5;B3@FE
I7D7D75DG;F763@6BD7E7@F76E;J:;EFAD;53>F7JFEG@67DF:D776;887D7@F5A@6;F;A@E:;9:
FDGEFIADF:;@7EE >AIFDGEFIADF:;@7EE 3@6 FDGEFIADF:;@7EE6;887D7@57E  .:7;D 7K7
?AH7?7@F I3E D75AD676  .:7 D7EG>FE ;@6;53F76 F:3F F:7 D7367DE D3F76 F:7 >AIFDGEFIADF:;@7EE5A@6;F;A@3E:3H;@9?AD75A@8>;5FF:3@F:7AF:7DE 66;F;A@3>>KF:7>AIFDGEFIADF:;@7EE 5A@6;F;A@ :36 >7EE 8;DEFB3EE D736;@9 F;?7 F:3@ F:7 FDGEFIADF:;@7EE
6;887D7@57E5A@6;F;A@ ,7367DEEB7@F?AD7D7D736;@9F;?7A@F:75A@F7@FF:3@A@F:7
EAGD57 A@>K ;@ F:7 FDGEFIADF:;@7EE6;887D7@57E 3@6 >AIFDGEFIADF:;@7EE 5A@6;F;A@E 
1:7@D736;@9F7JFI;F:>AIFDGEFIADF:;@7EEF:7D7367DEB7@F?AD78;DEFB3EED736;@9
F;?7A@F:7EAGD57F:3@A@F:75A@F7@F3@6EB7@F?AD7D7D736;@9F;?7A@F:75A@F7@F
F:3@ A@ F:7 EAGD57  .:7E7 8;@6;@9E :3H7 E7H7D3> ;?B>;53F;A@E  ;DEF>K D7367DE I7D7
AH7DI:7>?76 3@6 @77676 9G;63@57 A@ I:;5: ;@8AD?3F;A@ FA 47>;7H7 >736;@9 FA F:7
B7D57BF;A@A8?AD75A@8>;5FI:7@D736;@9>AIFDGEFIADF:;@7EE5A@6;F;A@E -75A@6F:7
D7367DEI7D73I3D7F:3FF:7EAGD57E77?76G@FDGEFIADF:K>736;@9F:7?FA675D73E7
D736;@9 F;?7 FA BDA57EE F:7 8A>>AI;@9 5A@F7@F  8F7D 5A@8;D?;@9 F:7 EAGD57E
FDGEFIADF:;@7EE D7367DE F7@676 FA D7D736 F:7 5A@F7@F FA D7EA>H7 F:7 5A@8>;5F  &3EF>K>AIFDGEFIADF:;@7EE EAGD57E 675D73E76 BDA57EE;@9 F;?7 ;@ F:7 8A>>AI;@9 5A@F7@F;@6;53F;@9 F:3F D7367DE D79G>3F7 F:7;D D736;@9 B3FF7D@E 67B7@6;@9 A@ F:7 F7JFE
FDGEFIADF:;@7EE )H7D3>>F:;EEFG6KBDAH;67E;@E;9:FE;@FA:AID7367DEBDA57EE:;EFAD;53>
?G>F;B>7F7JFEI;F:H3DK;@9679D77EA8FDGEFIADF:;@7EE 
!31*, - ?G>F;B>7F7JF7K7FD35=;@95A@8>;5F;@9:;EFAD;53>FAB;5).,* /.%*)

E F75:@A>A9K 36H3@57E B7AB>7 53@ 97F ?G5: 4GF 5A@8>;5F;@9 ;@8AD?3F;A@ A@ F:7#@F7D@7F .A5A?BD7:7@63@;EEG7677B>KD7367DE@776FAD736?AD7F:3@A@7F7JFFAH3>;63F7
F:7 FDGEFIADF:;@7EE A8 F:7 ;@8AD?3F;A@ !A>6?3@  -53D63?3>;3 $3DA6L=3  D3@6
!DGI7> 1:7@D736;@9?G>F;B>7F7JFED7367DE@776FAE7>75FBDA57EE3@67H3>G3F7
F:7D7>7H3@FA@7EFA8AD??73@;@98G>?7@F3>D7BD7E7@F3F;A@DNF7@-FDS?ES #@F:75A@F7JFA8?G>F;B>7F7JFGE7D736;@9EFD3F79;7E3D73EEA5;3F76I;F:7H3>G3F;@9
F:7FDGEFIADF:;@7EED33E5:DNF7@ 1:7@D736;@95A@8>;5F;@9;@8AD?3F;A@D7367DE
B7D8AD? 47FF7D ;8 F:7K 7H3>G3F7 F:7 EAGD57E 3GF:ADE:;B ;@F7@F 3@6 D7>3F76 388;>;3F;A@@?3D=DG6 7F 3>  &;EF  >7J3@67D  "AI7H7D 5A@F7@F BDA57EE;@9 ;E 7CG3>>K
7EE7@F;3>4GFE77?EFA47@79>75F76;@F:77K7FD35=;@98;7>6-3>?7DO@7F3>  1:7@
<G69;@9FDGEFIADF:;@7EED7367DEE:AG>6D7>KA@4AF:F:7EAGD573@65A@F7@FE;?G>F3@7AGE>K199

3DL;>3; 7F 3>   .:7D78AD7 F:7 3;? A8 F:;E EFG6K ;E FA 7JF7@6 BD7H;AGE D7E73D5: 4K
;@H7EF;93F;@9F:778875FA8FDGEFIADF:;@7EE6;887D7@57E5A@F7@F3@6EAGD57A@7K7?AH7?7@F
BDA57EE7EI:7@D736;@95A@8>;5F;@9;@8AD?3F;A@ 

%.!,./,!!0%!1 A@8>;5F;@9;@8AD?3F;A@;@?G>F;B>7F7JFE.:7 ;@8AD?3F;A@ 47FI77@ F7JFE 53@ 47 3 5A?B>7?7@F3DK D76G@63@F AD 5A@8>;5F;@9
D7>3F;A@E:;B3@6 F:7 5A@8>;5F;@9 ;@8AD?3F;A@ :3E 477@ F:7 8A5GE A8 ?G5: D7E73D5: 3FF7@F;A@
*D;?AD%3FL;D *D7H;AGED7E73D5::3EBDAH76F:3FD7367DEB3K3FF7@F;A@FAF:7EAGD57
;@8AD?3F;A@3@6?7?AD;L7F:7?D33E5:7F3> 
%3??7D7D7F3> ,AG7F7F3>  D33E5:3@6DNF7@
BDABAE76F:76;E5D7B3@5K;@6G576EAGD57?A67>#-?A67> FA 7JB>3;@ F:;E B:7@A?7@A@  .:7 ;@5A:7D7@57 BDAHA=76 4K 5A@8>;5F;@9 ;@8AD?3F;A@>736ED7367DEFA8A5GEA@EAGD57;@8AD?3F;A@FA7JB>3;@I:KF:75A@8>;5F:3BB7@76D7EG>F;@9;@677B?7?ADK7@5A6;@9A8F:7EAGD57;@8AD?3F;A@ 
1:7@7@5AG@F7D;@95A@8>;5F;@9;@8AD?3F;A@7H3>G3F;@9F:7EAGD57TEFDGEFIADF:;@7EE;E
A@7A8F:73BBDA35:7EFAD75A@5;>;@9F:;EE;FG3F;A@  AD;@EF3@57DNF7@7F3> ;@EFDG5F76
D7367DEFAD736H3D;AGEF7JFEA@5>;?3F75:3@978DA?6;887D7@FEAGD57E;@5>G6;@9F7JF4AA=E
@7IEB3B7DE ?393L;@7E 3@6 A88;5;3> 6A5G?7@FE  .:7 D7367DE D3F76 F7JF4AA=E 3@6 A88;5;3>
6A5G?7@FE3EFDGEFIADF:K3@6B3;6?AD73FF7@F;A@FAF:7E7EAGD57EF:3@AF:7DE !AFFE5:>;@9
7F3> D7CG;D76F:77JB7D;?7@F3>9DAGBFAD736FIA5A@8>;5F;@9E5;7@F;8;5F7JFEA@7ID;FF7@
4K3BDA87EEADIAD=;@98AD35A?B3@K3@6F:7AF:7D4K3G@;H7DE;FKBDA87EEADI:;>735A@FDA>
9DAGB D736 FIA E5;7@F;8;5 F7JFE ID;FF7@ 4K G@;H7DE;FK BDA87EEADE  .:7;D D7EG>FE D7H73>76 F:3F
D7367DED3F76F:7F7JF8DA?F:75A?B3@K3E>7EEFDGEFIADF:KF:3@F:7F7JF8DA?F:7G@;H7DE;FK *D7H;AGED7E73D5:A@F:;EFAB;5E77?EFA8A5GE?AD7A@E5;7@F;8;5F:3@:;EFAD;53>;EEG7EI:;5:
@7768GDF:7D;@H7EF;93F;A@  K7FD35=;@9;@?G>F;B>7F7JFD736;@9K7FD35=;@9 :3E 477@ GE76 FA ;@H7EF;93F7 F:7 ?A?7@FFA?A?7@F 3FF7@F;A@3>
6;EFD;4GF;A@,3K@7D ,7E73D5:7DEGEG3>>K678;@7F:7EAGD57;@8AD?3F;A@3EF:73D73A8
;@F7D7EFFA7J3?;@7D7367DET3FF7@F;A@I:7@D736;@9F:75A@8>;5F;@9;@8AD?3F;A@D33E5:7F3> !AFFE5:>;@97F3> %3??7D7D7F3>   AD7J3?B>7%3??7D7D7F3> 
7JB>AD76I:7F:7DD7367DEB3;63FF7@F;A@FABDA57EE;@9F:7EAGD57;@8AD?3F;A@I:7@D736;@9
5A@8>;5F;@9;@8AD?3F;A@A@I74B397E .:7D7EG>FD7H73>76F:3FD7367DE:36>A@97DFAF3>8;J3F;A@
6GD3F;A@A@F:7EAGD57;@8AD?3F;A@I:7@D736;@95A@8>;5F;@9I74B397EF:3@5A@E;EF7@FA@7E !AFFE5:>;@97F3> GE768;DEFB3EED736;@9F;?73@6D7D736;@9F;?7FA7J3?;@7D7367DET
5A9@;F;H7 BDA57EE I:7@ D736;@9 5A@8>;5F;@9 ;@8AD?3F;A@  .:7;D 8;@6;@9E D7H73>76 F:3F F:7FDGEFIADF:;@7EE6;887D7@57E;@;@8AD?3F;A@53GE76D7367DEFAEB7@6?AD7F;?7D7D736;@9F:7
>AIFDGEFIADF:;@7EEEAGD57 .:7E7EFG6;7EEG997EFF:3FD7367DE3>>A53F73FF7@F;A@3>D7EAGD57EFAEAGD57;@8AD?3F;A@
I:7@BDA57EE;@95A@8>;5F;@9;@8AD?3F;A@ "AI7H7DF:76;E5GEE;A@34AGF5A@F7@FBDA57EE;@9
E77?E>7EE;@F:77K7FD35=;@98;7>6A8?G>F;B>7F7JFGE7  .:75GDD7@FEFG6K
.:75A@8>;5F;@9?G>F;B>7F7JFE?3=7D7367DE5A@57@FD3F7A@F:7EAGD57;@8AD?3F;A@8AD
D7EA>H;@9F:75A@8>;5F *D7H;AGEEFG6;7E:3H7477@5A@6G5F76I;F:E5;7@F;8;5F7JFE #F@776EFA
475>3D;8;76:AID7367DEBDA57EE?G>F;B>7:;EFAD;53>F7JFE7EB75;3>>K5A@E;67D;@9F:75A@F7@F
3@6 EAGD57E E;?G>F3@7AGE>K  .:7D78AD7 F:7 3;? A8 F:;E EFG6K ;E FA ;@H7EF;93F7 D7367DET
3FF7@F;A@3>6;EFD;4GF;A@I:7@D736;@9?G>F;B>7:;EFAD;53>F7JFEI;F:5A@8>;5F;@9;@8AD?3F;A@ .:7
FDGEFIADF:;@7EEA8F:7 EAGD57E I3E ?3@;BG>3F76 ;@FA F:D77 5A@6;F;A@E :;9:FDGEFIADF:;@7EE
>AIFDGEFIADF:;@7EE3@6FDGEFIADF:;@7EE6;887D7@57E5A@6;F;A@E .IAD7E73D5:CG7EF;A@EI7D7
BDABAE76200

,+ "AI5A@8>;5F6AD7367DEB7D57;H7I:7@D736;@96;887D7@F?G>F;B>7F7JF5A@6;F;A@E,+  "AI 6A D7367DE BDA57EE F:7 EAGD57 3@6 5A@F7@F D78>75F76 8DA? F:7 7K7
?AH7?7@F63F3I:7@D736;@96;887D7@F?G>F;B>7F7JF5A@6;F;A@E

!.$*   *3DF;5;B3@FE
#F I3E 3@ 7JB>AD3FADK EFG6K 3@6 A@>K  B3DF;5;B3@FE I7D7 D75DG;F76 8DA? F:7/@;H7DE;FK ;@ (ADF:7D@ .3;I3@  >> F:7 B3DF;5;B3@FE B3EE76 F:7 53>;4D3F;A@E  .:;E EFG6K I3E
5A@6G5F76I;F:3I;F:;@835FAD;3>EG4<75F67E;9@ H7DKB3DF;5;B3@FI3E;@EFDG5F76FAD7363>>
5A@6;F;A@E .:7BD7E7@FAD67DI3E5AG@F7D43>3@57635DAEEF:7B3DF;5;B3@FE   '3F7D;3>E.:7 B3DF;5;B3@FE I7D7 9;H7@ E;J F7JFE D793D6;@9 F:7 E3?7 :;EFAD;53> FAB;5  .34>7 
BD7E7@F7667F3;>76;@8AD?3F;A@34AGFF:7E7F7JFE7;F:7D:;9:FDGEFIADF:;@7EE?73@;@9F:3FF:7
EAGD57I3E8DA?F:79AH7D@?7@FAD>AIFDGEFIADF:;@7EE?73@;@9F:3F3@3@A@K?AGEB7DEA@
B3D3B:D3E76 F:7 EAGD57  .:D77 F7JFE E:3D76 F:7 E3?7 H;7IBA;@FE I:;>7 F:7 AF:7D F:D77
ABBAE76F:AE7H;7IBA;@FE 35:F7JF5A@E;EF76A8FIAB3D39D3B:EEAGD573@65A@F7@F .:7
EAGD57I3EBAE;F;A@76478AD7F:75A@F7@F3EBD7E7@F76;@ ;9GD7 .:7E;JF7JFEI7D7B3;D76
;@FA F:D77 5A@6;F;A@E :;9:FDGEFIADF:;@7EE >AIFDGEFIADF:;@7EE 3@6 FDGEFIADF:;@7EE
6;887D7@57E .IAF7JFEI7D76;EB>3K76A@F:7E5D77@3FF:7E3?7F;?7 
.34>7 7)37-)81*5,)4-95)954

(A .DGEFIADF:;@7EE
0;7IBA;@FE(G?47DA8IAD6E;@F:7
EAGD57 5A@F7@F.DGEFIADF:;@7EE
6;887D7@57E


&AI
";9:


 

'G>F;B>7F7JF5A@6;F;A@&AI
FDGEFIADF:;@7EE


&AI
&AI



 

 

 

";9:
FDGEFIADF:;@7EE


";9:
";9:


 

 




-+63).:76;EB>3KA8F7JF5A@6;F;A@   BB3D3FGE3D73A8;@F7D7EF3@67K7?AH7?7@F?73EGD7E
.:77K7?AH7?7@FEI7D7D75AD6764K3@K7&;@=
67E=FABD7?AF77K7FD35=7DEKEF7?-,,7E73D5:&F6 3@363I;F:3E3?B>7D3F7A8
"L .:76;EF3@5747FI77@F:7
?A@;FAD3@6F:7B3DF;5;B3@FEI3E 5? 35::;@7E75:3D35F7DI3E V B;J7>E3@6F:7
EB35747FI77@>;@7EI3E B;J7>E  ;9GD7 BD7E7@F768AGD)#E3D73EA8;@F7D7EF678;@76;@ 735: FD;3> FIA EAGD57E 3@6 FIA 5A@F7@F  .IA 7K7?AH7?7@F ?73EGD7E I7D7 8;DEFB3EE

201

D736;@9F;?7 *,.3@6D7D736;@9F;?7,,.  *,.;EF:7EG?A8F:76GD3F;A@EA83>>8;DEF
B3EE8;J3F;A@E478AD7>73H;@9F:7)# ,,.;EF:7EG?A8F:76GD3F;A@EA83>>D7FGD@;@98;J3F;A@E
38F7D 8;DEFB3EE 8;J3F;A@E A@ F:7 )#  .A 5A@FDA> F:7 >7@9F: A8 B3D39D3B:E 7K7?AH7?7@F
?73EGD7EI7D76;H;6764KF:7@G?47DA85:3D35F7DE-F36F>7D7F3>    '73EGD7E

8F7DD736;@9735:5A@6;F;A@F:7B3DF;5;B3@FEI7D7;@EFDG5F76FA3@EI7DF:D77CG7EF;A@E
GE;@93BA;@F&;=7DFE53>7 )@7A8F:7CG7EF;A@E;@CG;D7634AGFF:7E7@E7A85A@8>;5F47FI77@
F:7FIAF7JFEI:;>7F:7AF:7DFIACG7EF;A@ED787DD76FA<G69;@9F:7FDGEFIADF:;@7EEA8735:F7JF   *DA576GD7
.:7 BDA576GD7 ;@ F:7 7JB7D;?7@F 5A@E;EF76 A8 FIA E75F;A@E  #@ F:7 8;DEF E75F;A@ F:7
B3DF;5;B3@FEI7D7BDAH;676I;F:5A@5;E7;@EFDG5F;A@EA@F:77K7FD35=;@9BDA576GD7 .:7KI7D7
BAE;F;A@76;@8DA@FA83?A@;FADI;F:35:;@D7EFFA3HA;6:736?AH7?7@FE .:7B3DF;5;B3@FE
I7D753>;4D3F76I;F:@;@7BA;@FE #@F:7E75A@6E75F;A@B3DF;5;B3@FEI7D7;@EFDG5F76FAD736
F:D77 5A@6;F;A@E  #@ 735: 5A@6;F;A@ F:7 F7JFE I7D7 5A@8>;5F;@9 I;F: 735: AF:7D  )@57 F:7
B3DF;5;B3@FE8;@;E:76735:5A@6;F;A@F:7KI7D7D7CG;D76FAD3F7735:F7JFTEE7@E7A85A@8>;5F
3@6FDGEFIADF:;@7EE 8F7DF:75A?B>7F;A@A8F:7F3E=EF:7B3DF;5;B3@FEI7D76;E?;EE76 

!-/'.
  .:7D3F;@9A8FDGEFIADF:;@7EE3@6E7@E7A85A@8>;5F
.A3@EI7D,+ I7GE76D7B73F76?73EGD7FF7EFFA7J3?;@7F:7FDGEFIADF:;@7EE;@
735: 5A@6;F;A@ 3@6 D7B73F76 ?73EGD7E ()0 FA 7J3?;@7 F:7 6;887D7@57E 47FI77@ F:7
?G>F;B>7F7JF 5A@6;F;A@E ;@ F:7 E7@E7 A8 5A@8>;5F  .:7 ?73@E 3@6 EF3@63D6 67H;3F;A@E ;@
B3D7@F:7E7E3D7BD7E7@F76;@.34>7 .:7 D7EG>F A8 F:7 FF7EF E:AI76 3 E;9@;8;53@F 6;887D7@57 ;@ F:7 FDGEFIADF:;@7EE
6;887D7@57E 5A@6;F;A@ ;@6;53F;@9 F:7E75A@6 F7JF:36:;9: FDGEFIADF:;@7EEF:3@ F:7 8;DEF F7JF5      2  66;F;A@3>>K F:7 :;9:FDGEFIADF:;@7EE 5A@6;F;A@ :36 :;9:7D
FDGEFIADF:;@7EEF:3@F:7>AIFDGEFIADF:;@7EE5A@6;F;A@5  2  .:7 D7EG>FE A8 ,'()0 E:AI76 F:3F F:7 E7@E7 A8 5A@8>;5F 6;887D76 E;9@;8;53@F>K
35DAEE F:7 ?G>F;B>7F7JF 5A@6;F;A@E       2      .:7 BAEF:A5
5A?B3D;EA@E E:AI76 F:3F F:7 >AIFDGEFIADF:;@7EE 5A@6;F;A@ ?367 D7367DE B7D57;H7 ?AD7
5A@8>;5FF:3@F:7:;9:FDGEFIADF:;@7EE5A@6;F;A@5 2 I:;>7AF:7D5A?B3D;EA@E
I7D7@AFE;9@;8;53@F .34>7 )%04%0(!*135,)536458135,-0)44%0(4)04)1*'10*.-'5-0)%','10(-5-10

(A .DGEFIADF:;@7EE
-7@E7A85A@8>;5F

.DGEFIADF:;@7EE
6;887D7@57E5A@6;F;A@



 
    'G>F;B>7F7JF5A@6;F;A@E
&AIFDGEFIADF:;@7EE
";9:FDGEFIADF:;@7EE
5A@6;F;A@
5A@6;F;A@




 

 
 
 
    
 



  K7?AH7?7@F?73EGD7E
.A3@EI7D,+ I73@3>KL76FIA7K7?AH7?7@F?73EGD7EGE;@9>;@73D?;J76?A67>
&'' I;F: >?7 B35=397 3F7E 7F 3>   #@ F:;E EFG6K B3DF;5;B3@FE 3@6 F7JFE I7D7
D793D676 3E D3@6A? 78875FE  .IA 8;J76 78875FE I7D7 F:7 ?G>F;B>7F7JF 5A@6;F;A@E I;F: F:D77
>7H7>E FDGEFIADF:;@7EE6;887D7@57E >AIFDGEFIADF:;@7EE 3@6 :;9:FDGEFIADF:;@7EE 3@6 F:7202

B3D39D3B:E I;F: FIA >7H7>E EAGD57 3@6 5A@F7@F  .IA 67B7@67@F H3D;34>7E I7D7 8;DEFB3EE
D736;@9 F;?7 3@6 D7D736;@9 F;?7  .34>7  BD7E7@FE F:7 ?73@E 3@6 EF3@63D6 67H;3F;A@E ;@
B3D7@F:7E7E8AD7K7?AH7?7@F?73EGD7E;@735:5A@6;F;A@ .34>7 )%0%0(!*13):)/17)/)05/)%463)4-0)%','10(-5-101*5)954


'G>F;B>7F7JF5A@6;F;A@E

*3D39D3B:E

.DGEFIADF:;@7EE
6;887D7@57E5A@6;F;A@-AGD57
A@F7@F

&AIFDGEFIADF:;@7EE
5A@6;F;A@-AGD57 A@F7@F;DEFB3EE
D736;@9F;?7     



,7D736;@9F;?7 





 


 


 
  ";9:FDGEFIADF:;@7EE
5A@6;F;A@-AGD57
A@F7@F               

.:7D7EG>FA88;DEFB3EED736;@9F;?7E:AI76F:3FF:7>AIFDGEFIADF:;@7EE5A@6;F;A@:36
>7EE8;DEFB3EED736;@9F;?7F:3@F:7FDGEFIADF:;@7EE6;887D7@57E5A@6;F;A@&  ! 5 2  )F:7D5A?B3D;EA@EI7D7@AFE;9@;8;53@F .:7 D7EG>F A8 D7D736;@9 F;?7 E:AI76 F:3F F:7 ?3;@ 78875F A8 F:7 B3D39D3B:E I3EE;9@;8;53@F&  ! 5  2 ;@6;53F;@9F:3FF:7D7367DEEB7@F?AD7F;?7
A@F:75A@F7@FF:3@A@F:7EAGD57 "AI7H7DF:;E78875FI3E?A67D3F764KF:7?G>F;B>7F7JF
5A@6;F;A@E4753GE7A8F:7FIAI3K;@F7D35F;A@78875F&  ! 5 2  -+63) .:7;@F7D35F;A@78875FA8?G>F;B>7F7JF5A@6;F;A@E3@6B3D39D3B:EA@D7D736;@9
F;?7 
#@ ;9GD7  BAEF:A5 5A?B3D;EA@E E:AI76 F:3F D7367DE EB7@F ?AD7 F;?7 A@ F:7
5A@F7@FF:3@A@F:7EAGD57A@>K;@F:7FDGEFIADF:;@7EE6;887D7@57E5A@6;F;A@& ! 5  2 3@6F:7>AIFDGEFIADF:;@7EE5A@6;F;A@&  !  5 2  .:7E78;@6;@9EI7D7;@5A@E;EF7@FI;F:BD7H;AGEEFG6;7EI:;5:D7H73>76F:3FI:7@
D736;@9FIA5A@8>;5F;@9F7JFED7367DEIAG>6E:;8FF:7;D3FF7@F;A@FAF:7EAGD57FAD75A@5;>7F:7
5A@8>;5F D33E5: 7F 3>  %3??7D7D 7F 3>   17 EB75G>3F76 F:3F D7367DE IAG>6
D79G>3F7F:7;D67BF:A8D736;@9B3FF7D@E43E76A@F:7EAGD57TEFDGEFIADF:;@7EE #8D7367DEI7D7
3I3D7A8F:7G@FDGEFIADF:KF7JFED7367DE?;9:F675D73E7F:7BDA57EE;@9F;?7;@F:78A>>AI;@9
5A@F7@F .AF7EFF:;E:KBAF:7E;EI7?7D976F:7E7F:D775A@6;F;A@EFA3@3>KL7F:7E7E;JF7JFE
I;F: 7;F:7D :;9: AD >AI FDGEFIADF:;@7EE 3@6 735: F7JF 5A@F3;@76 FIA B3D39D3B:E  #@ AF:7D
IAD6E FIA ;@67B7@67@F H3D;34>7E FDGEFIADF:;@7EE 3@6 B3D39D3B:E 3@6 FIA 67B7@67@F
H3D;34>7E8;DEFB3EED736;@9F;?73@6D7D736;@9F;?7I7D73@3>KL76 #@ ;9GD7   F:7 D7EG>F A8 8;DEFB3EE D736;@9 F;?7 >78F E;67 E:AI76 F:7 ?3D9;@3>>K
E;9@;8;53@F ;@F7D35F;A@ 78875F &   ! 
 5    2    .:7 BAEF:A5
5A?B3D;EA@E D7H73>76 F:3F I:7@ D736;@9 F:7 F7JFE I;F: >AI FDGEFIADF:;@7EE D7367DE EB7@F?AD7F;?7A@F:7EAGD57F:3@A@F:75A@F7@F&  ! 5 2 I:;>7F:;E

203

78875FI3E@AF8AG@6;@D736;@9F:7F7JFI;F::;9:FDGEFIADF:;@7EE&  ! 52  .:7D7EG>FA8D7D736;@9F;?7D;9:FE;67E:AI76F:7E;9@;8;53@F?3;@78875FA8
B3D39D3B:E;@6;53F;@9F:3FD7367DEEB7@F?AD7F;?7A@F:75A@F7@FF:3@A@F:7EAGD57& !  5 2  .:7;@F7D35F;A@78875F3>EAD735:763?3D9;@3>>KE;9@;8;53@F
>7H7>&  !   5 2  .:7BAEF:A55A?B3D;EA@EE:AI76F:3FI:7@
D736;@9F:7F7JFEI;F:>AIFDGEFIADF:;@7EED7367DEEB7@F?AD7F;?7A@F:75A@F7@FF:3@A@F:7
EAGD57&  !  5 2
I:;>7F:;E78875FI3E@AF8AG@6;@D736;@9
F:7F7JFEI;F::;9:FDGEFIADF:;@7EE& !  5 2  -+63) .:7;@F7D35F;A@78875FA8F7JFEFDGEFIADF:;@7EE3@6B3D39D3B:EA@8;DEFB3EE
D736;@9F;?7>78FE;673@6D7D736;@9F;?7D;9:FE;67  *)'/-%*)

.:7 BD7H;AGE EFG6;7E :3H7 8A5GE76 ?AD7 A@ F:7 EAGD57 F:3@ A@ F:7 5A@F7@F!AFFE5:>;@9  %3??7D7D !AFFE5:>;@9 7F 3>  :AI7H7D F:7 5A@F7@F ;E 3>EA
7EE7@F;3>8AD8AD?;@93@7>34AD3F;H76A5G?7@F?A67>*7D87FF;7F3>  -3>?7DO@7F3>  .:7D78AD7F:79A3>A8F:;EEFG6K;EFA;@H7EF;93F7D7367DET3FF7@F;A@3>6;EFD;4GF;A@I:7@
D736;@93:;EFAD;53>FAB;5I;F:5A@8>;5F;@9B;757EA8;@8AD?3F;A@H3D;76I;F:FDGEFIADF:;@7EE .:D77 ?3;@ 8;@6;@9E I7D7 6D3I@ 8DA? F:7 3@3>KE;E  ;DEF>K D7367DE D3F76 F:7 >AIFDGEFIADF:;@7EE5A@6;F;A@3E:3H;@9?AD75A@8>;5FF:3@F:7AF:7D5A@6;F;A@EEG997EF;@9F:3F
F:7KI7D7AH7DI:7>?763@6G@EGD7I:;5:;@8AD?3F;A@FA47>;7H7:7@57F:7KB7D57;H76?AD7
5A@8>;5F .:7AF:7D5A@6;F;A@EBDAH;676D7367DEI;F:3F>73EFA@7F7JFFA47>;7H7I:;5:>7EE7@76
D7367DEE7@E7A85A@8>;5F  GFGD7D7E73D5:E:AG>6;@H7EF;93F7:AID7367DE<G6975A@8>;5F -75A@6F:77K7?AH7?7@F3@3>KE;E;@6;53F76F:3FF:7>AIFDGEFIADF:;@7EE5A@6;F;A@:36>7EE8;DEFB3EED736;@9F;?7F:3@F:7FDGEFIADF:;@7EE6;887D7@57E5A@6;F;A@;?B>K;@9F:3F
D7367DE I7D7 3I3D7 F:3F F:7 EAGD57 E77?76 >7EE FDGEFIADF:K F:7D78AD7 F:7K 675D73E76
D736;@9F;?7FABDA57EEF:78A>>AI;@95A@F7@F .:;EFD7@6I3E3>EA7H;67@F;@F:7?73@A88;DEF
B3EE D736;@9 F;?7 ;@ .34>7  3>F:AG9: F:7 6;887D7@57 I3E @AF E;9@;8;53@F  66;F;A@3>>K F:7
D7EG>F A8 D7D736;@9 F;?7 D7H73>76 F:3F D7367DE EB7@F ?AD7 F;?7A@ F:7 5A@F7@F F:3@ A@ F:7
EAGD57EB3DF;5G>3D>KI:7@D736;@9F:7FDGEFIADF:;@7EE6;887D7@573@6F:7>AIFDGEFIADF:;@7EE
5A@6;F;A@E .:;EEG997EFEF:3FD7367DEF7@6FAD7D736F:75A@F7@F38F7D5A@8;D?;@9F:7EAGD57E
FDGEFIADF:;@7EEFAD7EA>H7F:75A@8>;5F .:;D6>K I7 ?7D976 F:7E7 F:D77 5A@6;F;A@E ;@FA E;J F7JFE I;F: 7;F:7D :;9: AD >AIFDGEFIADF:;@7EE 178AG@6F:3FA@>K>AIFDGEFIADF:;@7EEF7JFE53GE76D7367DEFA;@5D73E78;DEF
B3EE D736;@9 F;?7 BDA57EE;@9 F:7 EAGD57 3@6 D7D736;@9 F;?7 BDA57EE;@9 F:7 5A@F7@F  .:;E
78875FI3E@AF8AG@6;@D736;@9:;9:FDGEFIADF:;@7EEF7JFE #@AF:7DIAD6E>AIFDGEFIADF:;@7EE
EAGD57E >76 FA 675D73E7 BDA57EE;@9 F;?7 ;@ F:7 8A>>AI;@9 5A@F7@F ;@6;53F;@9 F:3F D7367DE
D79G>3F7 F:7;D D736;@9 B3FF7D@E 67B7@6;@9 A@ F:7 F7JFE FDGEFIADF:;@7EE  .:;E 7JB>3@3F;A@
75:A76F:7E75A@68;@6;@93@6E:AG>6475A@8;D?764K8GFGD7D7E73D5: -A?7 >;?;F3F;A@E E:AG>6 47 @AF76  ;DEF>K A@>K B3DF;5;B3@FE I7D7 D75DG;F76 
55AD6;@9>K EA?7 EF3F;EF;53> D7EG>FE A@>K D735:76 ?3D9;@3> E;9@;8;53@57  .:7D78AD7 F:7
E3?B>7 E:AG>6 47 ;@5D73E76 FA 5A@8;D? F:7 EF3F;EF;53> D7EG>FE  -75A@6>K I7 6;6 @AF 5A>>75F
D736;@9 5A?BD7:7@E;A@ AD ?7?ADK 63F3 I:;5: ;E 7EE7@F;3> ;@ F:7 6A5G?7@F ?A67>

204

7?B:3E;L;@9 F:7 >;@= 47FI77@ F:7 EAGD57 3@6 F:7 5A@F7@F *7D87FF; 7F 3>    GFGD7
D7E73D5:E:AG>67H3>G3F7D7367DETD736;@95A?BD7:7@E;A@ #@EG??3DKF:;EEFG6KBDAH;67E
7?B;D;53>7K7?AH7?7@F63F33@6;@E;9:F;@FA:AID7367DEBDA57EEEAGD57E3@65A@F7@F;@3
:;EFAD;53>FAB;5  GFGD7D7E73D5:E:AG>65A@F;@G7FA7JB>AD7F:7E7;EEG7E !"!,!)!-
@?3D=DG6R DNF7@# -FDS?ES" #  'G>F;B>76A5G?7@FE>;F7D35K-FD3F79;5BDA57EE;@9
EAGD573I3D7@7EE3@63D9G?7@F3F;A@I:7@D736;@9?G>F;B>75A@8>;5F;@96A5G?7@FE )%30-0+
%0(0(-7-(6%.-**)3)0')4  :FFBE 6A; AD9
 < >;@6;8
3DL;>3;- .:A?? -:>A?;>AAL.  73>;@9I;F:6;E39D77?7@F.:7DA>7EA8FAB;5
83?;>;3D;FK3@66;E39D77?7@F7JB>3@3F;A@;@7H3>G3F;A@A85A@8>;5F;@97JB7DF5>3;?E3@6
EAGD57E )%30-0+%0(04536'5-10  :FFBE 6A; AD9
 < >73D@;@EFDG5
3F7E 'M5:>7D' A>=7D 13>=7D-   ;FF;@9>;@73D?;J7678875FE?A67>E/E;@9>?7 1630%.1*!5%5-45-'%.!1*58%3)   :FFBE 6A; AD9 <EE H  ; D33E5:$ & ,AG7F$  0;47DF( D;FF'   ,7367DEGE7A8EAGD57;@8AD?3F;A@;@F7JF
5A?BD7:7@E;A@ )/1+0-5   :FFBE 6A; AD9 E    
D33E5:$ & ! DNF7@#  .:7;E5D7B3@5K#@6G576-AGD57A?BD7:7@E;A@#-'A67>3E;5EEG?BF;A@E3@6*D7>;?;@3DKH;67@57 (6'%5-10%.4:',1.1+-45   :FFBE 6A; AD9     
DNF7@# -FDS?ES" #  1:7@>3IEFG67@FED736?G>F;B>76A5G?7@FE34AGF9>A43>
I3D?;@97J3?;@;@9F:7DA>7A8FAB;5EB75;8;547>;78E34AGFF:7@3FGD7A8=@AI>76973@6
=@AI;@9 04536'5-10%.!'-)0')  :FFBE 6A; AD9E    
DNF7@# -FDS?ES" # -3>?7DO@&  .DGEF3@6?;EFDGEFI:7@EFG67@FED736?G>F;B>7
;@8AD?3F;A@EAGD57E34AGF5>;?3F75:3@97 )%30-0+%0(04536'5-10      :FFBE 6A; AD9
 < >73D@;@EFDG5!A>6?3@- , -53D63?3>;3'  '3@39;@9/@67DEF3@6;@9BB>K;@93@6D73F;@9%@AI>7697;@F:7#@8AD?3F;A@97(7JF!7@7D3F;A@:3>>7@97E3@6)BBADFG@;F;7E 1+0-5-10
%0(04536'5-10     :FFBE 6A; AD9    !AFFE5:>;@9- %3??7D7D2  ,7367DET,79G>3F;A@3@6,7EA>GF;A@A83-5;7@F;8;5A@8>;5F
3E76A@;887D7@57E;@-AGD57#@8AD?3F;A@@K7.D35=;@9-FG6K -4'1634)31')44)4 :FFBE 6A; AD9J  

!AFFE5:>;@9- %3??7D7D2 !7D<7FE*  ,7367DET*DA57EE;@93@6/E7A8-AGD57
#@8AD?3F;A@3E3 G@5F;A@A8#FE/E78G>@7EEFAJB>3;@A@8>;5F;@9-5;7@F;8;5>3;?E -4'1634)
31')44)4   :FFBE 6A; AD9J    $3DA6L=3" D3@6!DGI7>-  .D35=;@9F:7D736;@97K7FAI3D6E3?A67>A8D73>IAD>6
D736;@9 1630%.1*1/265)344-45)()%30-0+  :FFBE 6A; AD9<53>

%3??7D7D2 %3>487>> !7D<7FE*  #E.:;E#@8AD?3F;A@-AGD57A??7D5;3>>K;3E76
"AIA@FD36;5F;A@E7FI77@174*397E-F;?G>3F7F:7A@E;67D3F;A@A8-AGD57#@8AD?3F;A@ -4'1634)31')44)4   :FFBE 6A; AD9J 
&;EF >7J3@67D*   .AI3D63@#@F79D3F76 D3?7IAD=A8'G>F;B>7.7JF/E7 (6'%5-10%.4:',1.1+-45    :FFBE 6A; AD9    *7D87FF;  ,AG7F$  D;FF'    .:75A@EFDG5F;A@A8?73F3>D7BD7E7@F3F;A@E6GD;@9
D736;@9 #@- , !  #@" H3@)AEF7@6ADB6 "18%3(4%5,)13:1*(1'6/)054
3)23)4)05%5-10BB  '3:I3:($&3ID7@57D>43G?  
*D;?AD& %3FL;D.  '73EGD;@9'G>F;B>7.7JF#@F79D3F;A@,7H;7I 3105-)34-0
4:',1.1+:  :FFBE 6A; AD9 8BEK9 
,3K@7D%  K7?AH7?7@FE3@63FF7@F;A@;@D736;@9E57@7B7D57BF;A@3@6H;EG3>E73D5: 
924:',1.17)     :FFBE 6A; AD9    ,AG7F$  -3GJ! ,AE -F36F>7D' 0;47DF( D;FF'   #@E;67A5G?7@F'A67>E,A>7A8-AGD57FFD;4GF7E;@,7367DET#@F79D3F;A@A8'G>F;B>7.7JFA@F7@FE -4'1634)
31')44)4    :FFBE 6A; AD9J 

-3>?7DO@& !;>& DNF7@# 
 /E;@97K7FD35=;@9FA3EE7EEEAGD5;@96GD;@9?G>F;B>7
6A5G?7@FD736;@95D;F;53>3@3>KE;E *3105.-0))%30-0+ )4)%3', :FFBE 6A; AD9 8>D H; 
-F36F>7D' -5:3DD7D& DA??7,  "AI,7>7H3@578875FE/@67DEF3@6;@9A8A@8>;5FE
7FI77@'G>F;B>7A5G?7@FE@K7.D35=;@9-FG6K  )%(-0+ )4)%3',6%35)3.:   :FFBE 6A; AD9
DDC  
205

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Multimodal analysis oflearners’ communications in CSCL
of a mathematical proof
Masataka KANEKOa*, Hironori EGIb & Takeo NODAc
Faculty of Pharmaceutical Sciences, Toho University, Japanb
Graduate School of Informatics and Engineering, The University of ElectroCommunications, Japanc
Faculty of Science, Toho University, Japan*masataka.kaneko@phar.toho-u.ac.jpaAbstract: In this study, we explore the CSCL of a mathematical proof using HTMLbased dynamic geometry content. In addition to the learners’ transcribedcommunication, multimodal log data of their manipulations of the content andutterances are analyzed. First, we use multimodal data to segment the whole learning
process into stages and identify the characteristics of those stages. This segmentationis then aligned with the transcribed communication. Our finding is that the participants’choice of words varies according to each stage and is compatible with thecharacteristics derived from multimodal data. Resultingly, when and how theconvergent conceptual change occurred is inferred from the commognitive perspective.
Keywords: Mathematical proof, dynamic geometry, log data of manipulations, log dataof utterances, text mining of annotated dialogue

1. Introduction
While mathematicians construct proofs as part of their professional work, students also
construct them in mathematics classes. In both situations, some scientific discovery is
involved and the convergent conceptual change leading to it is often investigated throughethnomethodological studies (Garfinkel, 2002). One puzzle related to the investigation ofscientific discovery lies in how is it that we can talk about the matter being discovered whilestill engaged in the work of discovering it (Koschmann, & Zemel, 2009). Moreover, especiallyin the case of the activities in classroom, the account of the discovery tends to become
highly complex due to the learners’ lack of domain-specific knowledge and related
vocabulary. In their previous work (Kaneko, Egi, & Noda, 2022), the authors investigatedlearners’ collaborative activity of proving the addition theorem for trigonometric function while
manipulating HTML-based content generated using the dynamic geometry system CindyJS(https://cindyjs.org/). For that investigation, a Moodle plug-in of CindyJS was used to store
the log data of learners’ manipulations (Kaneko, Nakahara, & Noda, 2020). Moreover, theconversation analysis system Diana-AD was also used to record learners’ utterances andmeasure the loudness of their voices (Ishikawa, Okazawa, & Egi, 2019). One of the main
findings was that high-resolution multimodal log data derived from them could serve as wellgrounded material for inferring where to “locate a proposal for a possible discovery” in thewhole learning process (Koschmann, & Zemel, 2009). Based on that finding, in this study,
we explore another group’s learning in a similar setting and discuss how to align themultimodal log data with the transcribed communication. In the case of this proof activity,
learners can be assumed to experience some commognitive conflict while generalizing theirprior knowledge to newly emerging situations. Their participation in some discursive
processes (metadiscourse) to handle the relevant commognitive conflict can lead to thereification of those processes through which increasingly sophisticated approximations to thetargeted concept are constructed (Sfard, 2009). Since this reification can be assumed to206cause some change in the pattern of both manipulation and discourse, we first segment thewhole learning process according to the change in their manipulating pattern and the power
balance between each participant’s utterances. Then, we examine the difference in eachparticipant’s choices of words and their associated meanings among the segmented parts.

2. Research Methods and Results
Figure 1 (left) shows the content used in which the angles ߙ(red) and ߚ(green) can be
moved. Two first-grade female students with a pharmaceutical science major in a Japanese
university were asked to manipulate the content collaboratively on an iPad. Beforehand, they
were asked to fill in the worksheet in Figure 1 (center and right) to prove the theorem for the
rudimentary case where ߙ + ߚ was acute. The time they needed to do that was measured
and one participant (A) was assumed to have higher prior skill than the other (B).

Figure 1. The CindyJS content (left) and the worksheet (right) including the figure (center)
While Figure 2 (left) shows the fluctuations of the three angles ߙ(blue), ߚ(orange),and ߙ + ߚ(grey), the yellow curve in Figure 2 (right) shows the power balance of utterance
between A (lower domain) and B (upper domain). It can be seen from the left figure that,during 2:30-13:00, almost no manipulation was made. In fact, a videotaped image of theparticipants’ activities tells us that almost all of their motions were pointing to the figures inboth worksheet and iPad during this time interval. Thus, we divide the whole process into
three stages, Stage I (0min00sec - 2min30sec), Stage II (2min30sec – 13min00sec), andStage III (13min00sec – 17min00sec).

Figure 2. The fluctuation of angles (left) and the power balance of utterance (right)
Aligning these results with the transcribed communication, it can be seen that A played theleading role in the search for a probable positional relationship between angles while
manipulating the content at Stage I. Then, at Stage II, B gradually increased her presence
while comparing the situation they encountered with the rudimentary case on the worksheet.
At this stage, A became dominant only once between 6min00sec and 8min00sec when A
presented some assumptions. Finally, at Stage III, A again became dominant when she
claimed the orthogonality of ɳBHO, which is one of the key points.
To examine the participants’ choice of words, the author used the text mining
software KHCoder (https://khcoder.net) which is specialized for the Japanese language.
Some previous studies pointed out that a seemingly vague pronoun, which does not refer toany prior antecedent but is a prospective indexical, might leave room for later elaboration
leading to a new understanding (Koschmann, & Zemel, 2009). Therefore, we calculated thefrequency of the use of demonstrative pronouns (DP) and common nouns (CN) for eachparticipant and stage. Figure 3 visualizes the resulting cross table in which the sizes of the207rectangles represent the incidence of sentences including DP (upper row) and CN (bottomrow) among all ones in each category. As is seen, while B used DP throughout the wholeprocess, A gradually replaced the use of DP with the use of CN.

Figure 3. Visualization of each participant’s choice of words

3. Discussion and Future Works
While most of the DP referred to some specific antecedents, there do exist some DP without
any clear referent. For instance, after B said cautiously at 14min44sec, “The length of OH is,let me see, ܿ‫ ?ߚ ݏ݋‬I am not sure, but it will be the same,” A responded at 15min03sec,“Anyway, our task should be to check whether that still holds or not when the angles arechanged.” It can be judged from the context that the pronoun “that” in this utterance of A wasused not to refer to some specific fact but to propose the possibility of literally imitating thedescriptions in the rudimentary case. Thus, this pronoun use clearly indicates that thepreceding discourse had been reified into some newly proposed “discovery”.
Because of the limited number of cases examined, the findings in this study cannot
be generalized to other cases directly. In fact, in another case study where the prior skill of
the participants was fairly higher than that in this study, the utterances and manipulations of
one participant were almost always dominant and the other participant only listened to her
partner, occasionally throwing in an appropriate word or two. Therefore, many more cases
should be explored to search for the possible communication patterns in this proof activity.
Moreover, some situations in which “actions speak louder than words” were observed. The
fine-grained log data of learners’ gestures should be synchronized and analyzed.

Acknowledgements
This work was supported by JSPS KAKENHI 19K03175, 21K03175, and 22K02951.

References
Garfinkel, H. (2002) Ethnomethodology’s program: Working out Durkheim’s aphorism. Rowman &Littlefield.
Koschmann, T., & Zemel, A. (2009). Optical pulsars and black arrows: Discoveries as occasioned
products. The Journal of Learning Sciences, 18(2), 200–246.
Kaneko, M., Egi, H., Noda, T. (2022) Analysis of CSCL for mathematical proof based on the log dataof learners’ verbal and nonverbal communications. Proceedings of ICCE’22 (pp. 167-169).
Kaneko, M., Nakahara, T., Noda, T. (2020) Temporal analytics of log data derived from students’manipulating math objects. Companion proceedings of LAK20, 86–88.
Ishikawa, N., Okazawa, T., Egi, H. (2019) Diana-AD: Dialog analysis system for adjusting duration onreal time in discussion. Lecture notes in computer science, 11677, 212–221.
Sfard, A. (2008) Thinking as communicating: Human development, the growth of discourses, andmathematizing. Cambridge University Press.

208" !   !" " "   #" !
#" !"$  #" ! #"            
      

- 0/!$**'*"%!)!) !$)*'*#4)%1!-.%/4*".0&0+))./%/0/!*"4./!(.) )"*-(/%*))#%)!!-%)#)%1!-.%/4*".0&0+):3DG=3 F3638F> ;;F FEG=G43 35 <B   >;;01>A0C8E4 ;40A=8=6 8= ?A>6A0<<8=6 43D20C8>= 8B :=>F= C> 70E4
43D20C8>=0; 14=458CB 0=3 8B =>F ?A02C8243 F>A;3F834 >F4E4A C74 @D0;8CH >5
2>;;01>A0C8E4;40A=8=6E0A84B34?4=38=6>=C744GC4=CC>F7827C40274AB20=BD??>AC8C
DAA4=C;H C40274AB 0A4 >5C4= C>> 1DBH C> ?A>E834 BD558284=C BD??>AC 5>A 2>;;01>A0C8E4
;40A=8=6'74A45>A48=C78BBCD3HF45>2DB43>=>=45>A<>5BD??>AC;40A=4A<0C278=6
0=334E4;>?430<0C278=6BHBC4<C70CDB4B4<>C8>=A42>6=8C8>=A4BD;CB5A><;40A=4ABP=0CDA0; B?4427 *4 2>=3D2C43 2><?0A0C8E4 4G?4A8<4=CB F8C7 4G8BC8=6 <4C7>3B C>
2;0A85H F74C74A 6A>D? 5>A<0C8>= 10B43 >= C74 A4BD;CB >5 74C4A>64=48CH 8= 4<>C8>=
A42>6=8C8>= 8B <>A4 45542C8E4 C70= C74 4G8BC8=6 <4C7>3 '74 A4BD;CB BD664BC C70C C74?A>?>B43<0C278=6BHBC4<70BC74?>C4=C80;C>1A8=6?>B8C8E445542CBC>C7>B4;40A=4AB
F8C7 ;>F4A 5>D=30C8>=0; B:8;;B = 0338C8>= 8= BD??;4<4=C0AH BDAE4HB C74 <09>A8CH >5
?0AC828?0=CBA4?>AC43C70CC74H5>D=38C40B84AC>;40A=F8C7C748A<0C2743?0AC=4ABDB8=6
C74 ?A>?>B43 <4C7>3 C7DB BD??>AC8=6 C74 7H?>C74B8B C70C C74 ?A>?>B43 <4C7>3 8B
<>A445542C8E4C70=C744G8BC8=6<4C7>3 >;;01>A0C8E4 40A=8=6& &'!3D20C8>=$A>6A0<<8=63D20C8>=
4 40A=8=6 !"!
,:7;?BADF3@57A8E5;7@57F75:@A>A9K7@9;@77D;@93@6?3F:7?3F;5E+,&76G53F;A@;E;@5D73E;@93@6BDA9D3??;@976G53F;A@;E47;@9BDA?AF7635F;H7>K3DAG@6F:7IAD>6+M7L%NB7L7F3>  )DA9D3??;@976G53F;A@:3E477@E:AI@FA;?BDAH7EFG67@FES5A9@;F;H7
E=;>>EEG5:3E5D73F;H;FKD73EA@;@93@6?3F:7?3F;53>E=;>>E3@65A>>34AD3F;H7>73D@;@9;E
D75A??7@676;@EG5:76G53F;A@+5:7D7D7F3>  A>>34AD3F;H7>73D@;@9;E3F735:;@9?7F:A6;@I:;5:EFG67@FE3FH3D;AGE>7H7>E5AAB7D3F7;@E?3>>9DAGBEFAI3D635A??A@
9A3> +;@57F:7K3D7D7EBA@E;4>7@AFA@>K8ADF:7;DAI@>73D@;@94GF3>EA8AD735:AF:7DSE
>73D@;@9F:7EG557EEA8A@7EFG67@F53@F:7@:7>BAF:7DEEG55776 A=:3>7  
A>>34AD3F;H7>73D@;@9:3E76G53F;A@3>47@78;FEAH7D;@6;H;6G3>>73D@;@93@6EFG67@FEI:A
>73D@F:DAG9:5A>>34AD3F;H7>73D@;@93D7=@AI@FA4734>7FA7@9397;@>73D@;@9EGEF3;@34>K
3@6B7DE;EF7@F>K+!"'7F3>  !AI7H7D5A>>34AD3F;H7>73D@;@9;E@AF3>I3KE78875F;H73@6;FE78875F;H7@7EE67B7@6E
A@F:7D;5:@7EE3@6;@F7@E;FKA8F:7;@F7D35F;A@E47FI77@9DAGB?7?47DE6GD;@9F:7
5A>>34AD3F;A@;>>7@4AGD9!A@9 ,:7BD7E7@57A83F735:7D;E3>EA@757EE3DKFA
E:3B7F:7;@F7D35F;A@EI;F:;@F:79DAGB&G;D:736 !AI7H7D9;H7@F:75GDD7@F
E;FG3F;A@I:7D776G53F;A@3>6;EB3D;F;7E3D7I;67@;@9IAD>6I;67;F53@476;88;5G>F8ADF735:7DE
FABDAH;67EG88;5;7@FEGBBADF "@#3B3@76G53F;A@3>6;EB3D;F;7E:3H73D;E7@6G7FAF735:7DES
;@5D73E76IAD=>A36 "@#3B3@BDA9D3??;@976G53F;A@4753?7?3@63FADK;@7>7?7@F3DK
E5:AA>E;@FA8AEF7DBDA9D3??;@9F:;@=;@9E=;>>E !AI7H7D;@3EGDH7KA87>7?7@F3DK
E5:AA>F735:7DE?AEFA8F:7D7EBA@67@FE87>FAH7DI:7>?766G7FAAH7DIAD=3@6AGFA8
D7EBA@67@FE87>F3@J;AGE34AGFF735:;@9BDA9D3??;@976G53F;A@(E:;?37F3> ,:7D78AD7;@EG5:3E;FG3F;A@;F;E6;88;5G>FFABDAH;67BDA9D3??;@976G53F;A@7CG3>>K 

209

"@F:78;7>6A85A?BGF7D EGBBADF765A>>34AD3F;H7>73D@;@9+%E7H7D3>3BBDA35:7E
EGBBADF9DAGB;@F7D35F;A@I;F:5A?BGF7DE (@7A8F:7E73BBDA35:7E;E9DAGB8AD?3F;A@;>>7@4AGD9 !A@9 >F:AG9:9DAGB8AD?3F;A@;E;?BADF3@F;@5A>>34AD3F;H7>73D@;@9;F;EA8F7@AH7D>AA=764GFBDABAE;@93@3BBDABD;3F79DAGB8AD?3F;A@?3K?3=7;FBAEE;4>7FA
5D73F747@78;5;3>9DAGBE&G7:>7@4DA5= *757@FEFG6;7E:3H7GE76EGDH7KADF7EF
D7EG>FEEG5:3E39734;>;FK3@697@67D6;887D7@57E3E5D;F7D;38AD9DAGB8AD?3F;A@,ABB;@9 -5:;63 !AI7H7DF:;E;E@AF@757EE3D;>KD7>3F76FAF:7;@F7D35F;A@E47FI77@F:7
9DAGB?7?47DE3@6;F;E@757EE3DKFA5A@E;67D:7F7DA97@7;FKD793D6;@9>73D@7DES
5:3D35F7D;EF;5EEG5:3EF:7;DB7DEA@3>;FK3@6EA5;3>3FF;FG67E$;@<A,:7D78AD7I73EEG?7F:3FF:7:7F7DA97@7;FKA8F:7D7EG>FEA87?AF;A@D75A9@;F;A@
3@3>KL768DA?F:7@3FGD3>GFF7D3@57EA8>73D@7DE;ED7>7H3@F8AD 9DAGB;@F7D35F;A@E "@F:;E
EFG6KI7F3D97F76F:7E57@3D;A A8B3;DIAD= ;@DA4AFBDA9D3??;@93@667H7>AB76 3EKEF7?
A8?3F5:;@9 4KF:7:7F7DA97@7;FKA87?AF;A@D75A9@;F;A@D7EG>FEF:3F 3GFA?3F7E9DAGB
8AD?3F;A@;@5A>>34AD3F;H7>73D@;@9 "@366;F;A@3E3D7E73D5:CG7EF;A@I73;?76 FA5>3D;8K
I:7F:7D9DAGB8AD?3F;A@43E76A@F:7:7F7DA97@7;FKA8 7?AF;A@D75A9@;F;A@D7EG>FE;E?AD7
78875F;H7F:3@5GDD7@F5A@H7@F;A@3>?7F:A6E ,:7EKEF7?I3E7H3>G3F76F:DAG9:
7JB7D;?7@FEI;F:5A>>797EFG67@FE I:A3D773E;>K3557EE;4>78AD8776435=

 ##$  !% !
/767H7>AB763?3F5:;@9EKEF7?GE;@97?AF;A@D75A9@;F;A@D7EG>FE8DA?@3FGD3>EB775:
63F335CG;D76 8DA?BD7 >7EEA@;@F7D35F;A@E3E3@;@6;53FAD ,ABDA5GD7 @3FGD3>EB775:8DA?
>73D@7DEF:7K I;>>>73D@I;F:3H;DFG3>B3DF@7D;@3BD7 >7EEA@3EE:AI@;@;9GD7  ,:7
H;DFG3>B3DF@7D;E3@3H3F3DBDA9D3??76;@36H3@578ADEB775:F:3F F3>=EFAF:7>73D@7D>;=73
D73>5:;>6E3K;@9F:;@9E>;=7Q/:3FE:AG>6I76A:7D7RADQHave you finished yetRFA?3F5:
F:7?;EE;A@A8F:7BD7 >7EEA@ KD75AD6;@95A@H7DE3F;A@E47FI77@F:7H;DFG3>B3DF@7D3@6
F:7>73D@7D@3FGD3>EB775:63F38DA?F:7>73D@7DSEEB775:53@475A>>75F76,:7D73EA@8ADGE;@93H;DFG3>B3DF@7DD3F:7DF:3@ 3@35FG3>B7DEA@3E3BD7 >7EEA@
B3DF@7D;EFABD7H7@F5:3@97E;@F:7>73D@7DSEEB775:5A@F7@F;@D7EBA@E7 FAF:7
5:3D35F7D;EF;5EA8F:7B3DF@7D KGE;@93H;DFG3>B3DF@7DEB775:63F353@475A>>75F76
7CG3>>KG@67DF:7E3?75A@6;F;A@E8AD7H7DKA@7 ,:7E7 63F33D7 F:7@3@3>KL763@63?3F5:;@93>9AD;F:?;EGE7643E76A@F:73@3>KE;ED7EG>FEFA?3F5:>73D@7DEI;F:735:AF:7D ,:7?3F5:;@93>9AD;F:?I3E5D73F76F:DAG9:F:77JB7D;?7@FE;@ :3BF7D 
Pre-Lesson
with a Virtual Partner

Analysis of
Natural Speech Data

Matching Learners Basedon the Analysis Results  

VirtualPartnerMatching
Algorithm

Recording%#0-! &3F5:;@9%73D@7DE 3E76A@F:7@3>KE;EA8 3F38DA? F:7)D7 %7EEA@



#! !% !

,:77JB7D;?7@F;@HA>H763FAF3>A8 B3DF;5;B3@FE?3>7E 3@687?3>7EI:AI7D7
G@;H7DE;FKG@67D9D36G3F73@69D36G3F7EFG67@FE "@3>>EG4E7CG7@F7JB7D;?7@FE;@F:;EEFG6K
;@6;H;6G3> @3?7EI7D7@AF;67@F;8;34>7FA4KF:;D6B3DF;7EB3DF;5;B3F;A@I3EHA>G@F3DK3@6
F:7D7I3E@AB7@3>FK8ADD78GE;@9FAB3DF;5;B3F7 )3DF;5;B3@FEI7D7;@8AD?76A8F:7BGDBAE7
3@65A@F7@FA8F:;EEFG6K3@6BDAH;676 F:7;DH7D43>3@6ID;FF7@5A@E7@F ,:;EEFG6KI3E
3BBDAH764KF:7F:;5EA??;FF77A8F:7-@;H7DE;FKA8,EG=G43
*

210





Pre-Lesson for
Participants
and Virtual Partner

Participants were
Randomly Matched

Analysis of Speech Data

EmotionRecognitionLesson Among
Participants
Evaluation of Pair WorkPair Work Activation ScalecPair Work Satisfaction
Scale



Development
of of
the
Developmenta Matching Algorithm

AnalysisAnalyze of the Relationship between
Emotion Recognition Results and Evaluation of Pair Workork

%#0-! >AI5:3DFA8F:77H7>AB?7@FA83&3F5:;@9>9AD;F:?

 $&#!&%$!" +&
/7D3@6A?>K?3F5:76 B3DF;5;B3@FE3@6:36F:7?F3=73>7EEA@ ;@DA4AFBDA9D3??;@9 ,:7>73D@;@9?3F7D;3>8ADDA4AFBDA9D3??;@9GE76;@F:77JB7D;?7@F;@HA>H76F:7GE7A8
?;5DA4;F ?;5DA4;F &;5DA4;F;E3E?3>>5A?BGF7D5D73F764KF:7D;F;E:BG4>;5
4DA3653EF7DFAF735:BDA9D3??;@9FA5:;>6D7@ EF:;EI3E3@A@>;@77JB7D;?7@F
B3DF;5;B3@FEBDA9D3??76F:7?;5DA4;FGE;@93E;?G>3FAD $%%"!"$$&#!&%!$&'$&!$ !+%%"# &
35:B3DF;5;B3@FI3EB3;D76I;F:3H;DFG3>B3DF@7D8AD>73D@;@93@6F:7;DEB775:63F3I7D7
5A>>75F766GD;@9F:;EBDA57EE ,:763F3I7D7 3@3>KL76F:DAG9: 7?AF;A@D75A9@;F;A@3EE:AI@;@;9GD7 ;DEFF:AE7B3DFEA8F:7EB775:63F3F:3F7J577676357DF3;@3?B>;FG678AD?AD7
F:3@ E75A@6EI7D7;67@F;8;763EEB775:3@6F:7E7E79?7@FEA83G6;AI7D77JFD35F76 
?AF;A@D75A9@;F;A@I3EB7D8AD?76GE;@9F:7E7EB775:63F34K;@BGFF;@9;F;@FA3@7?AF;A@
BD76;5F;A@?A67> ,:77?AF;A@BD76;5F;A@?A67>I3E5D73F76GE;@9677B>73D@;@93@6F:7
*K7DEA@G6;AP.;EG3>3F343E7A8?AF;A@3>+B775:3@6+A@9%;H;@9EFA@7 *GEEAI:;5:;E363F3E7FA8  3G6;AE3?B>7EFAD7BD7E7@F7;9:F7?AF;A@3>EF3F7E@7GFD3>
53>?:3BBKE363@9DK873D8G>6;E9GEF763@6EGDBD;E76EFAD76;@F:763F3D7BAE;FADK
07@A6A ,:7?A67>EFDG5FGD7GE7E%A@9+:ADF ,7D? &7?ADK3@6F:7&7> 8D7CG7@5K
57BEFD3>5A788;5;7@FE&I7D7GE763EF:7873FGD7 ,:7D73EA@8ADGE;@9&3EF:7
873FGD7;EF:3F;F;E78875F;H78AD7?AF;A@D75A9@;F;A@+3FA (4G5:;
 ,:7?A67>I3E
FD3;@76GE;@9FD3;@;@963F33@6 DAG@6EA8?35:;@7>73D@;@9I7D7B7D8AD?76FA5D73F73@
7?AF;A@BD76;5F;A@?A67>I;F:3@355GD35KA834AGF A@F7EF63F3-E;@9F:;E7?AF;A@BD76;5F;A@?A67>7?AF;A@D75A9@;F;A@I3EB7D8AD?76 /:7@F:7
EB775:63F3A8B3DF;5;B3@F$SE! F:EB775:I3E;@BGFF76;@FAF:77?AF;A@BD76;5F;A@?A67>3@
3DD3K& EG5:3E7CG3F;A@ I3EAGFBGFF76 !7D7 2!3 D7BD7E7@FEF:7@7GFD3>7?AF;A@3>
>7H7> 2!3 D7BD7E7@FEF:753>?7?AF;A@3>>7H7>  2!3 D7BD7E7@FEF:7:3BBK7?AF;A@3>>7H7>
 2!3 D7BD7E7@FEF:7E367?AF;A@3>>7H7> 2!3 D7BD7E7@FEF:73@9DK7?AF;A@3>>7H7> 2!3
D7BD7E7@FEF:7873D8G>7?AF;A@3>>7H7> 2!3 D7BD7E7@FEF:76;E9GEF767?AF;A@3>>7H7>3@6
  
 2!3 D7BD7E7@FEF:7EGDBD;E767?AF;A@3>>7H7> AD7J3?B>7;8& 2!3 1    2;F53@47G@67DEFAA6F:3FF:7>7H7>A8@7GFD3>7?AF;A@I3E:;9:;@F:7;@BGFF76
EB775:63F3 ,:7E73DD3KEI7D7F:7@3H7D3976FA5D73F73E;@9>73DD3K "8F:7@G?47DA8
7JFD35F76EB775:63F3E79?7@FE8DA?B3DF;5;B3@F$ ;E# F:7@F:73H7D39763DD3K 53@47
A4F3;@763EE:AI@;@7CG3F;A@  ,:;E I3EGE763EF:77?AF;A@D75A9@;F;A@D7EG>FE8AD
B3DF;5;B3@F$

211

& 2!3 + . 2!3  2!3  2!3  2!3  2!3  2!3  2!3






 2!3/

23







0  2!3 
0  2!3 
0  2!3  2!3 + 6 0  2!3 
#
#
#
#





















0  2!3 
0  2!3 
0  2!3 
0
#
#
#
#


















 2!37

23+ .               /Emotion Recognition Using Deep LearningEmotionPrediction
Model

AverageAmplitude

Extraction of Expression Points in Speech Data

Elapsed Time(s)%#0-!  @3>KE;EF:DAG9:?AF;A@*75A9@;F;A@-E;@977B%73D@;@9 %%"! &)!$&#!&% ('&"!"$"$)3DF;5;B3@FEI7D7B3;D76GBFA7@9397;@>73D@;@935F;H;F;7E3@67H3>G3F;A@EA8F:7;DB3;D
IAD=I7D75A@6G5F76 "@3EE7EE;@9B3;DIAD=-5:;63GE7635F;H3F;A@3@6E3F;E835F;A@3E
H3D;34>7EFAH7D;8KF:778875F;H7@7EEA85A>>34AD3F;H7>73D@;@9-5:;63 /7GE7F:7E7
FIAE53>7E 5F;H3F;A@I3E678;@763EF:7@G?47DA8EB775:G@;FE678;@763E35A:7E;H7
E79?7@FA8EB775:3@65AG@F768DA?F7JF63F3A4F3;@764KFD3@E5D;4;@9D75AD676
5A@H7DE3F;A@E6GD;@9B3;DIAD= -5:;63GE7635F;H3F;A@3E3@7H3>G3F;A@?7FD;58AD
5AAB7D3F;H7>73D@;@93E;F;E47>;7H76F:3FF:7@G?47DA8GFF7D3@57E5AG@F76;E78875F;H7;@
7EF;?3F;@9F:7EF3F7A85AAB7D3F;H7IAD=3@6BDAH;67EGE78G>;@8AD?3F;A@8AD3EE7EE;@9<G69?7@FEA8B3;DEI;F:BAAD5AAB7D3F;H7D7>3F;A@E:;BE3@6;@F7D35F;A@E47FI77@9DAGB
?7?47DE /76;H;676F:7>7@9F:A8F:75>3EEB7D;A6;@FAE75A@6E4753GE76;887D7@FB3;DE:36
6;887D7@FEFG6KF;?7E ,:735F;H3F;A@8ADB3DF;5;B3@F 3@6B3DF;5;B3@F I3E67@AF764K' "@F:;E7JB7D;?7@FB3;DEI7D7?3F5:763E8A>>AIEB3DF;5;B3@F 3@6B3DF;5;B3@F 
B3DF;5;B3@F 3@6B3DF;5;B3@FB3DF;5;B3@F3@6B3DF;5;B3@FB3DF;5;B3@F3@6
B3DF;5;B3@FB3DF;5;B3@F3@6B3DF;5;B3@F3@6B3DF;5;B3@F 3@6B3DF;5;B3@F 3@6
F:7D7EG>F;@963F3E7FA835F;H3F;A@I3E4' ' ' '   '   ' 5,34>7 /%."/%*) 0!./%*))%-!1'0/%*)/!(. !/' 

)3;DIAD= I3E8G@)3;DIAD= 5A@EG>F3F;A@EI7D7:7>B8G>"F:;@=B3;DIAD= ;E39AA6?7F:A6"F:;@=;F;E73E;7D
FA3@EI7D;@B3;DEF:3@;@6;H;6G3>>K)3;DIAD= :7>B76?7
FA677B7@?KG@67DEF3@6;@9+3F;E835F;A@I3E?73EGD764K3BAEF IAD=EGDH7KF:3FFAF3>76F:7E5AD7EA8735:
CG7EF;A@;F7?FA67F7D?;@7F:7H3>G7A8E3F;E835F;A@ ,:7EGDH7KCG7EF;A@E3D7>;EF76;@,34>7
 3@6;@F:;E7JB7D;?7@FB3DF;5;B3@FE3@EI7D76F:78;H7CG7EF;A@E;@,34>7 A@3 BA;@F
E53>7D3@9;@98DA? FA3@6F:7EG?A8F:7E5AD7E8ADF:7B3;DA8B3DF;5;B3@FE;@6;53F76
E3F;E835F;A@ G7FAF:7;@8>G7@57A8EFG67@FES 5:3D35F7D;EF;5EEG5:3E788ADF3@6?AF;H3F;A@A@212

5AGDE7E3F;E835F;A@-5:;63GE76 E3F;E835F;A@3E3@7H3>G3F;A@?7FD;58AD5AAB7D3F;H7
>73D@;@93E;F;E47>;7H76FA47GE78G>;@67?A@EFD3F;@9F:7D7>3F;A@E:;B47FI77@F:7B3;D
78875F3@6B7DEA@3>;FKFD3;FE ,:7E3F;E835F;A@8ADB3DF;5;B3@FE  3@6 I3E67@AF764K( /:7@F:7B3;DEI7D7?3F5:763EB3DF;5;B3@F 3@6B3DF;5;B3@F B3DF;5;B3@F 3@6
B3DF;5;B3@FB3DF;5;B3@F3@6B3DF;5;B3@FB3DF;5;B3@F3@6B3DF;5;B3@FB3DF;5;B3@F
3@6B3DF;5;B3@F3@6B3DF;5;B3@F 3@6B3DF;5;B3@F F:7D7EG>F;@963F3E7FA8
E3F;E835F;A@I3E4( ( ( (   (   ( 5! !+%% " & &"!%# &)!
('&"!"$"$"&"! "!&"! %'&% !

  

%7F 47F:73DD3KA87?AF;A@3>>7H7>EA8B3DF;5;B3@F$ 3ED7BD7E7@F76;@7CG3F;A@  /:7@
?3F5:;@9B3DF;5;B3@FE F:DAG9: 3EB3;DEB3DF;5;B3@F 3@6B3DF;5;B3@F B3DF;5;B3@F 
3@6B3DF;5;B3@FB3DF;5;B3@F3@6B3DF;5;B3@FB3DF;5;B3@F3@6 B3DF;5;B3@F
B3DF;5;B3@F3@6B3DF;5;B3@F3@6B3DF;5;B3@F 3@6B3DF;5;B3@F >7F4 *    *   *    *    *    *  5 47F:763F3E7FA86;887D7@57E;@@7GFD3>
7?AF;A@3>>7H7>E47FI77@B3;DE3@6>7F4 )    )    )    )    )   )  5 47F:763F3E7FA8F:7EG?EA8@7GFD3>7?AF;A@3>>7H7>E47FI77@B3;DE +;?;>3D>K678;@763F3E7FE8ADF:76;887D7@57E3@6EG?EA8E7H7@AF:7D7?AF;A@3>EF3F7E
53>?:3BBKE363@9DK873D8G>6;E9GEF76EGDBD;E763E I7>>3E8ADF:76;887D7@57E3@6
EG?EA8F:77;9:F7?AF;A@D75A9@;F;A@D7EG>FE3@68ADF:763F3E7FEA835F;H3F;A@3@6
E3F;E835F;A@9;H;@9 5A?4;@3F;A@E;@FAF3>

 



        

  





 



       

%#0-! *7>3F;A@E:;B47FI77@5F;H3F;A@E3@6;887D7@57E;@@9DK?AF;A@3>%7H7>EGBB7D >78F;E9GEF76?AF;A@3>%7H7>EGBB7DD;9:F3@6+GDBD;E76?AF;A@3>%7H7>E
>AI7D >78F *7>3F;A@E:;B47FI77@+3F;E835F;A@ 3@6+G? A873D8G>?AF;A@3>%7H7>E>AI7D
D;9:F
"@F:73@3>KE;EI75A@6G5F763F7EFA8@A@ 5ADD7>3F;A@I;F:F:7@G>> :KBAF:7E;EF:3F
QF:7D7;E@A5ADD7>3F;A@47FI77@F:7FIAH3D;34>7ER3@6F:73>F7D@3F;H7:KBAF:7E;EF:3FQF:7D7
;E36;887D7@5747FI77@F:7BABG>3F;A@?73@3@6F:7E3?B>7?73@ R/7E7FF:7E;9@;8;53@57
>7H7>FA 3@65A?B3D76F7EFEF3F;EF;5F53>5G>3F768DA?F:75ADD7>3F;A@5A788;5;7@F3@63F
6;EFD;4GF;A@H3>G7A8" +  AD3>> 5A?4;@3F;A@EI753>5G>3F76F7EFEF3F;EF;5F3@68AG@6
F:3FF:7@G>>:KBAF:7E;EI3ED7<75F76;@8AGD53E7EF:7D7>3F;A@E:;B47FI77@F:763F3E7FA8
6;887D7@57E;@3@9DK7?AF;A@3>>7H7>E3@6F:763F3E7FA835F;H3F;A@% +  - "F:7
D7>3F;A@E:;B47FI77@F:763F3E7FA86;887D7@57E;@6;E9GEF767?AF;A@3>>7H7>E3@6F:763F3E7F

213

A835F;H3F;A@% +    - "F:7D7>3F;A@E:;B47FI77@F:763F3E7FA86;887D7@57E;@EGDBD;E76
7?AF;A@3>>7H7>E3@6F:763F3E7FA835F;H3F;A@% +   - "3@6F:7D7>3F;A@E:;B47FI77@
F:763F3E7FA8F:7EG?EA8873D8G>7?AF;A@3>>7H7>E3@6F:763F3E7FA8E3F;E835F;A@% +   " ,:7D78AD7I753@E3KF:3FF:75ADD7>3F;A@5A788;5;7@FE47FI77@F:763F3E7FA86;887D7@57E
;@3@9DK7?AF;A@3>>7H7>E3@6F:763F3E7FA835F;H3F;A@T 47FI77@F:763F3E7FA8
6;887D7@57E;@6;E9GEF767?AF;A@3>>7H7>E3@6F:763F3E7FA835F;H3F;A@  47FI77@F:7
63F3E7FA86;887D7@57E;@EGDBD;E767?AF;A@3>>7H7>E3@6F:763F3E7FA835F;H3F;A@ 3@6
47FI77@F:763F3E7FA8EG?EA8873D8G>7?AF;A@3>>7H7>E3@6F:763F3E7FA8E3F;E835F;A@T 3D7E;9@;8;53@F3FF:7 >7H7>  D3B:EE:AI;@9735:D7>3F;A@E:;B3@6F:7
D79D7EE;A@>;@75D73F764KF:7>73EFECG3D7E?7F:A63D7E:AI@;@;9GD7 EEG?;@9F:3F
F:763F3E7FE8ADF:76;887D7@57E;@3@9DK6;E9GEF76EGDBD;E763@6F:7EG?A8873D8G>
7?AF;A@3>>7H7>E3D7;@67B7@67@FI753@5A@E;67DF:78A>>AI;@98AGDBA;@FE;@AGD
6;E5GEE;A@ ,:7E?3>>7DF:76;887D7@57;@F:73@9DK7?AF;A@3>>7H7>F:79D73F7DF:735F;H3F;A@ ,:7>3D97DF:76;887D7@57;@F:76;E9GEF767?AF;A@3>>7H7>F:79D73F7DF:735F;H3F;A@ ,:7>3D97DF:76;887D7@57;@F:7EGDBD;E767?AF;A@3>>7H7>F:79D73F7DF:735F;H3F;A@ ,:7E?3>>7DF:7EG?A8F:7873D8G>7?AF;A@3>>7H7>F:79D73F7DF:7E3F;E835F;A@    ("# !&"&! "$& 
8F7D5A@E;67D;@98AGD3EB75FED7>3F76FA7?AF;A@3>>7H7>EI767H7>AB763?3F5:;@9
3>9AD;F:? ,:7BDA576GD78AD67H7>AB;@9F:73>9AD;F:?;E7JB>3;@763E8A>>AIE
;DEFI767F7D?;@76B3;DEA8FIA;@6;H;6G3>E 8DA?3>>F:7>73D@7DE
I:AI3@F76FA47?3F5:76 ;@6;53F7EF:7BA;@FF:3F;?BDAH7E3EF:76;887D7@57;@F:7
3@9DK7?AF;A@3>>7H7>475A?7EE?3>>7D;@6;53F7EF:7BA;@FF:3F;?BDAH7E3EF:7
6;887D7@57;@F:76;E9GEF767?AF;A@3>>7H7>475A?7E>3D97D;@6;53F7EF:7BA;@FF:3F
;?BDAH7E3EF:76;887D7@57;@F:7EGDBD;E767?AF;A@3>>7H7>475A?7E>3D97D3@6);@6;53F7E
F:7BA;@FF:3F;?BDAH7E3EF:7EG?A8F:7873D8G>7?AF;A@3>>7H7>475A?7EE?3>>7D "8I7
3EEG?7F:3FF:73DD3KA87?AF;A@3>>7H7>E8ADB3DF;5;B3@F$;E7JBD7EE763E F:7@F:7
5A?4;@3F;A@EA83@6 8AD>73D@7DE33@643D7678;@763E7JBD7EE;A@E
3@6D7EB75F;H7>K '7JF3EEG?;@9F:3FF:763F39DAGBE8ADF:76;887D7@57;@F:73@9DK7?AF;A@3>>7H7>
F:76;887D7@57;@F:76;E9GEF767?AF;A@3>>7H7>F:76;887D7@57;@F:7EGDBD;E767?AF;A@3>
>7H7>3@6F:7EG?A8F:7873D8G>7?AF;A@3>>7H7>I7D73>>;@67B7@67@FI7@AD?3>;L76F:7E7
8AGD63F39DAGBEFA:3H73?;@;?G?A8 3@63?3J;?G?A8 FAFD73FF:7E78AGDBA;@FE
7CG3>>K /:7@   3@6  I7D767@AF763EF:7@AD?3>;L7663F3   3@6  D7EB75F;H7>KF:7AH7D3>>BA;@F8AD>73D@7DE3@6I3E678;@763E
7JBD7EE;A@ ;@3>>K3EI75A@F7@6F:3F:;9:7DH3>G7E5ADD7EBA@6FA;@5D73E76B3;DIAD=35F;H;FK
3@6E3F;E835F;A@I7D75A??7@6>73D@7DE3E3?3F5:;@F:7AD67DA8F:7;DH3>G7E AD
7J3?B>7;8I753>5G>3F7F:7BA;@FEEG5:F:3F  ,    ,   3?A@9>73D@7DE   3@6
I7D75A??7@6>73D@7DE 3@6 ;@F:3FAD67DFA>73D@7D 


23
1
  +  1
 * 

2 3  +   *  

2 3  +   *  


2 3
1
   +  1
  )  

2 3  +     )     )     )    

214



#"! !% !

@7H3>G3F;A@7JB7D;?7@FI3E5A@6G5F76A@F:7BDABAE76?3F5:;@9EKEF7? ,:7
B3DF;5;B3@FE5A@E;EF76A83FAF3>A8 G@;H7DE;FK3@69D36G3F7EFG67@FE?3>73@6
87?3>7 ,:7BDABAE76?7F:A6I3E7H3>G3F764K5A?B3D;@9;FI;F:3B3;D;@9?7F:A6F:3F
GE76E7J3@643E;535367?;5E=;>>I:;5::3E477@5A@8;D?76FA4778875F;H74K-5:;63
-5:;63 -5:;637JB>3;@EF:7B3;D;@9?7F:A6GE;@9E7J3@643E;535367?;5E=;>> 3E
8A>>AIE +ADFF:7EFG67@FE;@67E57@6;@9AD67D43E76A@F:7;D43E;535367?;5E=;>> E5AD7E
 )3;DF:7?355AD6;@9FAF:7EADF;@9D7EG>FE"8F:7B3;D;@9D7EG>FE3D7;@E3?7 E7JB3;DE;@ D7B>357F:7?I;F:ABBAE;F7 E7JB3;DE
I;F:3E?3>>6;887D7@57;@43E;535367?;5E=;>> E5AD7E !AI7H7D;8F:76;887D7@57;@43E;5
35367?;5E=;>> E5AD7E7J5776E AGFA8 BA;@FEF:7KI;>>47=7BF3EE3?7 E7JB3;DE
@7JB7D;?7@FI3E5A@6G5F76FA5A?B3D7F:77J;EF;@9?7F:A6A8?3F5:;@9GE;@9
5:3D35F7D;EF;5EA4F3;@768DA?EGDH7KEADF7EFE3E;@6;53FADEI;F:F:7BDABAE76?7F:A6A8
?3F5:;@9GE;@9F:7:7F7DA97@7;FKD7>3F76FA>73D@7DES5:3D35F7D;EF;5E3E;@6;53FADE

*#$ !&&"%;DEFF:77JB7D;?7@F3>B3DF;5;B3@FE>73D@76I;F:B3DF@7DE?3F5:764KGE;@9F:77J;EF;@9?7F:A6 "@-5:;63SE7JB7D;?7@F3 BA;@FF7EFI3E5A@6G5F76FA?73EGD743E;535367?;5
E=;>>E "@F:;E7JB7D;?7@F3BDA9D3??;@9F7EFI;F:3?3J;?G?E5AD7A8 BA;@FEI3E
5A@6G5F76FA?73EGD743E;5BDA9D3??;@9E=;>>E ;@DA4AFBDA9D3??;@9 ,:7@43E76A@-5:;63SE3>9AD;F:? B3DF;5;B3@FEI7D7?3F5:763@67@93976;@B3;D>73D@;@938F7DI:;5:
B3;DIAD=I3E7H3>G3F76GE;@9-5:;63SE35F;H3F;A@ 3@6E3F;E835F;A@'7JFF:7B3DF;5;B3@FE>73D@76I;F:B3DF@7DE?3F5:76GE;@9F:7BDABAE76?7F:A6 >>
B3DF;5;B3@FE>73D@76I;F:H;DFG3>B3DF@7DE3@6F:7EB775:63F3A4F3;@76I7D73@3>KL76FA
?3F5:B3DF;5;B3@FE43E76A@F:7?3F5:;@93>9AD;F:?67H7>AB76 ,7@B3DF;5;B3@FEI7D7
?3F5:763@67@93976;@B3;D>73D@;@938F7DI:;5:B3;DIAD=I3E7H3>G3F76;@F:7E3?7
?3@@7D ;@3>>KF:7B3;DIAD=7H3>G3F;A@A8F:77J;EF;@9?7F:A63@6F:7BDABAE76?7F:A6
I7D75A?B3D76

*#$ !&%'&%! %'%%"!

 *(+-%.*)*"/$!(!)1'0!.*"+%-2*-&!1'0/%*)
 

 

%#0-!



   



A?B3D;EA@A8J;EF;@93@6)DABAE76&7F:A6E &,8AD5F;H3F;A@>78F3@6
+3F;E835F;A@D;9:F

;9GD7 5A?B3D7EF:735F;H3F;A@3@6E3F;E835F;A@A8F:77J;EF;@9?7F:A63@6F:7BDABAE76
?7F:A6 ,:7?73@H3>G7A835F;H3F;A@8ADF:77J;EF;@9?7F:A6I3E
3@68ADF:7BDABAE76
?7F:A6;FI3E  EEG?;@9F:3FF:7B3;DE;@F:77J;EF;@93@6BDABAE76?7F:A6EI7D7
;@67B7@67@FE;@57F:7B3DF;5;B3@FEI7D7F:7E3?74GFF:75A?4;@3F;A@EI7D76;887D7@FI7

215

B7D8AD?763@3@3>KE;EFA67F7D?;@7I:7F:7DF:7D7I3E36;887D7@57;@F:7?73@H3>G7EA8
B3;DIAD=7H3>G3F;A@8AD4AF:?7F:A6E FIA E;676F F7EFI;F:AGF5ADD7EBA@67@57I3E
B7D8AD?768ADF:73@3>KE;E ,:7@G>>:KBAF:7E;EI3EF:3FQF:7D7;E@A6;887D7@57;@F:7?73@
H3>G7EA8B3;DIAD=7H3>G3F;A@47FI77@F:77J;EF;@93@6BDABAE76?7F:A6ER3@6F:7
3>F7D@3F;H7:KBAF:7E;EI3EF:3FQF:7D7;E36;887D7@57;@F:7?73@H3>G7EA8B3;DIAD=
7H3>G3F;A@47FI77@F:77J;EF;@93@6BDABAE76?7F:A6E R/75A?B3D76F:7F H3>G7I;F:F:7
4AG@63DKH3>G7A8" + 8AD3FIA E;676F7EFI;F:3E;9@;8;53@57>7H7>A8  E3D7EG>F
F:7@G>>:KBAF:7E;EI3E@AFD7<75F763@6;FI3E8AG@6F:3FF:7D7I3E@AE;9@;8;53@F6;887D7@57
;@F:7?73@H3>G7EA8B3;DIAD=7H3>G3F;A@47FI77@F:77J;EF;@93@6BDABAE76?7F:A6E
4AF:8AD35F;H3F;A@2% +   $ +  33@6E3F;E835F;A@2% +  *  $ +  3 *(+-%.*)*"/$!(!)1'0!.*"+%-2*-&!1'0/%*),A;@H7EF;93F7F:778875F;H7@7EEA8F:7BDABAE76?7F:A6AH7D7J;EF;@93BBDA35:7E8AD
EB75;8;5FKB7EA8B3DF;5;B3@FEI73@3>KL76;@6;H;6G3>E3F;E835F;A@A@3E53>7A8BA;@FE ;DEFI77J3?;@76F:7B3DF;5;B3@FES43E;535367?;5E=;>>E43E76A@BD7 F7EFD7EG>FEF:7;D
E3F;E835F;A@I:7@?3F5:763@6>73D@;@9I;F:B3DF@7DEGE;@9F:77J;EF;@9?7F:A63@6F:7;D
E3F;E835F;A@I:7@?3F5:763@6>73D@;@9I;F:B3DF@7DEGE;@9F:7BDABAE76?7F:A6 ,:7
6;887D7@57E;@E3F;E835F;A@53>5G>3F764KEG4FD35F;@9F:7E3F;E835F;A@A4F3;@76I;F:F:7
7J;EF;@9?7F:A68DA?F:3FA8F:7BDABAE76?7F:A63D7BD7E7@F76;@,34>7 I:7D7
B3DF;5;B3@FEI;F:>AI7D43E;535367?;5E=;>>EE:AI76:;9:7DE3F;E835F;A@I;F:F:7BDABAE76
?7F:A65A?B3D76FAF:77J;EF;@93BBDA35: A@E7CG7@F>KI7BDA577676I;F:3@3@3>KE;EFA
7JB>AD7F:7D7>3F;A@E:;B47FI77@43E;535367?;5E=;>>E5AD7E3@66;887D7@57E;@E3F;E835F;A@ "@F:73@3>KE;EI75A@6G5F763F7EFA8@A@ 5ADD7>3F;A@I;F:F:7@G>>:KBAF:7E;EF:3F
QF:7D7;E@A5ADD7>3F;A@47FI77@F:7FIAH3D;34>7ER3@6F:73>F7D@3F;H7:KBAF:7E;EF:3FQF:7D7
;E36;887D7@5747FI77@F:7BABG>3F;A@?73@3@6F:7E3?B>7?73@ R/7E7FF:7E;9@;8;53@57
>7H7>FA 3@65A?B3D76F:7F7EFEF3F;EF;5F53>5G>3F768DA?F:75ADD7>3F;A@5A788;5;7@F3@6
3F 6;EFD;4GF;A@H3>G7A8" +  E3D7EG>FE;@57F:7@G>>:KBAF:7E;EI3ED7<75F762% + - "3I753@3EE7DFF:3FF:75ADD7>3F;A@5A788;5;7@FT 47FI77@35367?;5E=;>>
E5AD7E3@66;887D7@57E;@E3F;E835F;A@;EE;9@;8;53@F;@F7D?EA8F ,:7D7>3F;A@E:;B3@6F:7
D79D7EE;A@>;@75D73F764KF:7>73EFECG3D7E?7F:A63D7E:AI@;@;9GD7 DA?F:7D7;F
53@475A@F7@676F:3FF:7BDABAE76?3F5:;@9EKEF7??3K:3H739D73F7DBAF7@F;3>FAK;7>6
BAE;F;H778875FE8AD>73D@7DEI;F:>AI7D43E;535367?;5E=;>>E 
,34>7 !-.*)'*(+-%.*)*"/%."/%*)2%/$/$!3%./%)#) -*+*.! !/$* .
)3DF;5;B3@F











5367?;5+=;>>+5AD7











+3F;E835F;A@J;EF;@9&,



 







+3F;E835F;A@;887D7@57;@)DABAE76&, +3F;E835F;A@

T 

T

 
















 %-!/!1'0/%*)/$-*0#$,0!./%*))%-!.''*/! /*+-/%%+)/."@366;F;A@FAF:7E3F;E835F;A@EGDH7KD793D6;@9B3;DIAD=I75A@6G5F76;@F7DH;7IE38F7D3>>
F:77JB7D;?7@FEFA3E=I:;5:>73D@;@9B3DF@7DF:7B3DF;5;B3@FE8AG@673E;7DFAEFG6KI;F:3@6
F:7D73EA@E8ADF:7;DBD787D7@57E GD;@9F:7;@F7DH;7IEI767>;47D3F7>K5A@573>76I:7F:7D
F:7B3DF@7DEI7D7?3F5:76GE;@9F:77J;EF;@9?7F:A6ADF:7BDABAE76?7F:A6 ,:7D7EG>FEA8
F:7EGDH7K3D7BD7E7@F76;@;9GD7 ?A@9F:7B3DF;5;B3@FEFIA;@6;H;6G3>EI7D7?3F5:76

216



      

I;F:F:7E3?7B3DF@7DEGE;@94AF:F:77J;EF;@93@6BDABAE76?7F:A6EEAI76;6@AFA4F3;@
EGDH7KD7EBA@E7E8DA?F:7? !AI7H7D A8F:7B3DF;5;B3@FEEF3F76F:3FF:7K8AG@6;F
73E;7DFA>73D@I;F:B3DF@7DE?3F5:76F:DAG9:F:77J;EF;@9?7F:A6 GDF:7D?AD7;@F:7;@F7DH;7IEI;F:B3DF;5;B3@FEI:AD3F76F:7BDABAE76?7F:A6?AD7
83HAD34>KF:7?3<AD;FKD7BADF76F:3FF:7;D>73D@;@9B3DF@7DE;@F:7BDABAE76?7F:A6I7D7
73E;7DFAF3>=FA?3=;@9;FE;?B>7DFA3E=CG7EF;A@E3@6E77=36H;57 !AI7H7DEA?7
B3DF;5;B3@FE?7@F;A@76F:3F6GD;@9F:7BDABAE76?7F:A6B7D;A63E;FI3EF:7;DE75A@65>3EE
FA97F:7DF:7K87>F>7EED7E;EF3@57;@;@F7D35F;@9I;F:F:7;DB3DF@7DE5A?B3D76FAF:78;DEF5>3EE
I;F:F:77J;EF;@9?7F:A6BAEE;4>K4753GE7F:7KI7D797FF;@9355GEFA?76FAF:7
7JB7D;?7@F3>E7FGB ,:7E7D7EG>FEEG997EFF:3FF:7AD67DA8B3DF;5;B3F;A@3@6F:7@3FGD7A8F:7F3E=E?3K
:3H7;@FDA6G5764;3E3?A@9F:7B3DF;5;B3@FE ,AEGBBADFF:7EF3F7?7@F?367;@E75F;A@F:3FQF:7BDABAE76?3F5:;@9EKEF7??3K:3H739D73F7DBAF7@F;3>FABDA6G57BAE;F;H7
78875FE8AD>73D@7DEI;F:>AI7D43E;535367?;5E=;>>ER8GDF:7D7JB7D;?7@FE3D7@77676 ,:7E7
7JB7D;?7@FEE:AG>6;@5>G67;@5D73E;@9F:7@G?47DA8B3DF;5;B3@FE3@67@EGD7@A4;3E4K
3EE;9@;@96;887D7@F9DAGBEA8B3DF;5;B3@FEFA>73D@I;F:7;F:7DF:77J;EF;@9ADF:7BDABAE76
?7F:A6    




%#0-! *7>3F;A@E:;B47FI77@3E;55367?;5+=;>>+5AD7E3@6;887D7@57E;@+3F;E835F;A@                








 %#0-! +GDH7KD7EG>FEA@Q%73D@;@9B3DF@7DF:3FB3DF;5;B3@FE;@F:77JB7D;?7@F8AG@6?AEF
73EKFA>73D@I;F: R " "@F:;EEFG6KI7BDABAE763?3F5:;@9EKEF7?8AD3GFA?3F;@99DAGB8AD?3F;A@4KF:7
:7F7DA97@7;FKA87?AF;A@D75A9@;F;A@D7EG>FE;@A@>;@75A>>34AD3F;H7>73D@;@96GD;@9B3;D76
DA4AFBDA9D3??;@9 /75A@6G5F763@7H3>G3F;A@7JB7D;?7@F5A?B3D;@9F:77J;EF;@93@6
BDABAE76?7F:A6EA8F:7?3F5:;@9EKEF7?FA5>3D;8KI:7F:7D9DAGB8AD?3F;A@43E76A@F:7
:7F7DA97@7;FKA87?AF;A@D75A9@;F;A@D7EG>FE;E?AD778875F;H7F:3@GE;@95A@H7@F;A@3>
?7F:A6E /:;>7@AE;9@;8;53@F6;887D7@57I3EA4E7DH76;@F:73H7D397B3;DI;E77H3>G3F;A@
D3F;@9EI78AG@6F:3FF:7BDABAE76?7F:A6D7EG>F76;@:;9:7DE3F;E835F;A@D3F;@9E8ADF:AE7

217

B3DF;5;B3@FEI;F:>AI7D43E;5E=;>>E !AI7H7DF:77JB7D;?7@F:36>;?;F3F;A@EEG5:3E3E?3>>
@G?47DA8B3DF;5;B3@FE3@6F:GE;@EG88;5;7@F63F33@6F:7BAEE;4;>;FKA84;3E6G7FAF:7
5A?B3D;EA@A8F:77J;EF;@93@6BDABAE76?7F:A6EI;F:F:7E3?7B3DF;5;B3@FE3@66;887D7@F
F3E=5A@F7@F ,:7D78AD7;@8GFGD7EFG6;7EI7B>3@FA;@5D73E7F:7@G?47DA8B3DF;5;B3@FE3@6
5A@6G5FEG88;5;7@FF7EF;@9FAD78;@7F:7?3F5:;@93>9AD;F:? "@366;F;A@I7B>3@FA5A@6G5F
47FI77@ B3DF;5;B3@F5A?B3D;EA@EI;F:EF3@63D6;L76F3E=5A@F7@FD3F:7DF:3@I;F:;@
B3DF;5;B3@F5A?B3D;EA@E "@F:;EI3KI7I;>>53D78G>>K355AG@F8AD4;3E7E6G7FA6;887D7@57E;@
AD67DA8B3DF;5;B3F;A@3@6F3E=5A@F7@F3@65A@F;@G7I;F:F:77H3>G3F;A@7JB7D;?7@FE $! ,:;EIAD=I3EEGBBADF764KJSPS KAKENHI

D3@F'G?47D !

 
&J4I K?4I!%><J=>=IJ;4I!)JI@D4I0=>)8BD0;?A>6A0<<8=6
;0=6D064B8=C46A0C4302A>BBC742DAA82D;D<8=;4<4=C0AH&27>>;CF>H40A20B4BCD3HDB8=6
NB2A0C27O8=58E4B27>>;B/-043'12&4%#3+/.  M

7CC?B3>8>A6 92><?43D
 
&274A4A%&8338@&J=274I)8E4A>B
'742>6=8C8E414=458CB>5 40A=8=6><?DC4A
$A>6A0<<8=6<4C00=0;HB8B>5CA0=B54A45542CB/41.#,/(&4%#3+/.#,26%*/,/)6 M 7CC?B3>8>A6 43D>:70;4 >;;01>A0C8E4;40A=8=64=70=24B2A8C820;C78=:8=6/41.#,/('%*./,/)6&4%#3+/.  7CC?B3>8>A6  9C4E8 0 &"+%!(, '
&C4<A4;0C4305C4AB27>>;?A>6A0<02C8E8C84B0=3
0BB>280C43>DC2><4B>=BCD34=C;40A=8=6&4%#3+/.#,%+'.%'2*'/161#%3+%'   M8;;4=1>DA6$>=6
'74<4270=82B>5& !02A>&2A8?CB.3'1.#3+/.#,/41.#,/(/-043'1400/13'&/,,#$/1#3+5''#1.+.)  M 7CC?B3>8>A6 
B
 

 
!D8A7403
=70=28=6&>280;=C4A02C8>=8=><?DC4A!4380C438BC0=243D20C8>=/41.#,
/(&4%#3+/.#,'%*./,/)6/%+'36   M 7CC?FFF9BC>A>A6BC01;4943D2C427B>28   #B78<0'&08C>#:098<0+>FDBH'40274AB44;1>DC(?2><8=6$A>6A0<8=63D20C8>=4,,'3+./(/'324.+5'12+36/(&4%#3+/.!04$,+2*'&+.#0#.'2'"    
Muehlenbrock, M. (2006). Learning Group Formation Based on Learner Profile andContext. International Journal on E-Learning, 5(1), 19-24. Chesapeake, VA: Association for
the Advancement of Computing in Education (AACE). Retrieved July 30, 2023
from https://www.learntechlib.org/primary/p/21767/.
'>??8=6
!4C7>3>;>6820;@D0=30A84B8=BCD3H8=6?A>24BB0=3>DC2><4B8=?44A
0BB4BB<4=C'#1.+.)#.&.2314%3+/.   M

7CC?B3>8>A6 9;40A=8=BCAD2
 

(27830
&CD3H>=$08AF>A:!4<14A>A<0C8>=!4C7>3B8==5>A<0C8>= 8C4A02H3D20C8>=+%*+1'('%341#,.+5'12+361#&4#3'%*//,/(.(/1-#3+/.%+'.%'#.&'%*./,/)6
/%3/1#,+22'13#3+/.!04$,+2*'&+.#0#.'2'"
8=9>
'74D=2C8>=8=6>5<068=0C8E4'78=:8=618;8CH8=$A>1;4<&>;E8=68=!DB82!0:8=6
8=;4<4=C0AH&27>>;;0BB4B>2DB8=6>=4C4A>64=48CH8=0 40A=8=6A>D?-?D1;8B7438=
0?0=4B4./41.#,/(3*'34&6/(%*//,42+%&4%#3+/.#,1#%3+%'   
7CC?B3>8>A6   BB<4?  / 
8E8=6BC>=4&%%DBB>
?A8;*'6'12/.4&+/ +24#,#3#$#2'/('-/3+/.#,
20''%*#.&/.) ,4=>3>7CC?BI4=>3>>A6A42>A3 
&0C>"#1D278+<>C8>=A42>6=8C8>=DB8=6!4;5A4@D4=2H24?BCA0;2>4558284=CB/41.#,
/(#341#,#.)4#)'1/%'22+.)   M7CC?B3>8>A6  9=;?  / 218

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Assessing college students’ sense ofcommunity for advancing community
knowledge
Chih-Hui SEETa*, Yi-Ning TSAIb*, Guo-Tsai HUNGc & Huang-Yao HONGd
a,b,d
Department of Education, National Chengchi University, TaiwanbDepartment of Japanese Studies National Taichung University of Science and Technology,
Taichung*111152518@g.nccu.edu.tw, *108152513@nccu.edu.tw,barryhuck@gmail.com, hyhong@nccu.edu.tw
Abstract: This case study aims to examine students’ sense of community (SOC)
during online learning. Participants were college students data came from the“Classroom Community Scale” and students’ online discussion in a platform designed
based on knowledge building theory and pedagogy. The results showed that students
were able to develop a strong sense of community and perceived learning as aknowledge creating endeavor as guided by knowledge building pedagogy.
Keywords: sense of community, knowledge building, online discussion

1. Introduction
Knowledge economy has accelerated the speed of educational innovation and reform thatvalues collaborative knowledge creation more than individual knowledge acquisition.
Knowledge creation or knowledge building (KB) is defined as a social process focusing onsustained idea improvement and on knowledge advancement as a collective endeavor. In a
KB environment, students continually contribute, refine, and build on their ideas for advancing
collective knowledge as a knowledge community (Bereiter & Scardamalia, 2014). Knowledge
building is thus idea-centered (Scardamalia & Bereiter, 2006) and its community particularly
highlights the importance of working creatively and interactively with ideas (Hong &Scardamalia, 2014), and members of the community are deemed as legitimate knowledge
contributors for the overall achievement of group knowledge goals.
In a knowledge building community, how to develop a stronger sense of community(SOC), is important and remains a challenge, which has also received increasing attention
from many educators. With stronger SOC, people can feel that they belong to a group or thegreater community and those who are in the group or in the community can also better relate
to each other with similar ideas or learning commitment. Stronger SOC also implies that
community members can be more easily motivated to interact with other community members
for sustained knowledge work that leads to the achievement of the community goals(Cacciamani & Perrucci, 2020). With increasing SOC, community members may also feel
more psychological safe and thus are more likely to participate more in a more proactive
manner. For students in a knowledge building community, SOC is therefore essential as it canfoster a better class climate for collective knowledge creation. Accordingly, in this study, we
engage students in KB activities, hoping that through various guided KB activities, students
could enhance their SOC as a knowledge-based community.

2. 濁濙thod
This case study adopted a mixed-methods research design to explore students’ sense ofcommunity in a knowledge building environment. Participants were 21 college andpostgraduate students taking a class titled “Study on Instructional Design of Educational
Technology” in a national university in Taiwan. The course encouraged students to interact
with one another and to contribute ideas to improve their assigned design projects. Two setsof surveys were administrated. One is a pre-established “Classroom Community Scale”(Rovia, A. P., 2020) which evaluates students’ sense of community (in two dimensions

219

including awareness of connectedness and awareness of learning); and the other is “Types ofKnowledge Community,” which contains self-develop, open-ended questions to assess
students’ sense about three types of knowledge community (i.e., knowledge telling, sharing,and creating communities). Sample questions are such as: What is a knowledge community?
What constitutes a good knowledge community? Data were mainly collected from online
activities/discussion, and the above mentioned two surveys. A paired-sample t-test analysis
was conducted to examine students’ perceived sense of community (survey 1) and sense ofknowledge community (survey 2). Z-test was executed to classify students’ perception ofknowledge community as knowledge-telling, knowledge-sharing, or knowledge-creating
community.

3. Findings
As Figure 1 shows, about students’ sense of community, first it was found the mean numberof times from the pre-test to the post-test in the awareness of connectedness were from 2.88
to 3.46 (t=3.58***), and in the awareness of learning were from 2.76 to 3.53 (t=6.06***,p= .001). It shows that there were significant differences after one-semester’s KB activity andstudents enhanced their sense of community in a knowledge building community. On the other
hand, Figure 2, about students’ sense of 3 types of knowledge community, it was found themean number of times from the pre-test to the post-test in knowledge telling were from 1.89
to 0.58 (t=-3.842***). It shows that while there was no statistically significant difference instudents’ perception of knowledge sharing community and knowledge creating community,
there was significant differences in the pre and post-test of students’ perception of knowledge
telling community. Figures 3 further shows social network analysis. It was found thatinteractions regarding students’ ideas submitting, exchanging, and suggestions andelaborations for their design projects are very frequent and consistent. Finally, regarding typesof knowledge community, it also indicates that students were more aware of the importanceof knowledge community for knowledge creation in a KB community. Figure 4 also shows
students weekly online activity performance.
Figure 1. Students’ sense of community in KB community; Figure 2. Types of knowledge community

4. Discussion
This study investigated students’ SOC after engaging students in an online knowledge building
environment. It was found that first, the participants enhanced their SOC, both on thedimensions of “connectedness” and “learning”; second, regarding students’ sense of 3 typesof knowledge community, students tended to consider it as knowledge telling before thecourse but they changed to that of knowledge creating after the whole semester. To conclude,
students enhance their SOC via KB and students were able to become more connected andinteractive community to achieve community’s learning goals. In sum, our study shows thatworking in a KB community makes it possible for student’s developing a stronger sense ofknowledge community (Bereiter & Scardamalia, 2014). While knowledge creation ismeaningful for the future progress of human beings, SOC is useful for enhancing the goodwill
to work collaboratively among members in a society that values group work more than220

individual work. Although this study shows that SOC can be fostered through a KB learning
environment, future discussion from different educational context and researches are needed
to triangulate the findings in this study.
Figure 3. The author networks

Figure 4. Online weekly performance in KF
Each student's weekly-based action on KF
28.38

30.00

25.57

24.48

25.00

23.86

23.19
18.81

20.00

17.24

15.95
15.00
10.19
10.00
5.00

6.86
1.33
1.95

8.24
4.00
2.90

5.19
2.10

3.29

9.71
7.76

3.86
1.90

2.62

2.00

7.00

1.57
0.33

0.33 2.86
2.67
0.00
0.10

0.00

5.10
2.86
1.38

0.38

0.76
1.33 2.00
0.57

5.71
2.76

5.86
2.38

0.48
3.00
1.05

5.95

5.33
0.10

0.19

1.90
1.05

0.00
W1

W2

W3

W4

W5

W6

W7

W8

READ

W9

CREATE

W10

W11

W12

W13

W14

W15

W16

W17

W18

MODIFY

References
Bereiter, C. & Scardamalia, M. (2014). Knowledge building and knowledge creation: one concept, twohills to climb. Knowledge Creation in Education (pp. 35-32), Springer.
Cacciamani, S. & Perrucci, V. (2020). Peer feedback and Sense of Community in a blended university
course. University of Valle d’Aosta, Strada Cappuccini 2/A, Aosta, Italy.
Hill, J. L. (1996). Psychological sense of community: Suggestions for future research. Journal ofCommunity Psychology. 24(4) 431-438.
Hong, H. Y. & Scardamalia, M. (2014). Community knowledge assessment in a knowledge building
environment. Computer and Education. 71, 279-288.
Rovia, A. P. (2020). Development of an instrument to measure classroom community. The Internet andHigher Education. 5, 197-211.
Scardamalia, M. & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K.
Sawyer (Ed.), Cambridge Handbook of the Learning Sciences (pp. 97-118).

221

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Teachers developing more creative learningviews via online knowledge building activities
Mei-Ju Chen*, Chao-Yu Guo**, Huang-Yao Hong**meiju@mx.nthu.edu.tw, chaoyu@nccu.edu.tw, hyhong@nccu.edu.tw
National Tsing Hua University*, National Chengchi University**
Abstract: This study investigates how teachers who were engaged in online knowledge
building via guided perspective-taking (PT) activities to discuss their critical teaching incidents(CTIs) helped them to develop creative learning views that see learning not merely asknowledge acquisition and exchange, but also as knowledge creation. Data were mainly fromonline discussion. The main findings indicate that PT-based KB activities in a technologicalplatform can guide teachers to reflect deeply on the CTIs they experienced before and on how
they might be able to help students learn better in a more constructivist and creative manner.
Keywords: perspective taking, knowledge building, online learning environment, discussion

1. Introduction
The importance of helping teachers develop more informed, creative learning views, so that they may
be more likely to succeed in engaging their students’ in more constructive and innovative learning has
been well recognized in literatures (e.g., Antink-Meyer & Arias, 2022; Park et al.,2022). How to engage
teachers in in-depth reflection on their views of learning, however, remains a pedagogical challenge formany educators. In the present study, we introduce perspective taking (PT) as a means to engage
teachers in such deep reflection. PT is a way of understanding one’s own mind or other’s minds frominside (Kloo et al., 2021), and it can help teachers to develop self-awareness and emotional clarity(Campbell & Pennebaker, 2003). In education, most PT-related research is concerned with students(Kazuo et al., 2020), focusing on how teachers use PT to help students to develop multiple views orcultural understanding (Abacioglu et al., 2020; Athanases & Sanchez, 2020). Yet, PT is rarely used asa tool to improve teacher professional development (TPD). As a reflective means, PT can provide threedifferent introspective levels to help teachers conduct progressively deeper reflection, from selfreflection (I–PT); to self-distancing reflection (meaning to reflect upon personal teaching experiences
through the perspective of an outside observer; Other–PT); to more refined self-reflection (Back to I–PT) via interpreting and synthesizing insights from previous I-PT and Other-PT reflection. In particular,when the object of reflection is on critical teaching incidents (CTIs), PT may reveal many unknown
aspects relating to teachers’ self-understanding of own teaching practice (Mor-Hagani & Barzilai, 2022)
and to their teaching emotions (Rademacher et al., 2010). Therefore, CTIs as recurring teachingproblems or challenges (Joshi, 2018) are appropriate topics for teachers’ reflection (Badia et al., 2021)
to help teachers develop robust reflection-in-action capacity (Schon,1983). To this end, we employed
PT in an online KB community (Hong & Scardamalia, 2014) to foster both individual and group reflection
among teachers to help them build more in-depth self-knowledge of teaching (as contrasted with thetypical core teaching knowledge or skills specified in teacher education curriculum).
The purpose of the study was to explore how in-service teachers can collaboratively help one
another to reflect on their CTIs via PT in an online platform for developing more informed andproductive learning views that can help them become more mentally adaptive teachers. Building on theabove discussion, the main research question is: Does engaging participants in PT while discussing
critical teaching incidents (CTIs) in an online knowledge building environment help them develop more
informed learning views to help their students learn more constructively?

2. Method
We recruited fifty-one incumbent teachers as participants who were taking a graduate course in auniversity from Taiwan. We engaged these participants in online learning in a knowledge building222

environment called Knowledge Forum (KF). We then introduced PT as a reflective measure to deepen
participants’ discussion about some critical teaching incidents (CTIs) occurred during their recent
teaching. The course duration was 18 weeks which were intentionally segmented into threephases/stages for PT activities--starting from I-PT, to Other-PT, and to Back-to-I-PT (see Figure 1 forour pedagogical design). Data sources and analyses focus on (1) the number of notes contributed, read,and modified in KF, and words written per note; (2) discussion that reveals participants’ learning viewsand emotions (in this paper we did not report the emotional aspect). We adopt Hong’s (2009)
conceptualization of learning as acquisition, or as participation, or as creation into our coding scheme
and Kappa coefficient of inter-coder reliability was calculated to be 0.74. Wilcoxon signed-rank test
was employed for analysis.

Figure 1. Students working in Knowledge Forum to perform three different perspective taking activities

3. Findings
Table 1 summarizes the key findings from the analysis throughout the whole semester. First, regarding
the overall online reflection trend, our analysis indicates that there was a decreasing trend of note
contributing, note modification, and note reading from Stage 1 to 3. However, when looked into thenumber of words (i.e., Chinese characters) written per note, there was an increasing trend from Stage 1
to Stage 3 (Figure 2). The fact that participants gradually posted fewer notes but wrote more words ineach note implies that participants progressively became more reflective on the CTIs discussed online
in KF from Stage 1 to Stage 3.
Second, regarding more specific reflection trend on learning views, Z tests to compare Stages 1 and3 indicated that there was a significant decrease in participants’ reflection on acquisition-oriented views(z = 5.064**, p < 0.01) and a significant increase in participants’ reflection on creation-oriented views(z = –2.428*, p < 0.05; see Figure 3), indicating that participants were able to develop more informed
learning views after taking this course.
Table 1. Changes in Participants’ Overall Online Reflective Activities and Specific Reflective trend on
Views of Learning in 3 PT stages
Knowledge building via Guided Perspective-Taking (PT)
Change in
Reflection

Stage 1

Stage 2

N
M
SD
N
M
SD
Online activities
Contribution
179
3.5
1.08
75
1.5
1.034
Modification
351
6.9
2.87
113
2.2
2.23
Reading
1,734
34
37.67
920
18
15.07
Words per note
10,311
202 56.046
18,156356 154.97
Views of learning
Acquisition 23 (21.70%) 0.45
0.642
13 (15.48%) 0.26
0.337
Participation 39 (36.79%) 0.76
0.885
32 (38.69%) 0.64
0.679
Creation 44 (41.51%) 0.86
0.775
38 (45.83%) 0.76
0.643
* p < 0.05, ** p < 0.01, *** p < 0.001. Note: contribution excluding feedback notes.

223

Z TestPre-Post
ChangeStage 3
N

M

SD

Z

54
78
836
41,535

1.1
1.5
16.4
814.4

0.31
1.08
16.57
368.0515.678***12.593***3.086**–11.861***0 (0.00%)
38 (37.62%)
63 (62.38%)

0
0.75
1.24

0
0.845
0.79

5.064**
0.116–2.428*

814.4

1000
500
0

202.2
179

356
75

54

I-PT Other-PT Back to I(stage 1) (stage 2)
PT(stage 3)
Note-contribution
Words per-note

Figure 2. online performance

Figure 3. Change in learning views

4. Discussion
This study engaged the participating in-service teaching in three different perspective-taking (PT)
activities (including I-PT, Other-PT, and Back-to-I-PT) while discussing some critical teachingincidents (CTIs) in an online knowledge building environment (i.e. Knowledge Forum, KF).
Conventional teachers’ professional development tends to focus on helping teachers develop coreteaching knowledge and skills. In contrast, our study is trying to guide teachers to perform self andsocial reflection on their learning views. The main finding indicates that discussing critical teachingincidents via taking different perspectives indeed can effectively help teachers become more reflective
to come up with more ideas for how to teach better, or to help students learn better. Accordingly, they
were more willing to embrace a more informed and creative learning view. Thus, we conclude that PT
as a self-distancing, reflective means, indeed, can transform teachers’ past teaching experience into
more practical self-knowledge that is essential for more effective teacher professional development. At
the same time, our finding also suggests that KF was able to provide the participants an online
knowledge space for perform the required self and social (collective) reflection (Kim, et al., 2018). It
also helps the participating teachers to form some community-based, in addition to merely individualbased, professional development experience. Future analysis will focus on analyzing the emotional
impact from PT activities and on how emotional transformation may be related to the change inparticipants’ learning views.

References
Abacioglu, C. S., Volman, M., & Fischer, A. H. (2020). Teachers’ multicultural attitudes andperspective taking abilities as factors in culturally responsive teaching. British Journal of
Educational Psychology, 90(3), 736-752. https://doi.org/10.1111/bjep.12328
Athanases, S. Z., & Sanchez, S. L. (2020). ‘A Caesar for our time’: toward empathy and perspectivetaking in new teachers’ drama practices in diverse classrooms. Research in Drama Education: The
Journal of Applied Theatre and Performance, 25(2), 236-255.
Badia,A., Becerril, L.& Gómez,M. (2021).Four types of teachers’ voices on critical incidents inteaching, Teacher Development, 25:2, 120-135, DOI: 10.1080/13664530.2021.1882549
Campbell, R. S., & Pennebaker, J. W. (2003). The secret life of pronouns: Flexibility in writing styleand physical health. Psychological Science, 14, 60–65. https://doi.org/10.1111/1467-9280.01419
Hong, H. Y. & Scardamalia, M. (2014). Community knowledge assessment in a knowledge building
environment. Computer and Education. 71, 279-288.
Joshi,K.R.(2018).Critical Incidents for Teachers’ Professional Development. Jurnal of NELTA
Surkhet, 5, 82–88. doi.org/10.3126/jns.v5i0.19493
Kazuo, H., Makoto, Y., & Kei, K. (2020). The Development of Emotion Understanding in Elementary
School Children: The Relation between Emotion Understanding and Perspective-Taking Skills.
Annual Bulletin, Graduate School of Education, Tohoku University, 6.
Kloo, D., Sodian, B., Kristen‐Antonow, S., Kim, S., & Paulus, M. (2021). Knowing minds: Linking
early perspective taking and later metacognitive insight. British Journal of Developmental
Psychology, 39(1), 39-53. https://doi.org/10.1111/bjdp.12359
Rademacher, R., Simpson, D., & Marcdante, K. (2010). Critical incidents as a technique for teachingprofessionalism. Medical Teacher, 32(3), 244-249. https://doi.org/10.3109/01421590903197001

224

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Unveiling University Students' Data
Literacy: A Case Study on Modeling
Reasoning in Data Mining Projects
Tianqi ZHANG
East China Normal University, China
51214108044@stu.ecnu.edu.cn
Abstract: In the era of big data, cultivating students' data literacy is of paramountimportance. Data literacy encompasses the abilities to collect, manage, analyze, andapply data. As data science is a superset composed of mathematics and statistics,
computer science, and specific application fields, it is crucial to investigate data literacy
development through the lens of statistical education. An essential component ofstatistical reasoning is modeling reasoning, which is also fundamental to data literacy.
However, limited research exists on the manifestation of data literacy among university
students in real-world tasks. Therefore, this study explores the forms of modelingreasoning exhibited by university students in a data mining project. The findings reveal
that, in a real teaching setting, university students' modeling reasoning ability follows aspiral progression. Considering these insights, modeling reasoning, as the core of data
mining activities, plays a pivotal role in fostering students' data literacy. To cultivate
data literacy among university students in the age of big data, we recommend
implementing project-based learning, incorporating ill-structured problems and real,complex, massive datasets as project backgrounds.
Keywords: Data literacy, modeling reasoning, statistical reasoning, project-basedlearning

1. Introduction
In the era of big data, the rapid advancement of intelligent technology and its
increasing integration into everyday life have transformed data into a valuable resource inthe 21st century. Both society and the labor market now demand graduates who possess theknowledge and skills to effectively utilize statistics, data management, and computer science
to make informed decisions. Consequently, the cultivation of data literacy, which
encompasses the abilities to collect, manage, analyze, and apply data, has become
increasingly crucial. In fact, data literacy has emerged as an upgraded version of information
literacy within the realm of 21st-century skills. To address the growing demand for dataexperts in the age of big data, universities have established data science majors.
Researchers argue that data science should encompass mathematics and statistics,
computer science and programming, as well as specific application fields (Rosenberg,
Lawson, Anderson, Jones & Rutherford, 2019). As data availability improves andtechnological advancements introduce user-friendly data processing software, researchers
in statistical education are actively exploring methods to foster students' data literacy in theera of big data (Ben-Zvi, Makar & Garfield, 2018).
Within statistical education, modeling reasoning is a fundamental underlying ability inboth statistical reasoning and data literacy (Bakker & Hoffmann, 2005; Wild & Pfannkuch,1999). Rubin (2019) synthesized key aspects derived from a series of works related to dataprocessing in the era of big data, including context, variation, aggregation, visualization, andinference. In particular, inference, specifically Informal Statistical Inference (ISI), is closely
associated with modeling inference. In the realm of statistical education, modeling reasoning
serves as the foundation for data analysis and result interpretation. It involves simplifying

225phenomena based on data and theories, generating models that can explain, predict, or
define these phenomena (Aridor & Ben-Zvi, 2017).
Modeling reasoning not only simplifies the process of defining, explaining, andpredicting phenomena based on data and theories (Aridor & Ben-Zvi, 2017), but it also
constitutes a key element of data literacy and statistical reasoning (Wild & Pfannkuch, 1999).
Generally, models are regarded as forms of interpretation, while modeling is a process that
simplifies, evaluates, and enhances our understanding of a phenomenon using key theories
and data from a specific discipline, either by incorporating it into existing theoreticalframeworks or by generating new discoveries (Lehrer & Schauble, 2010). Models can beclassified as abstract models (conceptual models) or concrete models (e.g., figures andtables). Abstract models represent real-world systems and conjecture about their behavior todescribe, explain, predict, and elaborate on their behavior (Wild & Pfannkuch, 1999).
Concrete models, on the other hand, serve as tools to represent processes such as
identifying key components or properties of a population, making predictions or samples,and drawing inferences about the representativeness of a random sample (Aridor & Ben-Zvi,2017). Modeling is a multifaceted process in which the role of the model evolves alongsidechanges in thinking (Gravemeijer, 1999). Initially, a model takes the form of an "informal
reasoning model." As a new concept emerges, the model's role changes, and it transforms
into a "formal reasoning model." These two processes are accompanied by a third processthat shapes the model into a set of symbols guiding previous reasoning processes(Gravemeijer, 1999).
Practical activities in statistical education are often seen as a form of modeling as they
involve understanding variation and uncertainty, processing data, and constructing models(Lehrer & English, 2018). The modeling process entails evaluating and optimizing models,
including generating new theoretical ideas or discoveries based on data (Dvir & Ben-Zvi,2018). Modeling reasoning can be viewed as an analogical process that simplifies real
phenomena, elucidates connections and relationships among their components, and deals
with inherent uncertainties (Wild & Pfannkuch, 1999). Familiarity with the modelingreasoning process greatly aids the cultivation of data literacy. In an effort to enhance thequality of statistical education in primary schools, Biehler et al. interviewed four pre-service
primary school teachers who participated in a statistical modeling reasoning training course.
They evaluated and analyzed the teachers' reasoning processes, scrutinized their utilizationof Tinkerplots to model statistical situations, and assessed their ability to evaluate models inaccordance with the given statistical situations, providing recommendations accordingly(Biehler, Frischemeier & Podworny, 2017). The learning environment associated withmodeling reasoning can also facilitate the development of students' data literacy. Conway et
al. conducted quasi-experimental research on primary school students and found thatadhering to the principles of a statistical reasoning learning environment in both beliefs andpractices had a more positive impact on students' statistical reasoning ability compared to
traditional classroom teaching (Conway, Martin, Strutchens, Kraska & Huang, 2019). This
indicates that training in modeling reasoning is pivotal in statistical and data science teachingactivities, and that modeling reasoning ability plays a vital role in students' comprehension,
analysis, and application of data.
Despite the considerable research on modeling reasoning in primary and middle school
students, few studies have focused on the data literacy of university students (Setiawan &Sukoco, 2021). Currently, there is a dearth of empirical research examining the modelingreasoning ability of university students. Thus, a better understanding of how university
students manifest modeling reasoning ability is crucial for the development of data science
programs. Additionally, as there is limited guidance on the theoretical exploration andteaching practices for specific data literacy training, further theoretical and empiricalresearch is necessary to guide the development and implementation of data scienceeducation.
Therefore, this study aims to explore the manifestation of data literacy among
university students during real-world tasks, specifically by examining the forms of modelingreasoning employed by university students in a data mining project. Due to the limited
knowledge regarding modeling reasoning among university students, a qualitative research

226approach, specifically a case study, was deemed appropriate for exploring this phenomenon.
The study utilized a bottom-up grounded theory method to provide a detailed analysis of the
manifestation of modeling reasoning among university students.

2. Method
2.1 Participants
The participants in this study were three university students who enrolled in an
educational data mining course at a university in Shanghai, China. These students worked
collaboratively on a data mining project as part of the course. The project involved analyzing
real and complex education data to predict students' answering efficiency in a specificscenario. The scenario was as follows: Some students took an online math test consisting of
two sets of exams (exam A and exam B) with the same type and number of questions. Eachtest lasted for 30 minutes, and once the time expired, students were automaticallyterminated and unable to continue. The project provided students with four data documents:(1) a sequence of students' 30-minute behavior on exam A for Group 1 students, (2) the
efficiency of Group 1 students on exam B, (3) the performance of Group 2 students on examA, and (4) the prediction results of the answering efficiency of Group 2 students on exam B.
The objective of the project was for students to build a model based on the performance ofGroup 1 students on exam A and their efficiency in completing exam B, and then use thismodel to predict the efficiency of Group 2 students in completing exam B based on theirperformance on exam A.
The dependent variable in this project was the answering efficiency of exam B, which
was defined as a binary variable: “Yes” for efficient and “No” for inefficient. Efficiency wasdetermined based on two criteria: (1) whether all questions on exam B were answered and(2) whether the time allocation for each question was reasonable. Reasonableness wasdetermined by comparing the time distribution of all students for each question and setting
the threshold as the shortest possible time for each question (specifically, the first 5% of thetime distribution).
The independent variables included student ID, test type, a unique question number foreach question, question type, specific behaviors of students during the answering process,additional information about their behaviors, and the time stamp of each answering behavior.

2.2 ProcedureThroughout the 9-week course, the students utilized the Zoom meeting tool to engage
in discussions twice a week, with each session lasting 50-60 minutes. The entire discussion
process was recorded using the Zoom meeting feature. These recorded videos weremanually transcribed to obtain a written record of each discussion.

2.3 Data Analysis
The transcribed discussions resulted in a conversation document comprising a total of
82,003 words and 3,209 statements. A statement was defined as a complete conversation inwhich a student participated and contained at least one sentence.
The data analysis employed a bottom-up grounded theory method to derive acomprehensive understanding of the data. Initially, the conversation document was analyzed
sentence by sentence using open coding to identify and extract several categories.
Subsequently, specific categories were grouped together through axial coding to generate
overarching themes. Finally, each category was further refined using selective coding to
extract core concepts and capture the key content of the conversations.
To ensure the reliability of the coding and analysis, two researchers coded respectively
under the guidance of professionals and got good consistency (Kappa=0.81). Furthermore,227

data from different sources, such as the discussion videos and the team's final project
report, were also integrated into the category analysis. This additional data served as across-reference and validation for the coding process.

3. Findings
3.1 Overall Status
To depict the manifestation of university students, the coding results are shown in
Table 1. By coding the conversation document sentence by sentence, and with repeatedreading, comparison and modification, the whole conversation document was finally divided
into 41 activity segments, each representing a complete activity discussion among thestudents in the group, such as understanding the task requirements and dealing with missing
values. After removing repeated activities, 19 first-level codes were identified through open
coding (see Table 1). Axial coding was then conducted to analyze the categories based on
the 19 first-level codes and establish relationships between them. The first-level codes wereorganized chronologically, and their relationships in terms of causality, situation, similarity,
difference, function, and process were considered, resulting in the formation of 8 secondlevel codes (see Table 1). Finally, selective coding was performed, and all categories wereclassified into corresponding core categories that encompassed all the activities in thesession document. Five core categories, referred to as third-level codes, were extracted:
understanding tasks, organizing data, selecting variables, selecting sampling methods, andselecting modeling methods (see Table 1).
Table 1. Coding Results of Bottom-Up Grounded Theory
First-level Codes
Understanding the task
Assigning phased task
Preparing software environment
Dealing with missing values
Exploring preprocessing data
Preprocessing data
Reflecting on preprocessing data
Exploring understanding data
Reflecting on understanding data
Exploring
independent
variables
selection
Reflecting on independent variables
selection
Exploring sampling methods
Understanding sampling
Reflecting on sampling methods
Exploring modeling process
Reflecting on modeling process
Selecting model
Evaluating model
Optimizing model

Second-level Codes
Understanding thetaskPreparing softwareenvironment

Third-level Codes

Preprocessing data

Organizing data

Understanding
task

the

Understanding data
Selectingindependent
variables
Selecting sampling
methodUnderstanding
modeling
process

Selecting variables

Sampling

Modeling

Selecting model

By summarizing the second-level and third-level codes, it was observed that themodeling reasoning process of university students in the educational data mining project
followed a spiral pattern. This process involved the steps of defining the problem, generating228

and selecting variables and attributes, selecting modeling and sampling methods, organizing
and structuring data, exploring and analyzing data, interpreting data, and presenting theresults (see Figure 1). The process went through several iterations, with the students
continuously optimizing and improving their understanding of specific concepts, selection of
independent variables, data processing methods, sampling techniques, and modelingmethods over time, ultimately leading to a progressive spiral form.

Figure 1. The spiral progress of modeling reasoning

3.2 Detailed Manifestation
Since the goal of the data mining project was to build a prediction model for students'
answer efficiency, the entire process of completing the project represented a comprehensivemodeling reasoning process. Additionally, since the students engaged in discussions twice a
week, it was evident from the analysis that they would review the task flow at the beginningof almost every weekly discussion and allocate tasks accordingly. This iterative nature of
their discussions contributed to the spiral development of their modeling reasoning process,wherein each week built upon the progress made in the previous week. Therefore, todemonstrate the spiraling process of modeling reasoning, relevant conversation fragmentsthat were important and non-repetitive were selected based on the chronological order oftask completion.
3.2.1 Understanding the meaning of the task
After reading the task description document, the students (referred to as student X,
student Y, and student Z) engaged in a discussion to accurately comprehend the task's
purpose, define its objectives, and determine the analytical process and steps [2-104](Numbers in square brackets denote the statement numbers of the conversation document).
Since the process of understanding the meaning of the task was relatively fragmented, andtheir understanding of the task's purpose did not undergo significant changes thereafter, theproject report is used as a demonstration. The students clarified the task's objectives asfollows: "How to predict the answering efficiency of (another) group of students in exam B byanalyzing the data of the existing group of students' answering behavior in exam A and theanswering efficiency in exam B, with the ultimate goal of maximizing the accuracy of theprediction" [Project report, p1] (means the source of information is from page1 of the teamproject report). They also depicted the analysis process in Figure 2 [Project report, p2].
Although the students had a clear understanding of the general process, they encounteredsome difficulties and uncertainties in the execution and comprehension of specific steps. For
instance, student Z expressed hesitation regarding the final result, questioning whether it is a
probability value for a binary variable: "Yes, then what we finally get should not be, is aclassification of 0 and 1, right? The probability of 0 is probably 40%, and the probability of 1
is 60%, right? That's probably the case, right?" [54]. However, the students demonstrated a229

clear understanding of the general modeling process, enabling them to confirm and clarify
related concepts, explore and select better methods guided by a clear analytical process,and ultimately develop a spiral modeling reasoning process.

Figure 2. The analysis process made by the students
3.2.2 Exploration of model selection
During the initial exploration, the students contemplated the types of predictive models
that should be built. Student Y suggested “examining numerous existing models” [111].
Student X proposed the idea of “an integrated model, considering the limitations of a single
model's predictive performance” [114]. Student X exhibited a thorough understanding ofintegrated models, explaining the concept of re-modeling a single model and providing
examples to elucidate the voting and stacking methods to other students [125,127]. Thisinitial assumption served as a guiding principle for the subsequent modeling process.
3.2.3 Understanding the data
Comprehending the meaning of the data served as a prerequisite for variable selection
and data processing. The students referred to the task description document to understandthe variables in the dataset, their respective representations, and the type of data
[201,204,211]. Student X raised concerns about missing values, although the discussion on
how to handle them remained superficial, yet still acknowledged the importance ofaddressing this issue [201,203]. However, student Z displayed limited ideas regarding
missing value treatment, suggesting their deletion without further elaboration [202].
3.2.4 Data preprocessing
The students encountered difficulties in selecting independent variables from the real
dataset, even after understanding the data's meaning. Extracting information from thedataset proved challenging due to its complex presentation. The students actively engaged
in discussions on data preprocessing, particularly regarding the conversion between long
and wide formats. Student X demonstrated understanding of long data, explaining that eachline represents a record of behaviors for each student [295]. Similarly, student Z expressed
comprehension of wide data, noting the need to combine all the students to analyze"student" as a whole [297]. The mutual conversion between long and wide format data is a
crucial step in data preprocessing for data mining projects, as it facilitates subsequent
analysis and modeling by providing "tidy data" (Wickham, 2014).

230

3.2.5 Selection of independent variables
The selection of independent variables was a crucial step in the modeling reasoning
process as it bridged the understanding of data meaning and the establishment of themodel, providing clues to connect the real world with the data world (Lehrer & Schauble,2004). Through discussion, the students identified two main independent variables forpredicting answering efficiency: "the answering time of each question" and "the number ofquestions answered within the specified time" [395]. Additionally, the students decided toinitially include all variables and later choose the most influential ones through variable
analysis [396]. This approach demonstrated the generation and selection of variables andattributes in the modeling reasoning process, incorporating problem definition, data
understanding, and exploratory data analysis (EDA) to inform important modeling decisions.
3.2.6 Selection of sampling method
The selection of a sampling method was an indispensable step in the modelingreasoning process, as it determined the use of samples in model construction. The students
not only employed "principal component analysis" and "factor analysis" as part of EDA toselect independent variables [778], but also demonstrated understanding and discussion ofsampling methods. Student X initially perceived sampling as "data segmentation" anddescribed the ten-fold cross-validation method as dividing the data into ten parts, repeatedly
selecting one part as the test set while using the others as the training set [780]. Afterextensive discussions and trials, the students ultimately chose the ten-fold cross-validation
sampling method with three repetitions based on the results of model evaluation andoptimization [Project report, p4].
3.2.7 Selection of modeling method
The selection of a modeling method directly influenced the modeling results. However,in educational data mining projects, specific modeling methods were often chosen based on
the results of evaluation models. The students utilized data analysis software RStudio toconvert complex probability formulas into specific codes, facilitating the implementation andanalysis of different modeling techniques. The selection of modeling methods followed theprinciples of EDA, saving time and allowing the students to focus more on the data itself
rather than the application of intricate statistical probability formulas. The students used
RStudio and focused on understanding how the code was implemented when choosing themodeling method, considering the previously mentioned integrated modeling approach,specifically the voting and stacking methods [1145, 1151]. The choice of modeling method
primarily considered the kappa value, which aligned with the EDA approach of focusing on
the results and outcomes rather than becoming overly fixated on the modeling method itself[1149].
3.2.8 Model evaluation
Model evaluation closely followed the model selection process and significantly
impacted all previous decisions, including the choice of independent variables, sampling
method, and modeling method. The evaluation of the model focused on two key metrics:
kappa and AUC [Project report, p4]. During the evaluation, the students primarily paid
attention to the kappa value, while AUC was not considered at this stage. Different modelingmethods produced varying kappa values, allowing for a direct comparison of their strengths
and weaknesses. For instance, "regression is 0.15, decision tree is 0.17, and random forest
is also 0.17" [1724]. This comparison revealed that both random forest and decision tree
methods performed similarly and better than the regression method.
3.2.9 Model Optimization

231Model optimization involved adjusting the input model data and parameters based on
the evaluation results. The evaluation outcomes influenced the selection of independent
variables and data preprocessing. The students attempted to optimize the model by
adjusting its parameters, referred to as parameter adjustment. The presence of underfitting
or overfitting was an important consideration during model optimization and closely tied tothe model's parameters. Underfitting occurred when the model failed to predict results
accurately, while overfitting referred to the model performing exceptionally well on sampled
data but lacking generalization ability to apply to the entire population. The AUC value partly
reflected the model's generalization ability. The students demonstrated an understanding ofunderfitting, overfitting, and generalization ability, engaging in discussions about theseconcepts: "It’s under-fitting, so it lacks an ability to identify different independent variables,
so he finally gets the same probability" [3076]; "Does underfitting mean no generalization
ability? " [3080]; "Sure, it's too precise " [3081].

4. Discussion
Data literacy has emerged as a crucial component of 21st-century skills in the era of
big data, encompassing various aspects such as modeling reasoning, data aggregation,
context understanding, and data visualization (Rubin, 2019). Among these, modelingreasoning plays a significant role in establishing connections between the real world and thedata world, enabling individuals to define, explain, and predict phenomena based on data
and theories (Aridor & Ben-Zvi, 2017). However, there is limited understanding of how
university students manifest modeling reasoning, which can impede the development of data
science subjects in universities and hinder the cultivation of students' data literacy.
Therefore, this study aimed to explore university students' manifestation of modelingreasoning through a case study approach.
The findings of this study reveal that the process of modeling reasoning among
university students follows a spiral development form in educational data mining projects.
This spiral form is observed due to the iterative and continuous optimization nature ofmodeling reasoning itself, as well as its role as the framework for data mining activities. The
analysis process made by the students (see Figure 2) is divided into three parts: data
exploration and preprocessing, modeling and application, results and feedback, which
demonstrates their clear understanding of the general modeling process that enables thestudents to develop a spiral progress of modeling reasoning (see Figure 1). This spiraldevelopment form shares similarities with the integrated modeling approach (Bakker &Hoffmann, 2005), which also exhibits a spiral rise in its entirety. However, in the current
study, university students utilized the idea of integration to link the real world and the dataworld at the initial stage of the modeling reasoning process, specifically during the task
understanding phase. Consequently, the spiral progressive form was manifested as circular
steps of modeling reasoning, accompanied by a deepening understanding of relevant
concepts. Establishing the connection between data and probability is crucial in data mining
activities (Konold & Kazak, 2008), which is also evident in this study through repeatedunderstanding of sampling methods and repeated selection of modeling methods.
Teaching data science or big data remains an area with limited research (Saltz &Heckman, 2015). This study not only sheds light on the manifestation of university students'
modeling reasoning but also provides insights into cultivating data literacy in universities.
Project-based learning has been recognized as an effective teaching method in statisticaleducation, promoting the development of statistical reasoning and data literacy (Ben-Zvi etal., 2018). Additionally, in the era of big data, exposing students to complex and realscientific datasets facilitates the development of statistical thinking and data literacy (Saltz &Heckman, 2015). The present study aligns with these findings by employing a project-basedlearning approach, providing students with real, unprocessed data to tackle authentic
problems. This approach significantly enhances and develops students' modeling reasoning
abilities (Konold & Kazak, 2008). Particularly in the establishment of data science programsin universities, the training of students' data literacy holds significant importance. While well-232structured problems and small dataset cases have proven effective in training students' data
literacy in the past (Ben-Zvi et al., 2018; Koparan & Güven, 2015), the era of big datanecessitates consideration of ill-structured problems and real, complex, and massive
datasets within a project context when implementing project-based learning to cultivate
university students' data literacy.
In this study, the utilization of RStudio, a data analysis software, allows students to
explore the data analysis process through exploratory data analysis (EDA), transcending thelimitations of understanding statistical formulas and probability models alone (Moore, 1997).
While this approach enables students to focus on the analysis process in real data mining
projects, it is important to investigate whether the excessive focus on result-oriented EDA
may have adverse effects on the cultivation of students' data literacy. Furthermore,computational thinking holds significant importance in modeling reasoning activities. In the
educational data mining project examined in this study, the field of data science
encompasses not only knowledge of mathematical statistics and specific domain knowledge(e.g., education-related knowledge), but also knowledge of computer programming(Rosenberg et al., 2019). Given the reliance on data analysis software RStudio,
understanding and proficiently utilizing coding is crucial during the programming process.
The students encountered difficulties and had to halt the modeling reasoning process due tounfamiliar code, emphasizing the need for computational thinking. This study highlights theimportance of preparing the software environment before initiating the modeling reasoning
process among university students. Consequently, further research is warranted to explore
the relationship between computational thinking and modeling reasoning.

5. Conclusion
In conclusion, this study contributes to the understanding of university students'
manifestation of modeling reasoning, shedding light on the cultivation of data literacy inhigher education. The findings reveal that the process of modeling reasoning among
university students follows a spiral development form in educational data mining projects.
Besides, it is suggested to apply project-based learning and the utilization of real, complex,
and massive datasets in developing students' statistical thinking and data literacy. However,the potential impacts of result-oriented EDA on students' data literacy and the significance ofcomputational thinking in modeling reasoning warrant further investigation. By addressingthese areas, educators can enhance their pedagogical approaches to effectively cultivate
students' data literacy in the era of big data.

Acknowledgements
This essay was funded by the Special Fund for Postgraduates of East China Normal
University to Participate in International Conferences. I would like to express my sincere
gratitude to Associate Professor Bian Wu for his meticulous guidance and support
throughout this research project.

References
Aridor, K., & Ben-Zvi, D. (2017). The Co-Emergence of Aggregate and Modelling
Reasoning. Statistics Education Research Journal, 16, 38-63.
Bakker, A., & Hoffmann, M.H. (2005). Diagrammatic Reasoning as the Basis for Developing
Concepts: A Semiotic Analysis of Students' Learning about Statistical Distribution. Educational
Studies in Mathematics, 60, 333-358.
Ben-Zvi, D., Makar, K., & Garfield, J. (2018). International handbook of research in statistics
education. Cham: Springer.

233

Biehler, R., Frischemeier, D., & Podworny, S. (2017). Elementary Preservice Teachers' Reasoning
about Modeling a "Family Factory" with TinkerPlots--A Pilot Study. Statistics Education Research
Journal, 16, 244-286.
Conway, B., Martin, W. G., Strutchens, M.E., Kraska, M.F., & Huang, H. (2019). The Statistical
Reasoning Learning Environment: A Comparison of Students’ Statistical Reasoning
Ability. Journal of Statistics Education, 27, 171 - 187.
Dvir, M., & Ben-Zvi, D. (2018). The role of model comparison in young learners’ reasoning withstatistical models and modeling. ZDM, 50, 1183-1196.
Gould, R.G. (2017). Data Literacy is Statistical Literacy. Statistics Education Research Journal, 16,
22-25.
Gravemeijer, K.K. (1999). How Emergent Models May Foster the Constitution of Formal
Mathematics. Mathematical Thinking and Learning, 1, 155-177.
Konold, C.E., & Kazak, S. (2008). Reconnecting Data and Chance. Technology Innovations in
Statistics Education, 2.
Koparan, T., & Güven, B. (2015). The effect of project-based learning on students’ statistical literacy
levels for data representation. International Journal of Mathematical Education in Science andTechnology, 46, 658 - 686.
Lehrer, R., & English, L. (2018). Introducing children to modeling variability. In D. Ben-Zvi, J. Garfield,& K. Maka (Eds.), International handbook of research in statistics education (pp. 229-260).
Cham: Springer.
Lehrer, R., & Schauble, L. (2004). Modelling natural variation through distribution. American
Educational Research Journal, 41(3), 635-679.
Lehrer, R., & Schauble, L. (2010). What Kind of Explanation is a Model. In M. K. Stein (Eds.),
Instructional explanations in the disciplines (pp. 9-22). New York, US: Springer.
Moore, D.S. (1997). New Pedagogy and New Content: The Case of Statistics. International Statistical
Review, 65.
Rosenberg, J.M., Lawson, M.A., Anderson, D., Jones, R.S., & Rutherford, T. (2019). Making Data
Science Count In and For Education. Research Methods in Learning Design and Technology.
Rubin, A. (2019). Learning to Reason with Data: How Did We Get Here and What Do We
Know? Journal of the Learning Sciences, 29, 154-164.
Saltz, J., & Heckman, R. (2015). Big Data science education: A case study of a project-focused
introductory course[J]. Themes in Science & Technology Education, 8(2), 85-94.
Setiawan, E. P., & Sukoco, H. (2021). Exploring first year university students’ statistical literacy: A
case on describing and visualizing data. IndoMS-Journal on Mathematics Education, 12(3), 427448.
Wickham, H. (2014). Tidy data. Journal of statistical software, 59(10),1-23.
Wild, C. J., & Pfannkuch, M. (1999). Statistical thinking in empirical enquiry (with discussion).
International Statistical Review, 67(3), 223-248.

234

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Mathematic Learning-by-Teaching:Video
Creation and Cross-Schools Staging
abc

Te-Yang CHOUa*, Yen-Cheng YEHb &Hong-Ming TUc
Graduate Institute of Network Technology,National University,Taiwan*aaronzhou939@gmail.com

Abstract: With the advancement of technology and the transformation of media,
traditional paper-based teaching is no longer the only way for students to learn. With
streaming media becoming more and more popular, the "video" is a supplementaryresource to be integrated into teaching. In this study, we based on the Interest-Driven
Creator Theory and integrated video creation and the learning-by-teaching model todesign learning activities. By creating videos, students can play the role of teachers to
explain their mathematics concepts to their peers (Peer Tutoring) and enhance theirinterest in mathematics. Moreover, Students also can discuss with other peers andenhance their video. Students can construct their mathematical concepts and deepen
their impressions in this activity.
Keywords: Video Creation, Peer Tutoring, Learning-By-Teaching, Cross-Schools
Staging

1. Introduction
In traditional math classes, the teaching tools used by teachers primarily consist of textbooks
and blackboards. The knowledge is usually imparted through direct instruction, with theteacher taking the main role in the teaching process, and students primarily absorbing thecontent presented by the teacher. In mathematics, units and concepts build upon each other.
If the foundation is weak, students may struggle with more complex topics in higher grades,
leading to misunderstandings and difficulties in connecting concepts (Lewin, 2006). This cancreate a bottleneck in their learning, causing a sense of aversion, fear, and anxiety towards
math. Eventually, it may impact their motivation, and some might even give up on the subject.
In learning mathematics, students actively construct logical concepts and deepen theirunderstanding, addressing issues in traditional education. Practical application and hands-on
experiences can cultivate learners' knowledge application, reinforcing existing knowledge(Dewey, 1944). The "learning by teaching" approach transforms student and teacher roles,
enhancing abilities through instruction. Video production facilitates interactive learning.

2. Literature Review
2.1.Interest-Driven Creator Theory
The 21st century is an era of rapid change. In order to survive in such an environment, it isnecessary to possess skills such as problem-solving abilities, collaboration andcommunication skills, critical thinking, adaptability, creativity, and imagination. Lifelong
interest-driven learning is also essential (Chan, 2013; Griffin et al., 2012). To address thechallenges faced in the field of education and bring about sustained impact in Asian education,
Chan and a group of Asian scholars proposed the "Interest-Driven Creator Theory" (IDC) in2018 (Chan et al., 2018). The theory consists of three core concepts: Interest, Creation, andHabit. Interest is a crucial concept that runs throughout the theory, emphasizing that learning
should be driven by interests. When students are interested in their learning, it becomes
enjoyable and efficient, especially when they acquire knowledge that aligns with their interests.
Creation makes learning more effective and rewarding. Habit serves as an indicator of
students becoming interest-driven creators because their behavioral patterns shape the kind235

of individuals they become. Therefore, if students can develop a habit of creating based on
their interests, they will ultimately become lifelong interest-driven creators.
2.2.Learning-by-Teaching
Learning By Teaching is an activity of "Nathematical creation and sharing" that helps students
reduce math anxiety. It adheres to the belief that students "can learn fast, learn slowly, butcannot fail to learn," aiming to enhance students' interest in math learning, self-confidence,and self-efficacy, thereby improving their math learning performance. Students engage in selfdirected learning and grasp mathematical thinking, surpassing the traditional approach ofunderstanding and applying conceptual problem-solving in math learning. In other words, after
learning mathematical concepts, students reconstruct and explain these concepts, mastering
mathematical thinking through practice. Learning By Teaching can be viewed as a form ofpeer tutoring, where learners help each other and learn through the process of teaching (Seel,2011). Learners simultaneously take on the role of both learner and teacher, assisting peers
as teachers. This approach effectively promotes the learning of both oneself and peers(Kobayashi, 2022). Utilizing such teaching methods reduces student pressure andencourages active participation. Peers can grow together, and explaining teaching content toothers enhances one's own learning concepts (Fiorella & Mayer, 2013). Through the "Learning
by Teaching" model, students can demonstrate their acquired mathematical knowledge, gain
a sense of achievement and ownership, and enhance their self-efficacy and interest in learning.

3. Procedure
Video creation constitutes the core activity of this research. The filming approach involves
utilizing physical tools such as paper, pens, whiteboards, and blackboards for explaining
mathematical concepts and showcasing the processes of solving problems. Subsequently,
recording equipment such as cameras, smartphones, and tablets with recording capabilities
is employed to capture the video content. Following the completion of filming, the videos areedited to cater to students' specific requirements, utilizing technological tools. Upon the
conclusion of the editing process, the works are considered finalized.

4. Research Method
Participant
The subjects of this research were fourth and fifth-grade students from three elementary
schools in Taoyuan City. Specifically, there were 21 students from Class 1 of the fourth grade
and 9 students from Class 1 of the fifth grade in School A, as well as 26 students from Class
1 of the fifth grade in School B, making a total of 56 students participating. Initially, the plan
was for the research subjects to be fully engaged in all stages of the experiment. However,
due to the impact of the pandemic, the experimental approach was adjusted to remote learning.
As a result, 4 students were unable to complete the questionnaire within the designated timeframe, leading to an adjusted sample size of 52.

5. Result
The analysis results of using paired sample t-test on the MSLQ motivation questionnaire forthe entire student population indicate a significant difference in students' overall motivation
towards mathematics after engaging in the activity of filming math lessons. It is evident thatstudents have shown a significant improvement in their motivation for learning mathematics.
In terms of the intrinsic goal orientation, the overall students' scores indicate a trend towards
significance, suggesting that students are more likely to engage in learning activities due totheir curiosity and desire for challenges. There is a significant difference observed in the
extrinsic goal orientation aspect. (Table 1)!

236

After the activities during the school phase, there is a significant difference in student ratings,
indicating that for peers, the second video shows an improvement in video quality comparedto the first video. In terms of teacher ratings, there is no significant difference after the activities
during the school phase, but there is a slight improvement observed in the mean scores.
However, in terms of the total score, there is a significant improvement after the activities
during the school phase. Based on the above results, it can be inferred that students, after the
first individual video recording and peer assessment, as well as teacher comments during theschool phase, have shown a significant improvement in video quality in the second video
based on the feedback received from classmates and teachers. (Table 2)
Table1 MSQL Motivation Questionnaire Results

Table2 Video Quality Sample T-test

6. Conclusion
Based on the aforementioned discussion, it was found that the mathematics teaching throughvideo creation activity had a positive impact on students' interest and motivation overall. There
was also an improvement in video quality. Students gained a clearer understanding ofmathematical concepts and a sense of achievement during the activity. Based on the actual
implementation of the activity and the results of the interviews, it can be concluded that themathematics teaching through video creation activity is feasible in different contexts. However,there are some minor details in activity and curriculum design that can be improved or given
more attention to.

Acknowledgements
This study was funded by the National Science and Technology Council of Taiwan (NSTC112-2423-H-008-003) and by Research Center for Science and Technology for Learning,
National Central University, Taiwan.

References
Chan, T.-W., Looi, C.-K., Chen, W., Wong, L.-H., Chang, B., Liao, C. C., Cheng, H., Chen, Z.-H., Liu,
C.-C., & Kong, S.-C. (2018). Interest-driven creator theory: Towards a theory of learning designfor Asia in the twenty-first century. Journal of Computers in Education, 5(4), 435-461.
Dewey, J. (1944). Democracy and education: An introduction to the philosophy of education. New
York: The Free Press.
Fiorella, L., & Mayer, R. E. (2013). The relative benefits of learning by teaching and teaching
expectancy. Contemporary Educational Psychology, 38(4), 281-288.
Griffin, P., Care, E., & McGaw, B. (2012). The changing role of education and schools. In Assessment
and teaching of 21st century skills (pp. 1-15). Springer.
Kobayashi, K. (2022). Learning by teaching face-to-face: The contributions of preparing-to-teach,
initial-explanation, and interaction phases. European Journal of Psychology of Education, 37(2),551-566.
Lewin, T. J. N. Y. T. (2006). As math scores lag, a new push for the basics. New York Times, 14(11),2006.
Seel, N. M. (2011). Encyclopedia of the Sciences of Learning. Springer Science & Business Media.

237

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The role of individual preparation forknowledge construction in collaborativeargumentation: An Epistemic Network
Analysis
Wenli Chena*, Junzhu Sua, Qianru Lyua, Siew Cheng Aileen Chaia, Xinyi Lib, Guo Su & Eng
Eng Ngaa
Nanyang Technological University, SingaporebBeijing Normal University*wenli.chen@nie.edu.sg
Abstract: Through collaborative argumentation, students gain in-depth understanding oflearning content when they build on one another’s knowledge. Although individual
preparation (IP) is found to be effective to foster collaborative learning, the mechanism of
how IP influence the knowledge construction behavior is underexplored. This study
investigated how IP influenced secondary school students in relation to knowledge
construction behavioral patterns when participating in online collaborative argumentation
activities. 20 students participated in two computer-supported collaborative argumentation
lessons with one group with IP, and the other group without. Screen video recordings ofstudents constructing arguments in groups during two lessons were collected andanalyzed. Epistemic Network Analysis was conducted to examine students’ knowledge
construction behaviors in the two lessons with and without IP. The results show that there
were significant impact on students’ knowledge construction characteristics between the
two lessons. Students who did not go through the IP phase tended to exhibit behaviors
related to ideas refinement more than the students who went through the IP phase. The
implications of how to design and implement effective knowledge construction are
discussed.
Keywords: Collaborative argumentation, individual preparation,
construction behaviors, Epistemic Network Analysis

knowledge

1. Introduction
Learning and working environments require people to solve problems collaboratively (Graesser
et al., 2018). To promote students’ collaboration competency, collaborative learning is widely usedin various classrooms (Nokes-Malach et al., 2015). Studies showed that collaborative
learning does not spontaneously bring benefits to students (Menekse & Chi, 2019). Various
strategies have been used to promote students’ effective collaborative learning. One of thestrategies is individual preparation (IP) for collaboration, which provides students with time toprocess the learning materials individually before collaborative learning was implemented(Tsovaltzi et al., 2015). To examine how IP influences the students’ knowledge construction
behaviors, this paper analyzes students’ on-screen knowledge construction behaviors incollaborative argumentation environments with and without IP.
2. Literature Review
2.1 Computer-supported Collaborative Argumentation

238

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education
Argumentation is crucial in solving ill-structured problems in the real world, which typically call forthe collecting of observational data, adherence to formal logic norms, and the reasonable settlingof conflicting viewpoints in discussions (Jonassen & Kim, 2010). As one of the promising
approaches to improve students’ argumentation, computer-supported collaborativeargumentation (CSCA) supports the sharing, constructing, and representing of arguments in
multiple formats. Various online systems with various learning affordances including graph-based
argumentation(Scheuer et al., 2014), representational guidance tools (Hsu et al., 2015), andmicro-scripting or macro-scripting (Noroozi & Hatami, 2018) have been designed to facilitate
students' collaborative argumentation processes. Graph-based CSCA was found to improvestudents’ learning outcomes (Chen et al., 2021). In graph-based argumentation, students use
nodes or bubbles to represent different argument parts and use links or arrows to show how theseparts relate to one another.
2.2 Individual preparation
In collaborative learning activities, various collaboration scripts have been designed andimplemented to foster the quality of students’ contributions and improve the collaboration process(Weinberger & Fischer, 2006). As one of the effective collaboration scripts, individual preparation(IP) before collaboration is defined as “providing students with time to perform activities directed
at processing the instructional material on their own before the collaboration” (Mende et al., 2021).
During IP, students can prepare for the subsequent discussion, e.g., recall their prior knowledge
and experience, create their own arguments, and prepare their individual ideas which they may
compare or combine during the subsequent collaboration (Asterhan & Schwarz, 2007).
Understanding how IP influences students’ subsequent knowledge-construction behaviors during
the collaborative learning phase is of great importance.
2.3 Knowledge construction
Knowledge construction emphasizes that students construct new knowledge from social
interactions (Leach & Scott, 2003). The quality of knowledge construction is dependent on howgroups of students negotiate meaning, come to an agreement, coordinate tasks, and monitor theknowledge consctruction or group work, which are all highly correlated with group performanceand project quality (Lin et al., 2016). The analysis of behavior counts across several categories
has been the main focus of research on online collaborative activities (Christy & Fox, 2014). Thereis a need to capture the process of online collaborative argumentation and to reveal the
relationships between different knowledge construction behavior.
This study investigated on students’ knowledge construction behaviors in a CSCA
context to gain more CSCA design insights. The research question is: What is the difference
between knowledge construction behaviors in the CSCA activities with and without individual
preparation conditions?
3. Method
The case study method was applied to examine the process of one class’s collaborativeargumentation that took place in two English language lessons. In the first lesson, students wereasked to do a 5-minute IP followed by a 15-minute collaboration. In the second lesson, students
were asked to have a 20-minute collaboration without IP. The teacher had rich experience infacilitating collaborative activities in classrooms. A CSCA environment entitled AppleTree System(Chen et al., 2021) was used to support the groups’ collaborative content editing, groupmanagement, and activity monitoring with learning analytics. When students co-constructed andrefined the arguments on the Appletree system, the on-screen behaviors on the Appletree Systemwere recorded for further analysis.
3.1 Participants239

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education
20 secondary school Singaporean students participated in this study. All the students were female
students in Grade 9, aged between 14 to 15. They had experience in collaborative discussions
on the CSCA system, AppleTree system. Students were randomly assigned into groups of fouror five by the teacher. The students in the class are familiar with each other and have previously
participated in group learning activities, although not in the same group settings as in this study.
3.2 CSCA system
The AppleTree system allows students to externalize their knowledge construction processes inan argumentation graph structure (Chen et al., 2021). On the system, students could develop
graph-based argumentation to represent argument elements, and relationships between them, inwhich different bubbles represent ideas, claims, and evidence. Learning analytics, including social
network analysis and contribution count, demonstrate on the system synchronously.
3.3 Design and Procedure
Two collaborative argumentation activities conducted across two weeks were co-designed by theteacher from the school and the researchers in this study. The students were engaged on two
topics: “Foreigners are not welcomed in Singapore. Do you agree?” and “The Singapore
government’s efforts have been effective in managing racial and religious tensions. Do you
agree?” Each group discussed and co-constructed their group argumentation in the joint working
space on the AppleTree platform for 20 minutes. Students sat together with their group members
and worked on the Appleree system with their personalized learning devices. In the lesson withIP, students were asked to write individual ideas for 5 minutes before they continued to write
group ideas collaboratively for another 15 minutes. In the lesson without IP, students were asked
to work collaboratively for 20 minutes without IP.
3.4 Data Collection and Analysis Method
To explore students’ online knowledge construction behaviors, this study collected each student’s
online collaboration process data. To answer the research questions, screen recordings of all thestudents (N = 20) of the two lessons were collected and analyzed. Each student's screen
recording was loaded into the Datavyu (release 1.3; Datavyu Team, 2014), a video coding tool toanalyze students’ online behaviors. The on-screen behaviors demonstrated how students
contributed and shared their ideas to the group through their behaviors.
A coding scheme was developed to examine the characteristics of the students’knowledge construction behaviors. The unit of analysis is each action presented by the student
in the online platform, such as writing one piece of evidence and monitoring the learning analytics
of the platform. Students’ knowledge construction behaviors were coded based on the coding
schemes adapted from the schemes of Curtis and Lawson (2001) and Popov et al. (2019) to
identify different behavioral categories of knowledge construction. Seven main categories of
online collaborative behaviors (Table 1) were identified: organizing, contributing, seeking input,
monitoring learning analytics, refining and revising, social interaction, and activity-related
individual behavior. To adapt to this study context, some codes were added to represent how
students add ideas to the collaborative argumentation diagrams. In total, there are sixteen
subcategories of behaviors, which are illustrated in the coding scheme in Table 1.
Table 1. The coding scheme of on-screen knowledge construction behavior
Behavior categories
Code
Description
Organizing

OGM

Organize group argumentation graph240

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education
Contributing

Seeking input

Monitoring learning analytics

Refining and RevisingSocial interaction
Activity-related
behavior

individual

OIM

Organize individual argumentation graph

ASASE

Add strategy
Add supporting evidence

AOE

Add opposing evidence

RGW

Read group members’ work

INRR

Internet information read

MCC

Monitor contribution count

MSNA

Monitor SNA

MAC

Monitor argumentation count

MMS

Monitor argumentation structure

RE

Revise by elaboration

CHANGE
RARE

Change position
Read activity requirements and extracts

NAV

Navigate CSCA system

INT

Internet information search

ROWW

Read one's own work

TH

Thinking, drafting, idling

Two coders coded the online screen recording data independently using the same coding
scheme. Cohen’s Kappa coefficient was used to calculate the inter-coder reliability and thereliability coefficient value between the two raters was 0.704, which suggested a reasonable levelof agreement between the two coders. The students' online behavior data was organizedaccording to the ENA data standard. The encoded data was analyzed using the ENA web
application. The codes in the scheme for classifying knowledge construction behavior werechosen as the codes. Utterances in both conditions were examined for the co-occurrence ofknowledge construction codes, and the relevant networks were displayed in an ENA space.
4. Results
The ENA results reveal the connections among the students’ different collaboration behaviors in
different conditions for both lessons. Each node in the network graphs (Figure 1) stands for adifferent collaboration behavior code. The links between the various nodes show that the twocodes co-occur. The red edges in Figure 2 indicate the network of the lesson with IP and show
the connections between the nodes that were stronger among Internet information read (INRR),
Organize individual argumentation graph (OIM), Organize Group argumentation graph (OGM),
Thinking, drafting, idling (TH), Read Group members’ work (RGW), and Read one’s own work
(ROWW) than the connections among other behaviors. The blue edges in Figure 1 indicate the
network of the lesson without IP and show the connections between the nodes that wereconnected among Internet information read (INRR), Organize individual argumentation graph(OIM), Organize Group argumentation graph (OGM), Thinking, drafting, idling (TH), Read Group
members’ work (RGW), and Read one's own work (ROWW), monitoring social network analysis(MSNA), and Revise by elaboration (RE) ) than the connections among other behaviors.

241

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Figure 1. Epistemic frames in the lesson with IP (left) and without IP (right).
Epistemic networks of the different knowledge construction behaviors in lessons with andwithout IP reveal the difference between the two conditions. First, connections among organizing
individual argumentation graph (OIM), organizing Group argumentation graph (OGM), thinking,drafting, and idling (TH) centroids are stronger in the lesson with IP than in the lesson without IP.
Second, Internet information reading (INRR), thinking, drafting, idling (TH), monitoring social
network analysis (MSNA), and Revise by elaboration (RE) centroids are stronger connected in
the lesson without IP than in the lesson with IP.
5. Discussion and Conclusion
In summary, students in the CSCA lesson with IP mainly differ from those in the lesson without
IP in which students have more behaviors connected with organizing group argumentation graph,organizing individual argumentation graph, but connected less with refining the argumentationgraphs and monitoring social network analysis during collaborative activities. These behaviors
represented how students’ knowledge construction happens and how these behaviors interact
with each other. In the future design and implementation of argumentation activities, educators
may provide specific guidance, such as encouraging more refining activities when students startcollaborative without IP.
This study also examined the students’ knowledge construction behaviors of collaborativeargumentation in two lessons, with and without IP. The ENA results contribute to and extend ourprevious IP research in argumentation activities by offering empirical evidence for supported
evidence of the knowledge construction process following IP. The results confirmed that (1)students' knowledge construction behaviors presented different characteristics with and without
IP conditions, (2) students tend to have more behaviors connected to refining ideas behavior
without IP conditions than with IP conditions, which corroborates with the past findings that IP
may lead to the solidification of prior misconceptions (Judele et al., 2014). Besides, the different
connections with refining behavior in two conditions indicates that students tend to makedecisions without further refining when they write arguments in groups, which resonates thatstudents with IP may rush to decision-making without integrating each other’s ideas (Lyu et al.,2022). In addition, though the preparatory mechanism confirms the key roles of cognitive
preparation activities before the collaboration (Lam & Kapur, 2018), however, when students
spend them developing their ideas during IP, it might solidify students’ cognition and that might
result in a reduction of students’ refining or integrating ideas in the subsequent collaborativeargumentation activities. Thus, this may result in students losing the opportunity to improving in
the multiple-perspective dimension (Stapleton & Wu, 2015) despite being in a collaborative
learning context, which is one indicator of an effective argumentation. There are a few limitationsof this study. Firstly, the time allocated to students to work on their group argumentation artifacts
is relatively short. Students had less than 20 minutes to construct knowledge and co-create the242Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Educationargumentation graph. Secondly, the sample size of the two lessons was small. Future work may
extend this exploratory application of ENA to more students.
References
Asterhan, C. S., & Schwarz, B. B. (2007). The effects of monological and dialogical argumentation on
concept learning in evolutionary theory. Journal of Educational Psychology, 99(3), 626.
Chen, W., Tan, J. S., & Pi, Z. (2021). The spiral model of collaborative knowledge improvement: an
exploratory study of a networked collaborative classroom. International Journal of ComputerSupported Collaborative Learning, 16(1), 7-35.
Christy, K. R., & Fox, J. (2014). Leaderboards in a virtual classroom: A test of stereotype threat and social
comparison explanations for women's math performance. Computers & Education, 78, 66-77.
Curtis, D. D., & Lawson, M. J. (2001). Exploring collaborative online learning. Journal of Asynchronous
learning networks, 5(1), 21-34.
Graesser, A. C., Fiore, S. M., Greiff, S., Andrews-Todd, J., Foltz, P. W., & Hesse, F. W. (2018). Advancing
the science of collaborative problem solving. psychological science in the public interest, 19(2), 5992.
Hsu, P. S., Van Dyke, M., Chen, Y., & Smith, T. (2015). The effect of a graphǦoriented computerǦassisted
projectǦbased learning environment on argumentation skills. Journal of Computer Assisted
Learning, 31(1), 32-58.
Jonassen, D. H., & Kim, B. (2010). Arguing to learn and learning to argue: Design justifications andguidelines. Educational Technology Research and Development, 58, 439-457.
Judele, R., Tsovaltzi, D., Puhl, T., & Weinberger, A. (2014). Collaborative learning in Facebook: Adverse
effects of individual preparation. 2014 47th Hawaii International Conference on System Sciences,Lam, R., & Kapur, M. (2018). Preparation for future collaboration: Cognitively preparing for learning fromcollaboration. The Journal of Experimental Education, 86(4), 546-559.
Leach, J., & Scott, P. (2003). Individual and sociocultural views of learning in science education. Science& education, 12, 91-113.
Lin, C.-L., Hou, H.-T., & Tsai, C.-C. (2016). Analyzing the social knowledge construction and online
searching behavior of high school learners during a collaborative problem solving learning activity:
A multi-dimensional behavioral pattern analysis. The Asia-Pacific education researcher, 25, 893906.
Lyu, Q., Chen, W., Su, J., & Heng, J. G. (2022). The Interaction Effect of Group Formation and Individual
Preparation on Computer Support Collaborative Engineering Design: An Exploratory Study.
ASCILITE Publications(Proceedings of ASCILITE 2022 in Sydney), e22102-e22102.
Menekse, M., & Chi, M. T. (2019). The role of collaborative interactions versus individual construction on
students’ learning of engineering concepts. European Journal of Engineering Education, 44(5),702-725.
Nokes-Malach, T. J., Richey, J. E., & Gadgil, S. (2015). When is it better to learn together? Insights fromresearch on collaborative learning. Educational psychology review, 27, 645-656.
Noroozi, O., & Hatami, J. (2018). The effects of online peer feedback and epistemic beliefs on students’argumentation-based learning. Innovations in Education and Teaching International.
Popov, V., Biemans, H. J., Fortuin, K. P., van Vliet, A. J., Erkens, G., Mulder, M., Jaspers, J., & Li, Y. (2019).
Effects of an interculturally enriched collaboration script on student attitudes, behavior, and learning
performance in a CSCL environment. Learning, Culture and Social Interaction, 21, 100-123.
Scheuer, O., McLaren, B. M., Weinberger, A., & Niebuhr, S. (2014). Promoting critical, elaborative
discussions through a collaboration script and argument diagrams. Instructional science, 42, 127157.
Stapleton, P., & Wu, Y. A. (2015). Assessing the quality of arguments in students' persuasive writing: A
case study analyzing the relationship between surface structure and substance. Journal of English
for Academic Purposes, 17, 12-23.
Tsovaltzi, D., Judele, R., Puhl, T., & Weinberger, A. (2015). Scripts, individual preparation and groupawareness support in the service of learning in Facebook: How does CSCL compare to social
networking sites? Computers in Human Behavior, 53, 577-592.
Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction incomputer-supported collaborative learning. Computers & Education, 46(1), 71-95.

243

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Students know more than they can tell:
Understanding learners’ ideas of heattransfer via model revision activities
Rajashri PRIYADARSHINI*, Chandan DASGUPTA, Sahana MURTHY
IDP in Educational Technology, Indian Institute of Technology Bombay, India*rajashri13@iitb.ac.in
Abstract: Students' preconceptions significantly influence learning as they bring priorexperiences and ideas alongside formal knowledge. Considering only explicit orconscious conceptual models can misinform teachers' efforts. Framing the problem
using both conscious (verbal-symbolic knowledge, conscious models) andunconscious (implicit models, core-intuitions) conceptual resources offer a powerfulframework. This study interprets the ideas of heat transfer of two grade 11 students in
depth (from a total N=9), through this framework. We look at how Nash and Payal use
different conceptual resources to build an explanatory model of heat transfer in a cup
through the model revision activities. We found different ideas of heat (heat as a
substance, steam as unique property of heat) to be unconsciously influencing how
students understand the concept. Harnessing core-intuitions into conscious models
enabled Nash to build better explanatory models, whereas Payal extensively relied onher verbal-symbolic knowledge. In the macroscopic activities, both students tended touse their judgments from their sensory perceptions, going against their verbal-symbolic
knowledge. Using the metacognitive prompts on Knowledge Forum also supported
student’s critical analysis about their ideas before uploading them on the forum.
Keywords: Conscious models, unconscious models, intuitive knowledge,
representational levels, perceptions, Knowledge Forum, explanatory models

1. Introduction
Students’ preconceptions have a serious impact on their learning of new material. They bringwith them their experiences and ideas of the world while also consciously reasoning about
concepts taught as formal knowledge (Piaget, 2013). Hence, considering student responses
to draw only from their verbal explanations or simply identifying non-normative notions to be
the result of incorrect interpretation of instruction may misinform an instructor’s efforts inharnessing students’ ‘own’ ideas (Taber,2010). Looking at the problem through the models ofknowledge classified as explicit/conscious (decisions we consciously and deliberately take)and implicit/unconscious (decisions taken intuitively, mostly drawn from our mechanistic
interaction with the world) can be a powerful framework to achieve this (Brock, 2015;
Taber,2014). There is much evidence of learning difficulties in science education because of
the ready activation of different implicit knowledge elements, overpowering the need forcanonical or sophistication explanation (Brown and Hammer 2008; Taber and Garcia-Franco,2010). Students use of conscious and unconscious conceptual resources have been reported
previously. Students express ideas where unstated implicit models and intuitions may underly
their conscious models (diSessa,1988; Vosniadou and Brewer, 1992; Watts & Taber,1996). Conscious conceptual resources include the formal knowledge that students stateusing verbal-symbolic knowledge or use conscious models like gesturing or representing
unseen elements in diagrams. Unconscious conceptual resources include core-intuitions
about domain-general ideas about how things work or the implicit models like domain-specifictacit assumptions (Brown, 1993). Students created sophisticated explanatory models ofmagnetism when involved in unconscious ways of thinking (implicit models and core intuitions)(Brown & Cheng, 2010). Different instances such as use of verbal-symbolic knowledge
abstractly, connecting verbal-symbolic knowledge only to implicit models, over reliance on244Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

implicit models, incorrect use of core intuition or anthropomorphic models improperly
impeded students’ explanatory model building processes (Brown, 2017). Implicit knowledge
elements in chemical contexts (“materials naturally react”) were more about the fundamental
nature of the world (Taber,2010) and less about the intuitive sense of the mechanism asreported by diSessa’s in the context of physics(diSessa,1993).
Despite knowing the value of looking at students’ conceptions through the implicit
knowledge elements views, there is dearth of such research in science education. There arestill many unanswered questions such as - do students’ intuitions align with normative models
of science, what activates the use of certain intuitions over symbolic knowledge, etc. (Brock,2015). We situate the current paper’s work across the multiple levels of representation to
uncover students’ understanding of a transport phenomenon, heat transfer. We use
Knowledge Forum (KF) to help us support and elicit students’ scientific knowledge building
process (Lin & Chan,2018). This paper focuses on addressing a few of the reported gaps
through the research question : How do novice students build and refine their explanatorymodels of a scientific phenomenon on Knowledge Forum using explicit and implicit cognitive
resources?

2. Methodology
Students who had completed the topic of thermal properties of matter and, heat transfer wereconsidered for this study using a purposive sampling. A total of nine grade 11 science
students, from schools in the closest cities were part of the study. The schools follow the SSC
and ICSE curriculum. For the purpose of this study, we have reported the analysis of two
students whose data was pertinent to our research. The study was conducted in, a studiostyle collaborative classroom at IIT Bombay, India. Each group had access to a laptop,
materials, and papers. Students’ discourse in the group were captured using a voice recorder,
camcorder, and a screen recorder. Students’ interactions in the simulation platform,
Knowledge Forum was captured by recording the screen in the laptop. Explanatory models on
the Knowledge Forum and the paper sketches were used for artifact evaluation.
To elicit students’ reasoning, we used an explanatory model-based approach, wherestudents could organically discuss and bring out their individual ideas as they worked
together. The study was designed around the multiple representational levels of chemistrythe macroscopic level (experimentation with cups), microscopic level (simulation andmolecular explanation) and symbolic level (explanatory models). The groups were asked tocreate an explanatory model on Knowledge Forum (KF), to explain the heat transfer
phenomenon in a coffee cup. Before the activity began, the researchers demonstrated thedifferent prompts and features of Knowledge Forum, and how to go about using them. The
default prompts in the KF were used throughout the activities as it is as they were suitable toaddress our study. We particularly did not direct students to use the scaffolds and they had
the autonomy to use them. The explanatory models were revised on the Forum by the groups
after each activity. A total duration of approximately 3.5 hours was utilized in the execution of
the activities. The description of the activities is detailed in Table 1.
Table 1. Summary of Activities
Activities

Description

Pre-Knowledge

Researcher conducts active recall about concepts of heat, temperature,and examples
Observe and discuss (single Researcher pours hot water and adds coffee powder. Each group is askedcup)
to observe the cup and discuss their ideas of the cup is cooling down.
Explanatory model on KF Groups create an explanatory model to explain the heat transfer in a coffee
cup on KF
Observe, discuss, choose Groups are given different cups, aluminum foil, milk, coffee, hot water to(multiple cups)
play around and choose the cup which will cool down the fastest.
Explanatory model on KF Groups update their model on KF with new insights or observations

245

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education
Molecular simulation andrepresentation of cup

Groups are asked to create(draw/describe) a molecular representation of
the chosen cup. The groups are asked to interact with the molecular
workbench simulation on heat and temperature during the activity.
Explanatory model on KF Groups are asked to update the molecular representation in the model on
KF
Expertise level
Groups are asked to divide themselves each into one type of heat transfer(conduction, convection, and radiation). Each expert group is given a mix
of examples of conduction, convection and radiation and asked to choose
three examples of their respective expert group and justify their choice)
Explanatory model on KF Groups create an expertise model on KF
Final Discussion

All members return to their home groups and discuss their findings

Explanatory model(updating)

Final updating of models, if any by the groups

We found the framework by Brown (1993) to be best suited to analyze our students’conceptions. The elements of the framework consist of verbal-symbolic knowledge, conscious
models, implicit models, and core intuitions. Verbal-symbolic knowledge: consciously
remembered verbal principles, generalizations, or equations, conscious models: conscious
imaging like gestures, drawings, verbal, or written descriptions, etc. to explain different causal
entities, Implicit models: tacit assumptions about a phenomenon, core intuitions : unconscious
model where students use gut-level intuitions about causal interactions of different entities.
Using these categories of conceptual resources, we attempt to construct an understanding ofstudent’s ideas about the heat transfer phenomenon. Using a comparative-cross analysis
method, we looked at two cases of students from two different groups to understand the ways
different knowledge elements are used. During the initial screening of the data, we found these
two students to be using contrasting reasoning while explaining the heat transfer process
where Nash was attempting to integrate different ideas (his own and from experimenting), andPayal resorted to formally taught ideas often. We thought it to be useful to further investigate
these two cases in detail. We employed a generative interpretation method based on theverbal and non-verbal discourse by students, the diagrams, the explanatory models on the
Knowledge Forum platform and the sketches and explanation on the paper (Clement,2000).

3. Findings
Ideas about heat- Heat as a substance: Nash’s explanations during his observations withthe cup helped us uncover his implicit views about heat. For example, “When we touch it, we
actually take some heat energy to us” tells us about how Nash might unconsciously beconsidering heat as a substance. The properties of substances such transitional, locational,
containable, etc. became implicitly applicable while reasoning about the phenomenon. We
also observed Nash’s overreliance of an observed behavior of heat transfer, steam. Nash uses
conscious models gesturing the way heat moves from cup to the table. Steam was seen as asource or container of heat. In his statement, “Vapor will be transferred, and the table will
absorb the heat energy,” we see how heat is considered to be contained in the vapor,confirming how heat is viewed as a substance. Payal also demonstrates this view of heat insome instances like “when we give it heat, it becomes hot, only then it will conduct right?”
where she considers heat as a substance given. Even in another instance where she mentions“hot molecules of water coming from the shower,” hot molecules are considered as
substances.
Ideas about heat- A phenomenon should have something unique: Unlike Nash, Payal
had another view of heat should have something ‘unique’. In her statement, “There should besomething unique about heat, right? We can sense heat, yeah! ,” she identified ‘steam’ as aunique property of heat. She used it to reason across the macroscopic level activities whilechoosing if something is hot/cold. When she had to choose the cup that cooled down thefastest, Payal relied on her reasons generated by touching the cup and ‘sensing’ the heat of246Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

the cup. This idea about heat could also have supported Payal’s reasoning with sensory
perceptions.
Ideas about heat- Agentic views: Students unconsciously assign different sorts of agencies
to the actors in the given situation. These attribute clusters operate at a deep, unconscious
level of core intuition, and are not articulated in the verbal language. In Nash’s statement,“when we touch it, we actually take some heat energy to us,” the person is the initiating agent,and the touching initiates an agentic behavior that ‘takes’ the heat from the cup (affected
responder, heat taken from the cup by the initiating agent (see Figure 1). In the conversation[Nash: The part of the table in contact with the cup, absorbs the heat from the cup, Riya: (Thecup) it transfers the heat, Nash: “Absorb or transfer? Absorb. Transfer means it gives.”] evenwhen his teammate suggests ‘transfer’ in place of ‘absorb,’ Nash insists on using ‘absorb’ andadds that transfer means ‘it gives’ to indicate the initiating agent as the table, as opposed tothe cup.
Sensory perception over Formal knowledge: During the macroscopic level activities, Nashextensively used his actions of ‘feel’ and ‘sense’ to reason or make judgments. He investigatedperceptually (by holding, touching and watching the materials) as well as scientifically (bysystematically measuring temperatures of different coffee cups using thermometer). Even
though he recalled and stated his verbal-symbolic knowledge about metal being a good
conductor of heat, experimental measurements of the cups along with perceptual ‘feel’
gathered by interacting with the materials including the cup, dominated his judgment. Payal
also encountered multiple instances where her formal knowledge about metals being a good
conductor of heat was questioned because of her perceptual observations where she decided
the degree of hotness of the cups by sensing them. Another reason why the perceptual viewsof both students overpowered their formal knowledge could be because of their implicit views
about how conduction works. They may be considering the metal cup to be hot because it is‘holding’ the heat, hence no heat transfer. While they were correct in observing it, their implicit
ideas about heat impeded their inferences.
Integration of conscious and unconscious knowledge: The most noticeable difference inNash’s and Payal’s ways of scientific reasoning was the use of the different conceptual
resources. Nash, who had a sophisticated explanatory model, largely used his intuitive
knowledge, perceptual observations, and meta-conceptual awareness . He rarely relied on his
verbal-symbolic knowledge, using it only to support or anchor his reasoning. We could also
see him invoke conscious models to explain the mechanism of the phenomena like movementof steam or energy from the cup to the surroundings. Unlike Nash who relied on his verbalsymbolic knowledge occasionally as a support, Payal constantly relied on her verbal-symbolic
knowledge not as a support but as a main means to build the explanatory model. We could
rarely see her draw on her intuitive knowledge. We also see a lack of meta-conceptual
awareness in Payal’s model iteration process.
A framework of Nash’s ideas of heat transfer: Nash’s use of multiple conceptual resources
can help us understand how the resources interacted to build his understanding of heattransfer. Using his most dominant view of heat, we drew a representation to explain theinterplay of all the resources at the macroscopic level activity as depicted in Figure 1. The
verbal symbolic knowledge helped him create two different conscious explanatory models ofheat transfer. These conscious models were implicitly dictated by his views about heat being
an object(substance). Core intuitive resources such as attribute clusters underlined the
explanation of how Nash saw hand causing heat transfer. This representational diagram helps
us “see” briefly the way different conscious and unconscious conceptual resources could work
towards building an understanding of a phenomena for a student.

247

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Figure 1. An example of the Framework diagram for Nash’s dominant ideas in themacroscopic level activities

4. Discussion and Conclusion
From the findings of both cases, we observe how students employ diverse conceptual
resources to construct scientific ideas. Nash developed a sophisticated explanatory model ofheat transfer by effectively integrating his unconscious conceptual resources into his
conscious thinking. Core intuitions and implicit models played a significant role throughout his
sense-making process, with verbal-symbolic knowledge mainly supporting his model creation.
Nash initially leaned on verbal-symbolic knowledge, using definitions of heat and temperature
to initiate his reasoning. His modeling process remained predominantly conscious, using
gestures and diagrams to convey his ideas. In contrast, Payal struggled to access intuitive
conceptual elements and consistently relied on verbal-symbolic knowledge. While she didrevise her explanatory models, research shows that successful explanatory model creation
involves the incorporation of implicit models and core intuitions. Students who solely rely on
verbal-symbolic knowledge or inaccurately connect it to implicit models hinder their
explanatory model development (Brown & Cheng, 2010).
In both cases, students' implicit models of heat influenced their scientific reasoning about
heat transfer. Students often view heat as a substance or object, aligning with early caloric
theory. Previous research on high school students' misconceptions has also identified thistendency to conceptualize heat and temperature as a substance (Erickson, 1985). Payal
searched for a unique aspect in heat, almost treating it as a distinct "sense." Nash, while notexplicitly focusing on heat's uniqueness, also employed the notion of "sense" in explanations.
These notions might hinder students' comprehension of mechanisms like heat transfer.
Instead of seeking common underlying mechanisms, students become fixated on visible
behaviours when reasoning (Wagner, 2006; Taber, 2010). Instructors can seize such
instances to illustrate mechanistic similarities across phenomena. Students' heavy reliance on
perceptual experiences, particularly in macroscopic activities, impacted their verbal-symbolic
knowledge. Because heat is a phenomenological concept, rooted in sensory perception andfirsthand experience, it holds more persuasive power than formal knowledge. This tendency
led students to generate alternative explanations to match their observations—a trend notobserved in studies on electricity or magnetism (Brown, 2017).
During the activities, Nash systematically developed ideas from various sources,
integrating them across representational levels. He connected different heat transfer types in
his group’s model, showcasing a sophisticated understanding of heat transfer (Sunyono et al.,2015). Nash's engagement with prompts like 'my theory' and 'I need to understand' on

248Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Knowledge Forum encouraged metacognitive thinking, leading to in-depth inquiry and refining
his explanatory model (Tan & Loong, 2005). This approach could be valuable in large
classrooms where individual attention is challenging. Our study highlights how students'
conceptual resources influence learning about scientific phenomena, suggesting the benefitof merging intuitive and formal knowledge. Prompting students to explain divergent outcomesfor similar situations can enhance their epistemic motivation (Taber, 2010). Connecting
scientific principles to everyday experiences can further improve students' epistemologicalconsistency (Lemmer et al., 2020), with potential applications in various phenomena. This
article does not explore group dynamics' impact on students' resources, limiting its findings'
strength within the group activity context. The analysis acknowledges potential bias in student
utterances due to the interpretative approach taken. Despite considering the context,
misleading interpretations are possible. The study's scope is confined to two cases; future
research will encompass additional cases for robustness.

References
Alwan, A. A. (2011). Misconception of heat and temperature among physics students. Procedia-Social
and Behavioral Sciences, 12, 600-614.
Brock, R. (2015). Intuition and insight: Two concepts that illuminate the tacit in science education.
Studies in Science Education, 51(2), 127-167.
Brookes, D. T., & Etkina, E. (2015). The importance of language in students' reasoning about heat inthermodynamic processes. International Journal of Science Education, 37(5-6), 759-779.
Brown, D. E. (2017). Implicit conceptual dynamics and students’ explanatory model development inscience. In Converging Perspectives on Conceptual Change (pp. 105-112). Routledge.
Brown, D. E., & Hammer, D. (2008). Conceptual change in Physics. In S. Vosniadou (Ed.), Theinternational handbook of research on conceptual change (pp. 127–154). New York, NY:
Routledge.
Brown, D. E. (1993). Refocusing core intuitions: A concretizing role for analogy in conceptual change.
Journal of Research in Science Teaching, 30(10), 1273–1290
Cheng, M. F., & Brown, D. E. (2010). Conceptual resources in selfǦdeveloped explanatory models:
The importance of integrating conscious and intuitive knowledge. International Journal of Science
Education, 32(17), 2367-2392.
Clement, J. J. (2000). Analysis of clinical interviews: Foundations and model viability. In R. Lesh & A.
Kelly (Eds.), Handbook of research design in mathematics and science education (pp. 547–
589). Mahwah, NJ: Lawrence Erlbaum
diSessa, A. A. (1988). Knowledge in pieces. In G. Forman & P. Pufall (Eds.), Constructivism in thecomputer age (pp. 49–70). Hillsdale, NJ: Lawrence Erlbaum
diSessa, A. A. (1993). Toward an epistemology of physics. Cognition and Instruction, 10(2/3), 105–225
Erickson, G. L., & Tiberghien, A. (1985). Heat and temperature. In R. Driver, E. Guesne, & A.
Tiberghien (Eds.), Children’s ideas in science (pp. 52–84). Philadelphia: Open University Press
Lemmer, M., Kriek, J., & Erasmus, B. (2020). Analysis of students’ conceptions of basic magnetism
from a complex systems perspective. Research in Science Education, 50, 375-392.
Piaget, J. (2013). The construction of reality in the child (Vol. 82). Routledge.
Sunyono, S., Leny, Y., & Muslimin, I. (2015). Supporting students in learning with multiple
representation to improve student mental models on atomic structure concepts. Science
Education International, 26(2), 104-125.
Taber, K., & Garcia-Franco, A. (2010). Learning processes in chemistry: drawing upon cognitive
resources to learn about the particulate nature of matter. Journal of the Learning Sciences,
19,99–142.
Taber, K. S. (2014). The significance of implicit knowledge for learning and teachingchemistry. Chemistry Education Research and Practice, 15(4), 447-461.
Tan, S. C., Loong, D. H. W., & So, K. L. (2005). Fostering scientific inquiry in schools through science
research course and Computer-Supported Collaborative Learning (CSCL). International Journalof Learning Technology, 1(3), 273-292.
Vosniadou, S., & Brewer, W. F. (1992). Mental models of the earth: A study of conceptual change inchildhood. Cognitive Psychology, 24(4), 535–585
Wagner, J. F. (2006). Transfer in pieces. Cognition and Instruction, 24(1), 1–71
Watts M. and Taber K. S., (1996), An explanatory gestalt of essence: students’ conceptions of the‘natural’ in physical phenomena, Int. J. Sci. Educ., 18(8), 939–954.

249

6KLK -/ HW DO (GV   3URFHHGLQJV RI WKH VW ,QWHUQDWLRQDO &RQIHUHQFH RQ &RPSXWHUV LQ (GXFDWLRQ$VLD3DFLILF 6RFLHW\ IRU &RPSXWHUV LQ (GXFDWLRQ,PSDFW RI 95 RQ 3UH6HUYLFH
7HDFKHUV¶ (PSDWK\²²7DNLQJ(GXFDWLRQDO (TXLW\ DV DQ H[DPSOH<DQ-XQ &+(1  <L/LQJ +8 %LDQ :8'HSDUWPHQW RI (GXFDWLRQ ,QIRUPDWLRQ 7HFKQRORJ\ (DVW &KLQD 1RUPDO 8QLYHUVLW\ &KLQD
#TTFRP$EVWUDFW8QGHU WKH JRDO RI SURPRWLQJ D PRUH HTXLWDEOH LQFOXVLYH DQG UHVLOLHQW HGXFDWLRQ
V\VWHP KRZ WR XVH LQQRYDWLYH OHDUQLQJ WHFKQRORJLHV WR GHYHORS WHDFKHUV HPSDWK\ LV D
KRW UHVHDUFK WRSLF LQ WKH ILHOG RI HGXFDWLRQDO HTXLW\ 5HFHQW UHVHDUFK KDV HPSKDVL]HGWKH SRWHQWLDO RI 95 WR EROVWHU HPSDWK\ 7KH XQGHUO\LQJ PHFKDQLVPV RI VXFKRXWFRPH DUH KRZHYHU XQGHUH[SORUHG 7KH SUHVHQW VWXG\ DSSOLHG WKUHH W\SHV RIHGXFDWLRQDO HTXLW\ OHDUQLQJ PDWHULDOV LQFOXGLQJ 95 ' YLGHR DQG JUDSKLF WR
SUHVHUYLFH WHDFKHUV HPSDWK\ GHYHORSPHQW 5DQGRPO\ DVVLJQHG SDUWLFLSDQWV YLHZHG
D GRFXPHQWDU\ IHDWXULQJ D JURXS RI FKLOGUHQ OLYLQJ LQ UXUDO DQG XUEDQ DUHDV LQ D 95
RU ' YLGHR RU JUDSKLF IRUPDW DQG PHDVXUHV RI HPSDWK\ LQ WHUPV RI SHUVSHFWLYH WDNLQJ
DQG HPSDWKLF FRQFHUQ  LPPHUVLRQ HQMR\PHQW DQG GLVFRPIRUW ZHUH FROOHFWHG 5HVXOWVLQGLFDWHG WKDW WKH YLUWXDO UHDOLW\ H[SHULHQFH UHVXOWHG LQ D KLJKHU OHYHO RI HPSDWK\ EXW WKLVHIIHFW ZDV QRW VLJQLILFDQW LQ WHUPV RI SHUVSHFWLYH WDNLQJ ,PPHUVLRQ LV WKH SURFHVV RI
OLQNLQJ YLUWXDO UHDOLW\ H[SHULHQFH ZLWK HPSDWK\ DQG 95 EULQJV KLJKHU LPPHUVLRQ E\
WULJJHULQJ WKH LOOXVLRQ RI VSDWLDO SUHVHQFH ZKLFK ZDV DVVRFLDWHG ZLWK KLJKHU OHYHOV RIHPSDWK\ ,PSOLFDWLRQV RI WKH VWXG\ LQFOXGH LW LV IHDVLEOH WR XVH 95 WR GHYHORSWHDFKHU HPSDWK\ WUDLQLQJ EXW QHHGV WR EH FRPELQHG ZLWK HIIHFWLYH LQVWUXFWLRQDO DFWLYLW\
GHVLJQ DQG VFDIIROGLQJ GHVLJQ XVLQJ HPSDWK\ DV DQ HQWU\ SRLQW FDQ GHYHORS D WZRZD\
VWUXFWXUH IRU WHDFKHUV FRJQLWLYH DQG DIIHFWLYH FXOWLYDWLRQ RI HGXFDWLRQDO HTXLW\ DQGWHDFKHUV HPSDWK\ FXOWLYDWLRQ QHHGV WR EXLOG D ORQJWHUP LQFUHPHQWDO FXOWLYDWLRQ
V\VWHP.H\ZRUGV (PSDWK\ GHJUHH YLGHR HGXFDWLRQ HTXLW\ SUHVHUYLFH WHDFKHU ,QWURGXFWLRQ$V SUHVHUYLFH UHVHUYH WHDFKHUV QRUPDO VWXGHQWV ZLOO IDFH WKH GHYHORSPHQW RIFURVVUHJLRQDO DQG FURVVFXOWXUDO GLYHUVLILHG WHDFKLQJ DFWLYLWLHV LQ WKH IXWXUH 7KH VWXGHQWV WKH\
WUDLQ DOVR FRPH IURP UHJLRQV ZKHUH WKH GHYHORSPHQW RI HGXFDWLRQDO UHVRXUFHV LV XQEDODQFHG
7KHLU HPSDWK\ DELOLW\ IURP WKH SHUVSHFWLYH RI HGXFDWLRQDO HTXLW\ KDV EHHQ GHWHUPLQHG DV .H\
FKDUDFWHULVWLFV RI EHLQJ HIIHFWLYH LQ GLYHUVH WHDFKLQJ DV GHVFULEHG DERYH 'DUOLQJ+DPPRQG
 ,Q WKLV UHJDUG HPSDWK\ LV FRQVLGHUHG DV SDUW RI D FDULQJ VXSSRUWLYH DQG UHVSRQVLYH
WHDFKLQJ VW\OH 0F$OOLVWHU   ZKLFK LV EHQHILFLDO IRU WHDFKHUV WR UHVSRQG WR VWXGHQWVFRJQLWLYH DQG HPRWLRQDO QHHGV DQG WR FDUH DERXW WKH LPSDFW RI VWXGHQWV OLIH FLUFXPVWDQFHVDQG SULRU NQRZOHGJH RQ WKHLU DFDGHPLF GHYHORSPHQW HVSHFLDOO\ WKRVH LQ OHVV GHYHORSHG DUHDVWR DFKLHYH VWXGHQWFHQWHUHG WHDFKLQJ SUDFWLFHV DQG WR EXLOG WUXVW LQ UHODWLRQVKLSV ZLWK VWXGHQWV:DUUHQ   +RZHYHU UHVHDUFK RQ WKH HPSDWK\ RI SULPDU\ DQG VHFRQGDU\ VFKRROWHDFKHUV IURP WKH SHUVSHFWLYH RI HGXFDWLRQDO HTXLW\ VKRZV WKDW PRVW WHDFKHUV NQRZOHGJH DQG
SUDFWLFH LQ HGXFDWLRQDO HTXLW\ DUH QRW LQ V\QF $OWKRXJK VRPH WHDFKHUV DFNQRZOHGJH WKDWHTXLW\ LV RQH RI WKH EDVLF SULQFLSOHV RI LQWHUDFWLRQ ZLWK VWXGHQWV LQ WKH FODVVURRP $OWKRXJKVRPH WHDFKHUV DFNQRZOHGJH WKDW HTXLW\ LV RQH RI WKH EDVLF SULQFLSOHV RI LQWHUDFWLRQ ZLWKVWXGHQWV LQ FODVV WKH\ VWLOO KDYH VWURQJ ELDVHV LQ WHDFKLQJ SUDFWLFH LQ WHUPV RI VWXGHQWV JHQGHUSHUVRQDOLW\ DQG VRFLDO EDFNJURXQG

25095 LV NQRZQ DV DQ HPSDWK\ PDFKLQH LQ WHUPV RI HQKDQFLQJ HPSDWK\ EHFDXVH RI LWVLPPHUVLYH LQWHUDFWLYH QDWXUH DFFHVVHG 6HSW    +RZHYHU WKH FXUUHQW UHVHDUFK RQ
  95 WHFKQRORJ\ DQG WHDFKHU WUDLQLQJ IRFXVHV RQ WKH WUDLQLQJ RI WHDFKHUV WHDFKLQJNQRZOHGJH DQG VNLOOV LQ 95 HQYLURQPHQW DQG LQYROYHV OHVV LQ WKH ILHOG RI HPRWLRQDO FRJQLWLRQVXFK DV HPSDWK\ FXOWLYDWLRQ %DVHG RQ WKLV WKLV VWXG\ DWWHPSWV WR FRPSDUH WKH HIIHFWV RIOHDUQLQJ PDWHULDOV SUHVHQWHG E\ GLIIHUHQW VWUHDPLQJ PHGLD RQ WKH HPSDWK\ DELOLW\ RI
SUHVHUYLFH WHDFKHUV DQDO\]H WKH SRVVLEOH IDFWRUV DIIHFWLQJ WKH HPSDWK\ DELOLW\ RI SUHVHUYLFH
WHDFKHUV 7KH IROORZLQJ UHVHDUFK TXHVWLRQV DUH SURSRVHG
54:KDW DUH WKH GLIIHUHQFHV LQ SUHVHUYLFH WHDFKHUV HPSDWK\ IURP DQ HGXFDWLRQDOHTXLW\ SHUVSHFWLYH LQ WKH SUHVHQWDWLRQ RI 95 YLGHRV ' YLGHRV DQG JUDSKLF PDWHULDOV"
54+RZ GR XVHU H[SHULHQFHV VXFK DV LPPHUVLRQ DQG HQJDJHPHQW LQ 95
HQYLURQPHQWV LPSDFW RQ HPSDWK\" $QG ZKDW DUH WKH LQWHUDFWLRQV ZLWKLQ WKHP"

 0HWKRG$ WRWDO RI  SDUWLFLSDQWV  LGHQWLI\LQJ DV IHPDOH  PDOH DQG  RWKHU ZHUH
UHFUXLWHG IURP D 1RUPDO 8QLYHUVLW\ LQ &KLQD 7KH H[SHULPHQWDO JURXS SUHVHQWHG WKH OHDUQLQJPDWHULDOV LQ 95 95 *5283 Q  DQG WKH FRQWURO JURXS SUHVHQWHG WKH VDPH OHDUQLQJPDWHULDOV DV WKH 95 FRQGLWLRQ LQ ' IRUPDW ' *5283 Q  RU JUDSKLF IRUPDW *UDSKLF*5283 Q  UHVSHFWLYHO\ 7KH OHDUQLQJ PDWHULDO LV D VKRUW GRFXPHQWDU\ FRPSDULQJ WKH
OLYLQJ DQG HGXFDWLRQDO HQYLURQPHQW RI UXUDO DQG XUEDQ FKLOGUHQ 3DUWLFLSDQWV LQ WKH 95
JURXS ZRUH 95 KHDGVHW DQG REVHUYH WKH YLGHR HQYLURQPHQW IURP LQVLGH WKH VSKHULFDO VSDFH
FUHDWHG E\ 95 7KH ' IRUPDW ZDV FUHDWHG E\ FURSSLQJ WKH FHQWHU IUDPH RI D SDQRUDPLF
YHUVLRQ RI WKH  GRFXPHQWDU\ UHWDLQLQJ D WKLUG RI WKH RULJLQDO ILHOG RI YLHZ 7KH JUDSKLFYHUVLRQ ZDV FUHDWHG E\ LQWHUFHSWLQJ NH\ IUDPHV IURP WKH ' YLGHR ZLWK D QDUUDWLRQ DGDSWHGIURP WKH YLGHR FRPPHQWDU\3DUWLFLSDQWV FRPSOHWHG D WRWDO RI WKUHH TXHVWLRQQDLUHV EHIRUH DQG DIWHU WKH
H[SHULPHQW7KH SHUVRQDO LQIRUPDWLRQ TXHVWLRQQDLUH XVHG EHIRUH WKH H[SHULPHQW UHFRUGHG WKHSDUWLFLSDQWV SHUVRQDO LQIRUPDWLRQ VXFK DV JHQGHU PDMRU DQG JUDGH ZKLFK ZDV XVHG WR UHFRUG
LQIRUPDWLRQ RI SDUWLFLSDQWV DQG WR WHVW WKH VXFFHVV RI WKH UDQGRPL]DWLRQ7KH HPSDWK\ VFDOH
XVHG DIWHU WKH H[SHULPHQW LV DGDSWHG IURP 'DYLV ,QWHUSHUVRQDO 5HDFWLYLW\ ,QGH[ 'DYLV  
WZR PDLQ GLPHQVLRQV RI 3HUVSHFWLYH 7DNLQJ DQG (PSDWKLF &RQFHUQ VL[ LWHPV LQ WRWDO ZHUHVHOHFWHG IRU PHDVXUHPHQW 7KH ,95 XVHU H[SHULHFHV TXHVWLRQQDLUH XVHG DIWHU WKH H[SHULPHQWZDV EDVHG XSRQ SUHYLRXVO\ HVWDEOLVKHG LQVWUXPHQWV GHYHORSHG WR PHDVXUH LPPHUVLRQ&KDUOHQH -HQQHWW   HQJDJHPHQW -HDQQH + %URFNP\HUƏ DQG GLVFRPIRUW 6DUD
9HQWXUD 0'Ə  D WRWDO RI  LWHPV DUH LQFOXGHG

 5HVXOWV
5HVHDUFK TXHVWLRQ  LQTXLUHG ZKDW DUH WKH GLIIHUHQFHV LQ SUHVHUYLFH WHDFKHUV HPSDWK\IURP DQ HGXFDWLRQDO HTXLW\ SHUVSHFWLYH LQ WKH SUHVHQWDWLRQ RI 95 YLGHRV ' YLGHRV DQGJUDSKLF PDWHULDOV 7KH $129$ WHVW IRU WKH HPSDWK\ VFDOH VKRZV WKHUH ZDV D VLJQLILFDQW
GLIIHUHQFH LQ HPSDWK\ DPRQJ WKH WKUHH JURXSV )  S  DQG WKLV GLIIHUHQFH
ZDV DOVR VLJQLILFDQW LQ WKH GLPHQVLRQ RI (PSDWK\ &RQFHUQ )  S   ZKLFK LVVLPLODU WR SUHYLRXV VWXGLHV 0RUHRYHU QR VLJQLILFDQW GLIIHUHQFH LQ 3HUVSHFWLYH WDNLQJ DPRQJ
WKH WKUHH W\SHV RI PDWHULDO SUHVHQWDWLRQ SRVVLEO\ EHFDXVH WKH 3HUVSHFWLYH WDNLQJ GLPHQVLRQ LV
D PRUH VWDEOH WUDLW DW WKH OHYHO RI SHUVRQDO YDOXHV WKDW LV QRW LQIOXHQFHG E\ VKRUWWHUP
H[SHULPHQWDO WUDLQLQJ8VLQJ WKH JDPHVKRZHOO PHWKRG WR WHVW HPSDWK\ DIWHU WKH HYHQW VHH
7DEOH   WKH UHVXOW VKRZV WKDW WKH HPSDWK\ RI OHDUQHUV XVLQJ 95 OHDUQLQJ PDWHULDOV LVVLJQLILFDQWO\ KLJKHU WKDQ WKDW RI OHDUQHUV XVLQJ ' PDWHULDOV DQG LV QRW ORZHU WKDQ WKDW RIOHDUQHUV XVLQJ JUDSKLF PDWHULDOV 7KH XVH RI 95 WHFKQRORJ\ KDV D JRRG HIIHFW RQ WKHFXOWLYDWLRQ RI OHDUQHUV HPSDWK\ DELOLW\ ZKLFK DQVZHUV WKH ILUVW UHVHDUFK TXHVWLRQ ,Q FRQWUDVWWR SUHYLRXV VWXGLHV WKH UHVXOWV DOVR LQGLFDWHG WKDW WKH JUDSKLF PDWHULDOV ZHUH PRUH HIIHFWLYH
WKDQ WKH ' PDWHULDOV LQ SURPRWLQJ HPSDWK\251

5HVHDUFK TXHVWLRQ  LQTXLUHG KRZ GR XVHU H[SHULHQFHV VXFK DV LPPHUVLRQ DQG
HQJDJHPHQW LQ 95 HQYLURQPHQWV LPSDFW RQ HPSDWK\ 7KH UHVXOW VKRZV WKH FRUUHODWLRQV RIWKH IRFDO YDULDEOHV LQYROYHG LQ WKH VWXG\ 7KH UHVXOWV VKRZ WKDW LQ WHUPV RI WKH LQWHUQDOFRUUHODWLRQ DPRQJ 95 XVHU H[SHULHQFHV GLVFRPIRUW DQG LPPHUVLRQ VKRZHG D VWDWLVWLFDOO\VLJQLILFDQW QHJDWLYH FRUUHODWLRQ U S DQG LPPHUVLRQ DQG HQJDJHPHQW VKRZHG D
VLJQLILFDQW SRVLWLYH FRUUHODWLRQ U S  ,Q WHUPV RI WKH H[WHUQDO FRUUHODWLRQ EHWZHHQ
95 XVHU H[SHULHQFHV DQG HPSDWK\ WKH FRUUHODWLRQ EHWZHHQ HPSDWK\ DQG LPPHUVLRQ ZDV DOVRVWDWLVWLFDOO\ VLJQLILFDQW U S  7KHVH ILQGLQJV VXJJHVW WKDW SRVLWLYH ,95 XVHUH[SHULHQFHV VKRXOG EH KHOSIXO LQ SURPRWLQJ HPSDWK\ &RQYHUVHO\ DV ZHOO DV QHJDWLYH ,95
XVHU H[SHULHQFHV PLJKW KDYH QHJDWLYH LPSDFWV 7KHUH LV DQ LQWHUQDO RIIVHWWLQJ UHODWLRQVKLSEHWZHHQ WKHVH WZR H[SHULHQFHV 'LVFXVVLRQ
9LUWXDO UHDOLW\ H[SHULHQFH UHVXOWHG LQ D KLJKHU OHYHO RI HPSDWK\ DQG DOVR OHG WR KLJKHUOHYHOV RI HPSDWKLF DWWHQWLRQ LQ HPSDWK\ IRU DQ LQGLYLGXDO IHDWXUHG LQ WKH YLUWXDO UHDOLW\ PDWHULDOFRPSDUHG WR FRQWURO FRQGLWLRQV SUHVHQWLQJ WKH VDPH PDWHULDO LQ D QRQYLUWXDO UHDOLW\ IRUPDWV7KLV ILQGLQJ LV VLPLODU WR SUHYLRXV UHVHDUFK ILQGLQJV 6FKXWWH  &KLHQ   ,Q YLUWXDOHQYLURQPHQW YLHZHUV PD\ IHHO WKHLU HPRWLRQV RU VLWXDWLRQV PRUH VWURQJO\ DV WKH\ DUH FORVH WR
WKHP DQG VKDULQJ WKH VDPH VSDFH &RQWUDU\ WR RXU LQLWLDO K\SRWKHVLV WKH YLUWXDO UHDOLW\H[SHULHQFH KDG QR VLJQLILFDQW HIIHFW RQ SHUVSHFWLYH WDNLQJ $FFRUGLQJ WR 'DYLV UHVHDUFK WKH
SHUVSHFWLYH WDNLQJ LV GLIIHUHQW IURP WKH HPSDWKLF FDUH DQG LV DVVRFLDWHG ZLWK UHODWLYHO\ OHVVHPRWLRQDO UHDFWLYLW\ DQG KLJKHU LQWHUSHUVRQDO IXQFWLRQLQJ DQG VHOIHVWHHP 7KH YLUWXDO UHDOLW\PDWHULDOV VHOHFWHG IRU WKLV VWXG\ IRFXV PRUH RQ WKH HPRWLRQDO DVSHFWV RI XUEDQ DQG UXUDO
HGXFDWLRQ HTXLW\ VWRULHV UDWKHU WKDQ SURPRWLQJ WKH FRJQLWLYH SHUVSHFWLYH ZKLFK LV RQH RI WKHUHDVRQV ZK\ WKH FKRLFH RI SHUVSHFWLYH IRU OHDUQHUV LQ WKH 95 HQYLURQPHQW KDV QRW EHHQ
VLJQLILFDQWO\ LPSURYHG
)XUWKHUPRUH WKH LPPHUVLRQ WULJJHUHG E\ YLUWXDO UHDOLW\ UHLQIRUFHV YDULRXV HPRWLRQDO DQGFRJQLWLYH UHVSRQVHV WKXV HQKDQFLQJ HPSDWK\ 95 EOXUV WKH ERXQGDU\ EHWZHHQ WKH UHDOZRUOG DQG WKH YLUWXDO ZRUOG E\ FUHDWLQJ D YLUWXDO UHDOLW\ HQYLURQPHQW WKDW FORVHO\ UHVHPEOHV WKHUHDO SK\VLFDO VSDFH YLHZHUV JDLQ DQ LPPHUVLYH YLUWXDO UHDOLW\ PHGLD H[SHULHQFH JHQHUDWLQJ
WKH IHHOLQJ RI VKDULQJ WKH VDPH WLPH DQG VSDFH DV WKH FKDUDFWHUV LQ WKH YLGHR DQG SURPRWLQJ
HPSDWK\ WKURXJK WKH VKDULQJ RI SHUVSHFWLYHV 7KH GLVFRPIRUW RI YLUWXDO UHDOLW\ H[SHULHQFHVVXFK DV GL]]LQHVV DQG YRPLWLQJ FDQ LQKLELW YLHZHUV HPSDWKLF IHHOLQJV 7KLV LV EHFDXVH WKHKLJKGLPHQVLRQDO SHUFHSWXDO QDWXUH RI YLUWXDO UHDOLW\ WHFKQRORJ\ FDQ SODFH D GHJUHH RIFRJQLWLYH ORDG RQ WKH OHDUQHU UHVXOWLQJ LQ QHJDWLYH UHDFWLRQV VXFK DV YHUWLJR $V D
SV\FKRORJLFDO WUDLW HPSDWK\ LV UHODWHG WR DQ LQGLYLGXDO V SK\VLFDO FRQGLWLRQ 1HJDWLYH SK\VLFDO
H[SHULHQFH ZLOO LQKLELW DQ LQGLYLGXDO V FRJQLWLYH DQG HPRWLRQDO H[SHULHQFH WKHUHE\ SUHYHQWLQJ
WKH HPHUJHQFH RI HPSDWK\ WR D FHUWDLQ H[WHQW ,Q SUHYLRXV VWXGLHV WKH LPPHUVLRQ SURYLGHG E\
YLUWXDO UHDOLW\ HQYLURQPHQWV ZDV D NH\ IDFWRU LQ WULJJHULQJ KLJK OHYHOV RI HPSDWK\ +RZHYHU WKH
FRQFOXVLRQV RI WKLV VWXG\ VKRZ WKDW WKH DGYDQWDJHV RI YLUWXDO UHDOLW\ WHFKQRORJ\ UHODWHG WR
LPPHUVLRQ GR QRW QHFHVVDULO\ OHDG WR D KLJK OHYHO RI HPSDWK\ ,PPHUVLRQ LWVHOI LV QRW HQRXJK WR
JHQHUDWH FRQFUHWH DQG FRQWH[WXDO FRJQLWLRQ DQG LW PXVW EH DFFRPSDQLHG E\ KLJKTXDOLW\OHDUQLQJ FRQWHQW ,Q WKHVH WHDFKLQJ SUDFWLFHV ZH VKRXOG QRW EH OLPLWHG WR VLPSO\ FRQYHUWLQJ 'YLGHR LQWR SDQRUDPLF YLGHR SOD\EDFN EXW VKRXOG WDNH DGYDQWDJH RI WKH LPPHUVLRQ DQG
SDUWLFLSDWLRQ RI 95 WDNH DGYDQWDJH RI WHFKQRORJ\ WR FDUU\ RXW HIIHFWLYH WHDFKLQJ DFWLYLW\
GHVLJQ DQG LQWHJUDWH WHDFKLQJ LQWHUYHQWLRQ LQWR SUHVHUYLFH WHDFKHUV¶ 95 H[SHULHQFH

252

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Enhancing student teachers’ collaborativeinterdisciplinary design throughknowledge-building activities
Pei-Yi LIN a*a Department of Education and Learning Technology, National Tsing-Hua University, Taiwan*peiyilin@mx.nthu.edu.tw
Abstract: This study aims to investigate student teachers’ interdisciplinary designcollaboration and integrated teacher knowledge in a knowledge-building environment.
A class of 14 participants was divided into four groups to discuss interdisciplinary
learning design ideas using Knowledge Forum. Data include (1) a pre- and postquestionnaire designed to investigate the participants' interdisciplinary design
capacities (e.g., teacher efficacy, epistemic fluency, interdisciplinary designcommunication, integrated teacher professional knowledge) and (2) records of online
activities such as numbers of notes generated, modified, and read. The findings
showed that student teachers’ perception of teacher efficacy to design interdisciplinary
learning, epistemic understanding, interdisciplinary design communication,
pedagogical content knowledge (PCK), technological pedagogical knowledge (TPK),
and integrated technological, pedagogical, and content knowledge (TPACK) weresignificantly enhanced. The online knowledge-building activities were more likely torelate to student teachers’ epistemic understanding and integrated teacher knowledge(i.e., TCK and TPK). The significance of practice-based, design-oriented activities fordeveloping interdisciplinary collaboration in teacher preparation is discussed.
Keywords: teacher efficacy, epistemic fluency, interdisciplinary
communication, teacher knowledge, knowledge-building activities

design1. Introduction
In today's world, interdisciplinary design practices are crucial in preparing students totackle multifaceted, complex problems. The interdisciplinary curriculum combines skills andcontent from several subjects, allowing students to make meaningful connections acrosssubjects. Traditional teacher training programs focus on single-disciplinary knowledge.
Teachers should break the discipline boundary to consider students' interests when creatinginterdisciplinary curricula and relate the content to their lives (Chiang et al., 2020), recognize
and participate in pedagogical design discourses to enhance their epistemic understanding
when approaching problems and goals with different knowledge bases (Bopardikar et al.,2021). When teachers collaborate to create lesson plans from different disciplinary knowledge,
they externalize design thinking and contribute design ideas based on their expertise tocommunicate and develop new understanding. Research is needed to ascertain how teachers
from different fields establish a shared account of design tasks to synthesize their ideas tocreate interdisciplinary learning designs.
Interdisciplinary learning design entails how students learn to apply, conceptualize,
model, and generate solutions from multiple disciplines (Capobianco et al., 2018). Teachers
must integrate various subject knowledge and pedagogies to engage students. Technologies
can support student engagement and provide adaptive scaffolds to explore an in-depth
understanding of what they have learned (Ouyang & Chang, 2019). Hence, teachers’ design
capacities to integrate content knowledge from multiple disciplines and pedagogical uses oftechnologies are critical (Chai et al., 2011). TPACK is the synthesized knowledge for
integrating technologies into learning content and classroom teaching. When teachers designinterdisciplinary learning, they make a pedagogical decision in deciding how to present thelearning content (i.e., PCK), consider how to use technologies to address specific learning
contents (i.e., TCK), discuss how to use technologies to support student learning (i.e., TPK),253

and knowledge of using technologies to facilitate students learn and create an understandingof subject matters (i.e., TPACK) (Chai et al., 2011). When teachers design, they share theirunderstanding from different fields and consider how to engage students to approach solutions
to the problems that may require fluent design talks about integrating technologies,
pedagogies, and multiple learning content in design communities.
Knowledge building could facilitate student collaboration and take collective
responsibility for improving ideas and pursuing more coherent explanations (Scardamalia &Bereiter, 2014). It enables student teachers to work adaptively with their initial design ideas,
and through continuous discourse and redesign, they view community discourses as
progressive problem-solving opportunities to generate better ideas and solutions (Yang et al.,2020). This study investigates student teachers' interdisciplinary lesson design by focusing on
their efficacy in designing interdisciplinary learning, epistemic fluency, and interdisciplinary
communication in developing integrated technological, pedagogical, and content knowledge
practices through knowledge-building activities. The main research questions of this study are:Did student teachers enhance their efficacy, epistemic fluency, interdisciplinary designcommunication, and teacher knowledge across knowledge-building activities? Were their
online knowledge-building activities related to the above variables?

2. Methodologies
2.1 Participants and learning activities
The participants were 14 students in an interdisciplinary learning course from a national
university in Taiwan. They were heterogeneous and divided into four groups, three to four
people a group, specializing in different domains. In this course, students were required todesign an interdisciplinary learning micro-teaching plan based on sustainable development
goals (SDGs) and present the lesson plan as course reports in a seven-week-long learning.
The instructor adopted knowledge-building principles to engage students in design activities.
For instance, in the first week, the instructor encouraged the students to provide ideas basedon the SDGs topic they were interested in in Knowledge Forum (KF). Then, students read anddiscussed their initial ideas with group members before deciding on their group micro-teaching
topic. Each week, there were lectures and group discussions based on the weekly assigned
papers for two hours; the participants recorded their design ideas in KF and discussed themfor an hour. The data of this study focused on the latter part.

2.2 Data source
Data sources included (1) online activity logs in Knowledge Forum and (2) pre- and postquestionnaires, including teacher efficacy to design interdisciplinary learning, epistemic
fluency, interdisciplinary design communication (variables were validated in Lin et al., 2022),
PCK, TCK, TPK, and TPCK (variables were validated in Chai et al., 2011). All items werescored on a 7-point Likert scale (i.e., 1 = strongly disagree to 7 = strongly agree).

3. Results
3.1 Student teachers’ perception of interdisciplinary design capacities
A paired sample t-test was conducted to analyze student teachers’ perceptions of efficacy,
epistemic fluency, interdisciplinary design communication, and the development of integrated
teacher knowledge. The results indicated significant differences in teacher efficacy (pretest: M = 5.86, SD = 0.81; post-test: M = 6.64, SD = 0.44; t = -3.91, p < 0.01), epistemic
fluency (pre-test: M = 5.33, SD =1.06; post-test: M = 6.36, SD = 0.30; t = -4.14, p < 0.01),
interdisciplinary design communication (pre-test: M = 5.62, SD = 0.93; post-test: M =6.60, SD = 0.47; t = -4.07, p < 0.01), PCK (pre-test: M = 5.14, SD = 1.25; post-test: M =5.98, SD = 1.00; t = -2.65, p < 0.05), TPK (pre-test: M = 5.29, SD = 1.27; post-test: M =6.26, SD = 0.64; t = -2.83, p < 0.05), and TPACK (pre-test: M = 5.38, SD = 1.31; posttest: M = 6.23, SD = 0.67; t = -2.41, p < 0.05). However, TCK (pre-test: M = 5.39, SD = 1.29;
post-test: M = 5.33, SD = 0.32; t = 0.19, p = 0.85) showed no significant enhanced.

3.2 Relations among student teachers’ KF activities and perception of interdisciplinary254

design capacities
This study calculated the changes in student teachers’ perceptions of interdisciplinary designcollaboration (i.e., post-test – pre-test) and analyzed their correlations to online
interdisciplinary design discussion (e.g., # of notes generated, modified, and read). The
Spearman’s rho correlation analysis results indicated that idea generation was significantly
related to their TCK and TPK (ȡ= 0.60 and ȡ= 0.59, p < 0.05), and idea modification was
significantly related to epistemic efficacy (ȡ= 0.62, p < 0.05).

4. Discussion
Previous literature on interdisciplinary curriculum and instructional design focuses on students'
learning processes and competencies (e.g., Jiang et al., 2019), offers instructional guidance(e.g., Prain et al., 2023), and documents teachers' collaborative design processes (e.g.,
Bopardikar et al., 2021) through case studies. However, how student teachers collaborate indesigning interdisciplinary learning and integrating multiple content knowledge, pedagogies,
and technologies needs to be explored. The present study investigates student teachers'interdisciplinary design collaboration in online knowledge-building activities by employing preand post-questionnaires to assess their perceptions. The results indicated a significant
enhancement in their efficacy in designing interdisciplinary learning, interdisciplinary designcommunication, epistemic fluency, PCK, TPK, and TPCK.
Regarding their participation in knowledge-building activities, student teachers' idea
generation was related to the change in TCK and TPK. This may imply that the online
knowledge-building environment may make their design ideas explicitly to discuss integrating
technologies, learning content, and pedagogies. Furthermore, this study shows the potential
relation of student teachers' idea modification relates to their epistemic efficacy. When they
engage in collaborative interdisciplinary design, they share initial design ideas from different
disciplines, and after discussion, they may formulate new information or restructure their
existing knowledge. This study proves that knowledge-building activities enrich opportunities
to support the social construction of interdisciplinary knowledge and learning activities.

Acknowledgments
This work was financially supported by The National Science and Technology Council, Taiwan(NSTC 111-2410-H-007-003-MY2).

References
Bopardikar, A., Bernstein, D., Drayton, B., & McKenney, S. (2021). Designing educative curriculum
materials in interdisciplinary teams: designer processes and contributions. Instructional
science, 49(2), 249-286.
Capobianco, B. M., DeLisi, J., & Radloff, J. (2018). Characterizing elementary teachers’ enactment ofhigh Ʈ leverage practices through engineering design Ʈ based science instruction. Science
Education, 102(2), 342-376.
Chai, C. S., Koh, J. H. L., & Tsai, C. C. (2011). Exploring the factor structure of the constructs oftechnological, pedagogical, content knowledge (TPACK). Asia-Pacific Education Researcher,20(3), 595-603.
Chiang, F. K., Chang, C. H., Wang, S., Cai, R. H., & Li, L. (2020). The effect of an interdisciplinary
STEM course on children’s attitudes of learning and engineering design skills. International Journalof Technology and Design Education, 1-20.
Jiang, S., Shen, J., & Smith, B. E. (2019). Designing discipline-specific roles for interdisciplinary
learning: two comparative cases in an afterschool STEM+ L programme. International Journal of
Science Education, 41(6), 803-826.
Lin, P. Y., Chai, C. S., Di, W., & Wang, X. (2022). Modeling Chinese Teachers’ Efficacies for the
Teaching of Integrated STEM With Interdisciplinary Communication and Epistemic
Fluency. Frontiers in Psychology, 13.
Ouyang, F., & Chang, Y. H. (2019). The relationships between social participatory roles and cognitive
engagement levels in online discussions. British Journal of Educational Technology, 50(3), 13961414.
Prain, V., Xu, L., & Speldewinde, C. (2023). Guiding Science and Mathematics Learning when Students
Construct Representations. Research in Science Education, 53(2), 445-461.

255

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

From Individual Ideation to Group
Knowledge Co-Construction:
Comparison of High- and Lowperforming Groups
Wenli CHENa*, Guo SUa*, Xinyi LIb*, Qianru LYUa, Junzhu SUa, Siew Cheng Aileen CHAIa& Eng Eng NGaaNational Institute of Education, Nanyang Technological University, SingaporebFaculty of Education, Beijing Normal University, China*wenli.chen@nie.edu.sg*nie22.sg4560@e.ntu.edu.sg*li_xinyi@mail.bnu.edu.cn
Abstract: This study compares the high- and low-performing groups’ knowledge coconstruction process in the context of computer-supported collaborative argumentationfrom epistemic, argument, and social perspectives. Product analysis, lag sequential
analysis, Sankey diagram visualization, and social network analysis were used toanalyze groups’ written argumentation artefacts, on-screen behaviors, and online
interactions. Results show that the high-performing group students demonstrated ahigher level of engagement and cognitive elaboration than the low-performing group.
The high-performing group was more competent in integrating various argumentation
elements than the low-performing group. And the students in the high-performing grouptended to contribute equally to their group work. The implications of the findings indesigning and implementing knowledge co-construction activities are discussed.
Keywords: Knowledge co-construction, collaborative argumentation, computersupported collaborative learning

1. Introduction
Collaborative argumentation is an effective way for students to learn together. It deemphasizes
the social interactions that take place when students discuss divergent claims and come to aconsensus (Gao et al., 2022濎澳 Han et al., 2021). Existing research studies show thatcollaborative argumentation benefits students’ knowledge construction in classroom practice(Asterhan & Schwarz, 2016). Computer-supported collaborative argumentation (CSCA)
supports students to practice argumentation simultaneously through writing and discussion(Baker, 1999). In CSCA, students with differing views critically debate a subject. Ideally, theycome to an agreement and collaborate to construct argumentative knowledge after comparing,assessing, and synthesizing multiple points of view (Han et al., 2021). In order to learn about
argumentation and content knowledge, students generate and co-construct arguments byinteracting with their learning partners (Andriessen et al., 2003). Argumentative knowledge coconstruction is one of the important competencies for joint problem-solving (Vuopala et al.,2019).
To examine multiple dimensions of the knowledge co-construction process incomputer-supported collaborative learning (CSCL), Weinberger and Fischer (2006) proposeda framework that contains the epistemic, argument, and social-mode dimensions. The
epistemic dimension describes how learners work on the knowledge construction task (Fischer
et al., 2002). This dimension looks into whether students are working on the job at hand(engaged in on-task conversation) or if they are off-task. The argument dimension focuses on
how learners construct arguments with regard to defining formal relationships between specific256elements of arguments, such as claims, data, and warrants (Fischer et al., 2007). The
dimension of social modes indicates how learners interact with each other.
Previous studies attempted to unpack the complex and dynamic process of groupstudents’ argumentative knowledge construction. For example, Ouyang et al. (2023)
developed an integrated discourse analysis approach to examine how the group's discourse
movements, structures, and turn-taking processes evolve over time by analyzing thecollaborative argumentation from cognitive, metacognitive, and social perspectives. Lu et al.
(2011) explored whether discourse moves, sequences, and participant features were related
to two types of justifications (evidence and explanations) during online argumentative
discussions. Most of the existing studies attempted to study the group’s collaborativeargumentation process itself without considering how individual students in the groupcontribute to the knowledge co-construction of the group. Compared with direct collaboration,
adding individual ideation before collaboration may require different epistemic activities,
making it worthwhile for a detailed investigation (Valero et al., 2022).

2. Literature Review
Knowledge co-construction refers to collaborative processes in which learners interact with
one another in order to build new knowledge and enhance the knowledge and understandingof each individual student (Janssen et al., 2009). These productive interactional processes
include asking thought-provoking questions, exchanging ideas, negotiating and providing
justified arguments in order to build new knowledge (Arvaja et al., 2007; Sawyer, 2006). Thereis evidence that active participation in knowledge co-construction activities can result in highquality learning outcomes (Wang, 2009). Previous research has suggested that knowledge
co-construction activities can have a positive impact on both group productivity and learning
outcomes (Weinberger et al., 2007).
Collaborative argumentation is a group-based exercise in which students work
together to construct an integrated group view on a discussion topic (Noroozi et al., 2012).
Learners benefit from collaborative argumentation because they obtain new ideas andinformation while comparing, assessing, and synthesizing different points of view based onevidence and rationale (Han et al., 2021). Furthermore, collaborative argumentation is helpful
in the development of competencies for arguing and collaborating with others, both of which
are required for working and living in the future society (Noroozi et al., 2012).
Typical collaborative learning processes commence with the group or end at the grouplevel. In fact, participating in individual ideation before collaboration can benefit bothindividuals and groups. By engaging in individual ideation before intra-group knowledge coconstruction, each group member's ability to recall prior knowledge is enhanced, fostering agreater number of ideas that can subsequently contribute to the overall intra-group synergy(Chen et al., 2021). Additionally, individual ideation promotes increased levels of participation,
discourages the presence of free-riders, and equalizes the level of collaboration among groupmembers (Isohätälä et al., 2017).
This study analyzes the process of collaborative argumentation of two groups (onehigh-performing group and one low-performing group) from individual ideation to groupknowledge co-construction from epistemic, argument, and social dimensions. To be specific,
the research questions are:
RQ1: What are the sequential patterns of epistemic knowledge co-construction
behaviors of the high- and low- performing groups in CSCA?
RQ2: How did the high- and low-performing groups’ argument structures develop?
RQ3: What are the social interaction networks of high and low-performing groups’
argumentative knowledge co-construction?

3. Methodology
3.1 Context and Participants

257

The study took place in a secondary school English class in Singapore. A total of 24 secondary
school (15-year-old) students (male =12, female =12) participated in this study. The teacher
randomly assigned participants to 6 groups (4 members in each group). The participants inthis study possess experience in collaborative learning but lack familiarity with argumentation.
The experiment occurred in a blended learning environment; the teacher and students
interacted offline in a physical classroom while they engaged in collaborative arguments on
an online platform using their personal learning devices (PLDs). The teacher's expertise incomputer-supported collaborative learning (CSCL) contributed to the experiment's
instructional design. The learning objective is applying English language skills in anargumentation context and improving students' argumentation skills and critical thinking, as
highlighted by 21st-century student outcomes. IRB approval has been obtained from theuniversity of the first author before data collection commenced.

3.2 Procedure
The participants were asked to collaboratively co-construct an argumentation diagram inAppletree, an online collaborative argumentation platform (see Figure 1.
http://www.appletree.sg/ ) (Chen et al., 2013; Tan & Chen, 2022).
The argumentation topic is “AI will make our lives at home more enjoyable. How far do
you agree?” selected from the English lesson syllabus. The whole lesson lasted for 1.5 hours
which included teachers’ instruction, and students’ collaborative argumentation. The students
went through a 5-phase Spiral Model of Collaborative Knowledge Improvement (SMCKI)(Chen, et al., 2021). Phases 3, 4, and 5 are inter-group feedback, intra-group refinement, andindividual achievement, which touch on students’ feedback literacy and exceed argumentativeknowledge. To better understand argumentative knowledge construction, this study focuseson the first two phases which lasted for 30 minutes in total: 1) Individual ideation (10 minutes):
students individually generate ideas and take a position for the augmentative topic; 2) Intragroup synergy (20 minutes). Group members co-constructed an argumentation diagram byintegrating all individual ideas through discussion and negotiation.

Figure 1. Appletree working space. Phase 1 individual ideation (left), blue bubble
showcases the idea generated separately by each group member; Phase 2 groupsynergy (right) shows the argumentation work co-constructed by all groupmembers. The dark blue bubble is the main claim; the green/red represents
for/against the relationship between the two bubbles.

3.3 Data Collection and Analysis Method
Two types of data were collected: the group's argumentation diagrams (artifacts) in AppleTree,and the computer screen recordings from each group member of the 2 groups. Multiple
analytical methods were used to examine the students’ knowledge co-construction. The study
used product analysis (PA) (Novak & Cañas, 2008) to assess the quality of all groups’
argumentation diagrams in terms of four dimensions, i.e., clarity, multiple perspectives,
selection of evidence, and depth of elaboration (Marzano et al., 2000; Stapleton & Wu, 2015;
Tan & Chen, 2022). The unit of analysis is the argumentation diagram of the group. Each258

dimension is rated on a scale ranging from 0 to 3. Two of the authors scored the argumentation
diagrams independently (Cohen’s K = 0.70). Two groups with the highest and lowest qualityof argumentation artefacts were selected in this study to compare the process of knowledge
co-construction between the high- and low-performance groups. The high-performancegroup’s score was 8 (clarity = 2, perspective = 2, evidence = 2, elaboration = 2), and the lowperformance group’s score was 5.5 (clarity = 1.5, perspective =1.5, evidence = 1.5. elaboration= 1).
First, the coding scheme for on-screen behaviors was adapted from Chuy et al.
(2011)’s explanation-seeking dialogue framework and the Claim-Evidence-Reasoning (CER)
model (Toulmin, 2003). We take one behavior as a unit, for example, student add one subclaim,
then we code it as ABC. The explanation-seeking dialogue framework includes five epistemic
behaviors: 1) formulating thought-proving questions; 2) theorizing; 3) obtaining evidence; 4)
working with evidence; and 5) synthesizing and comparing. This framework was most used in
the knowledge building context (e.g., Knowledge Forum online platform). CER is anargumentation pattern including three argumentation elements: "claim", "evidence", and"reasoning". An argument's main goal is to justify a claim with evidence and reasoning(Osborne & Patterson, 2011). The coding scheme is shown in Table 1. The coding process
was iterative. The second and third authors coded the screen recordings separately and held
multiple meetings to resolve discrepancies. The inter-rater reliability, as measured by kappa
coefficient, between the two coders was 0.761.
Table 1. Coding Scheme for Knowledge Co-construction in Collaboration Argumentation
Category
Contribute

Seeking
input
Revising

Off task

Sub-category
1. Add a claim or sub-claim
2. Add new facts/examples (evidence) to support a claim/sub-claim
3. Add new facts/examples (evidence) to oppose a claim/sub-claim
4. Provide elaboration to explain evidence/sub-claim
5. Seek help from discussing with other group members
6. Seek input (from internet/discussion)
7. Copy & paste
8. Rephrase/Revising previous work
9. Delete information
10. Change link/position
11. Synthesize available evidence/elaboration (bubble)
12. Discuss irrelevant things/play with the system

Code
ABC
ASE
AOE
ELA
DIS
INP
C&P
RE
DE
CH
SYN

OFF
TASK

4. Results
4.1 The Sequential Patterns of Argumentative Knowledge Co-construction
The results of on-screen behavioural coding consist of a totally 111 and 67 behaviors for thehigh- and low-performing groups, respectively. The frequency and percentage of each coded
behaviour were calculated for the two groups (see Table 2). In the high-performing group, themost frequent on-screen argumentative knowledge co-construction behaviour was seeking
input from the internet (INP), whereas and least frequent behaviour was off-tasking(OFFTASK). By contrast, the most frequent on-screen behaviour of the low-performing group
was discussing (DIS) while behaviours like synthesizing existing and new information (SYN)
and copying and pasting (C&P) were not identified.
Table 2. Frequencies and Percentages of On-screen Argumentative Behaviors between theHigh- and Low-performing Groups
High

INP

DIS

SYN

RE

C&P

ASE

ABC

OFF
TASK

DE

ELA

27

14

9

17

11

7

5

4

7

10

259

(111)(24.32%)(12.61%)(8.11%)(15.32%)(9.91%)(6.31%)(4.5%)(3.6%)(6.31%)(9.01%)

Low

8

35

0

1

0

2

4

10

1

6

(67)

(11.94%)(52.24%)(0%)(1.49%)(0%)(2.99%)(5.97%)(14.93%)(1.49%)(8.96%)Lag Sequential Analysis (LSA) (Bakeman & Gottman, 1997) was conducted based on
the coding results to understand the sequential pattern of students' knowledge co-construction
from the epistemic dimension. Lag sequential analysis (Bakeman & Gottman, 1997) examines
the occurrence and temporal order between different learning behaviors (Kapur, 2011). The
coded knowledge co-construction behaviour sequence was imported into the GSEQ 5.1. The
z-score of every transition from one behaviour to another was reported in the LSA results,
indicating the significance level of each behaviour sequence (z-score > 1.96, Bakeman &Gottman, 1997). A behavior transition diagram was created to visualize the significant
sequences in each group.
Figure 1 visualizes the on-screen behavioral transition patterns in terms of theirargumentative knowledge co-construction of the high and low-performing groups. There were
4 and 2 significant behavioral sequences identified in the high and low performing grouprespectively. Firstly, the predominant patterns of high performing group’s argumentation coconstruction were "seeking input from the internet or discussion (INP)" → "synthesizing
internet information with existing evidence or reasoning (SYN)" and "synthesizing internet
information with existing evidence or reasoning (SYN)" → "seeking input from the internet(INP)". The findings show that the high-performing group students searched for information
online and then integrated that information into their proposed ideas. They tended to seek
information from the internet after synthesizing ideas. Another significant transition pattern forthe high-performing group was "seeking input from the internet (INP)" → "adding new evidence
to support a claim or sub-claim (ASE)". The high-performing group raised evidence for claims
after searching for relevant information. On the contrary, the low-performing group directly
proposed claims and sub-claims after seeking information from the Internet without deeper
information processing according to the transition pattern "seeking input from the Internet(INP)" → "adding a claim or a sub-claim (ABC)".
In addition, high-performing group students continued to construct knowledge via
rephrasing or revising previous ideas after copying and pasting information from the Internet,
the team members' work, or from their own individual ideation as demonstrated in thesequence pattern "copy and paste (C&P)" → "rephrase or revise previous work (RE)". One
significant sequence of the low-performing group was "OFFTASK" → "discuss the topic (DIS)",
which infers that the students tended to discuss irrelevant things or play with the system before
discussing the topic.

Figure 1. Argumentative Knowledge Co-construction Behavior Patterns of High- (left)and Low-performing (right) Groups.

4.2 The High- and Low-performing Group's Arguments Development260

To answer the second research question about the development of the high- and lowperforming groups’ arguments compositions, the Sankey diagram was employed to unpack
the knowledge co-construction process to show the reasoning flow to reveal the differences
between the two groups. Two Sankey diagrams were derived to visualize the development ofstudents’ argument construction (see Figure 2) from the individual ideation phase to the groupsynergy phase. The left column of the diagrams is the students瀠 individual ideation whereas
the right column is the results of group argumentative knowledge co-construction. The middle
column illustrates how each idea and argument element developed. For example, theevidence (E) in Argument 4 (left column) became the evidence (E1) in Argument 4 (rightcolumn) through synthesizing the original evidence (E) with new ideas from the Internet, theteam members' work, or from their own individual ideation.
Figure 2 shows that the high-performing group students constructed eight argumentsfrom seven different perspectives, including only two sound and completed arguments with a"C-E-R" structure (Argument 4 and 5), three arguments consisting of a claim and one piece ofevidence (Argument 6, 7 and 8), and three arguments with one claim (Argument 1 and 2).
After the knowledge co-construction, there were three arguments with a "C-E-R" structure andtwo of the three arguments consisted of richer evidence and stronger reasoning (Argument 4
and 7). Additionally, two arguments were made up of a claim and a piece of evidence(Argument 1 and 2), and in Argument 8 there were a claim and two lines of reasoning. The
arguments focusing on society and technology appeared in individual ideation passe weredeleted after the group knowledge-co-construction.
The low-performing group started with three completed arguments (Argument 1, 3, and5) from perspective of society, art, and environment; two arguments with one claim (Argument
2 and 4); one argument with a claim and a line of reasoning (Argument 6). Three ideas posted
were not considered as arguments by the researchers as they did not correspond to any
certain claim. At the end of group co-construction, four fragmented ideas remained and there
were only two sound arguments with a "C-E-R" structure (Argument 1 and 3) which was lessthan their individual ideation work, though Argument 3 added one line of reasoning. Throughrephrasing and synergizing, some ideas were still not integrated into an argument well.
The results from the Sankey diagrams show that there were differences in the processof argument development between the high- and low-performing groups. From the behavior
point of view, the high-performing group contributed new argument elements via creating
reasoning and evidence (N=11) during the argumentative knowledge co-construction, but thelow-performing group deleted nine elements and created only one element (R2) and a
fragmented idea (idea 6). Second, from the perspective point of view, the high-performing
group considered all the perspectives they proposed during the individual phase whereas the
low-performing group paid most of their attention to the perspective of art only.

261

Figure 2. Argument Development of High- (above) and Low-performing (below)
Groups. Claim (C), Evidence (E), Reasoning (R).

4.3 The Social Interaction Network of the High- and Low-performing Group
Social network analysis (SNA) was used to analyze the social characteristics of groupinteractions through online collaboration. This study focused on the frequency of interaction,
the frequency of built-on work a participant makes to others. The degree centrality and Groupcentralization to show whether the network is equal or not (Yustiawan et al., 2015); In addition,
the density and connectedness of each group were analysed to show the cohesiveness andreachability between group members.
The results show that the high-performance group had higher interaction frequency(freq. = 5, M = 2.5, SD = 1.732) than the low-performance group (freq. = 4, M = 2, SD = 0.816).
The high-performance group also has a higher average degree (outdegree=1.25, indegree =1.25) than the low-performance group (outdegree = 1, indegree = 1). The high-performancegroup has lower out-centralization (out-centralization = 0.111, in-centralization = 1) than the
low-performing group (out-centralization = 0.444, in-centralization = 0.889). On the contrary,
the low-performance group has higher density and connectedness (density = 0.333,
connectedness = 0.333) than the high-performing group (density = 0.25, connectedness=0.25).

5. Conclusions, Discussion, and Limitations
This study explored the high- and low-performing groups’ argumentative knowledge coconstruction from epistemic, argument, and social perspectives. The findings from LSA,
Sankey diagram visualization, and SNA show that the high-performing group possesses ahigher level of engagement, cognitive elaboration, and equal relationship, and is morecompetent in integrating argumentation elements than the low-performing group.
This study yields two findings on the epistemic dimension of collaborativeargumentation. First, the high-performing group was more engaged in the learning task thanthe low-performing group, which echoed the findings from the literature that the more engaged
group members are in the ongoing and constant elaboration of collaborative tasks, the higher
the quality of learning they achieve (Barron, 2003). Second, the high-performing group tended
to process information through rephrasing and synergizing pieces of information from theinternet and existing ideas to form higher levels of reasoning.
On the contrary, the low-performing group tended to copy information directly from theinternet without deep information processing. A possible reason for this finding is that the lesssuccessful groups tend to possess a lower level of cognitive elaboration competence.
Cognitive elaboration, a crucial type of cognitive processing with regard to knowledge262

construction (McNamara et al., 1996), is the enrichment of learning material using additional
information taken from or inferred in combination with prior knowledge. Arguments with highquality in terms of argumentative knowledge come with in-depth cognitive elaboration(Stegmann et al., 2007). This finding is consistent with previous research studies whichsuggested that deep cognitive elaboration of the learning material is causally related toknowledge acquisition (e.g., Stein & Bransford, 1979). Therefore, future pedagogical designthat aims to improve students’ argumentation skills can pay more attention to students’cognitive processing and elaboration.
There are two findings derived from the analysis of the argument dimension. The
argument structure of the high-performing group became stronger and richer, but it was hardfor the low-performing group to integrate fragmented reasoning and evidence to support a
corresponding claim so that a complete argument could be formed. The findings are consistent
with Sampson and Clark's (2011) research results: high-performing triads constructedarguments that included a sufficient and accurate explanation that was well supported byappropriate evidence and reasoning, whereas the low-performing triads produced arguments
supported by inappropriate and inadequate justification. A possible reason is that the lowperforming group students have difficulties in linking evidence to claims (Osborne et al., 2004;
Berland & Reiser, 2009; McNeill, 2011). As Osborne et al. (2004) observed, constructing agood argument is not a simple task for school students, and they need sufficient support andclear guidance to build their sense of what an effective argument is. Walker et al. (2019) also
reported that students may need additional guidance when they intend to revise or change
their claims and when they use scientific principles in their justification. Researchers andteachers would need to provide additional scaffoldings to guide students to construct effective
arguments.
The results of the social interaction show that high-performing group members
contributed actively and equally to their group work, while the contribution of low-performing
group students was less equal with two students dominating the group work. Borge et al. (2012)
reported a similar case that the greatest difference between high and low-performing groups
was how dominance shifted by person across activities. For example, in high-performing
teams, while one person dominated synthesis-related activities, another person dominatedactivities involving information interpretation or decision-making. In contrast, a low-performing
team would have the same person dominating both activities. This finding can be supportedby the previous research studies which showed that an equal and distributed collaborative
relationship with minimal hierarchical control could encourage students to engage more in thecollaborative process (Zhang et al., 2009). It is necessary to encourage students to build
collective responsibility (e.g., shared roles of knowledge builders) and both self- and sharedregulation skills (Park et al., 2019).
This study has some limitations. This study is exploratory in nature with only one highand low-performing group analyzed. The time duration of the study was short due to theschool’s curriculum tie constraint. Therefore, the findings may not be generalizable to othergroups and contexts. Nevertheless, the study is innovative in its analytical methods andprovides interesting insights into the different knowledge co-construction processes between
the high- and low- low-performing groups. Future research studies will be conducted with moregroups of data included and longer duration and deeper engagement of students’ knowledge
co-construction are designed and implemented.

Acknowledgements
This study was funded by Singapore Ministry of Education (MOE) under the Education
Research Funding Programme (OER 17/19 CWL) and administered by National Institute of
Education (NIE), Nanyang Technological University (NTU), Singapore. The study wasapproved by the NTU IRB (IRB-2020-04-031). Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do not necessarilyreflect the views of the Singapore MOE and NIE.

263

References
Andriessen, J., Baker, M., & Suthers, D. (2003). Argumentation, computer support, and theeducational context of confronting cognitions. In J. Andriessen, M. Baker, & D. Suthers (Eds.),
Arguing to learn: Confronting cognitions in computer-supported collaborative learning
environments (pp. 1–25). Dordrecht: Kluwer Academic Publishers.
Arvaja, M., Salovaara, H., Häkkinen, P., & Järvelä, S. (2007). Combining individual and group level
perspectives for studying collaborative knowledge construction in context. Learning andInstruction, 17, 448–459. https://doi.org/10.1016/j.learninstruc.2007.04.003.
Asterhan, C. S., & Schwarz, B. B. (2016). Argumentation for learning: Well-trodden paths andunexplored territories. Educational Psychologist, 51(2), 164-187.
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis.
Cambridge university press.
Baker, M. J. (1999). Argumentation and constructive interaction. Foundations of argumentative text
processing, 5, 179-202.
Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12, 307–359.
https://doi.org/10.1207/S15327809JLS1203.
Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science
Education, 93(1), 26–55. https://doi.org/10.1002/sce.20286
Borge, M., Ganoe, C. H., Shih, S. I., & Carroll, J. M. (2012, February). Patterns of team processes
and breakdowns in information analysis tasks. In Proceedings of the ACM 2012 conference on
computer supported cooperative work (pp. 1105-1114).
Chen W., Looi, C.K., Wen Y., & Xie, W. (2013). AppleTree: An assessment-oriented framework forcollaboration and argumentation. In Kapur, M., Nathan, M., & Rummel, N. (Eds.) Proceedings ofthe 10th International Conference on Computer-Supported Collaborative Learning (pp. 6-9).
International Society on Learning Sciences.
Chen, W., Tan, J. S., & Pi, Z. (2021). The spiral model of collaborative knowledge improvement: an
exploratory study of a networked collaborative classroom. International Journal of ComputerSupported Collaborative Learning, 16(1), 7-35.
Chuy, M., Resendes, M., Tarchi, C., Chen, B., Scardamalia, M., & Bereiter, C. (2011). Ways ofContributing to an Explanation-Seeking Dialogue in Science and History. QWERTY:Interdisciplinary Journal of Technology and Culture, 6, 242–260.
Fischer, F., Bruhn, J., Gräsel, C., & Mandl, H. (2002). Fostering collaborative knowledge construction
with visualization tools. Learning and Instruction, 12, 213–232.
Fischer, F., Kollar, I., Mandl, H., & Haake, J. M. (Eds.). (2007). Scripting computer-supported
collaborative learning: Cognitive, computational and educational perspectives (Vol. 6). Springer
Science & Business Media
Gao, L., Li, X., Li, Y., & Hu, W. (2022). Capturing Temporal and Sequential Patterns of SocioEmotional Interaction in High-and Low-Performing Collaborative Argumentation Groups. The
Asia-Pacific Education Researcher, 1-15
Han, J., Kim, K. H., Rhee, W., & Cho, Y. H. (2021). Learning analytics dashboards for adaptive
support in face-to-face collaborative argumentation. Computers & Education, 163, 104041.
Isohätälä, J., Järvenoja, H., & Järvelä, S. (2017). Socially shared regulation of learning andparticipation in social interaction in collaborative learning. International Journal of Educational
Research, 81, 11–24. https://doi.org/10.1016/j.ijer.2016.10.006
Janssen, J., Erkens, G., Kirschner, P. A., & Kanselaar, G. (2009). Influence of group member
familiarity on online collaborative learning. Computers in Human Behavior, 25(1), 161-170.
Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes
in a computer-supported collaborative environment. International Journal of Computer-Supported
Collaborative Learning, 6(1), 39-56.
Lu, J., Chiu, M. M., & Law, N. W. (2011). Collaborative argumentation and justifications: A statistical
discourse analysis of online discussions. Computers in Human Behavior, 27(2), 946-955.
Marzano, R. J., Gaddy, B. B., & Dean, C. (2000). What Works in Classroom Instruction. Mid-Continent
Research for Education and Learning, 2550 South Parker Road, Suite 500, Aurora, CO 800141678. https://eric.ed.gov/?id=ED468434
McNamara, D. S., Kintsch, E., Songer, N. B., & Kintsch, W. (1996). Are good texts always better?
Interactions of text coherence, background knowledge, and levels of understanding in learningfrom text. Cognition and instruction, 14(1), 1-43.
McNeill, K. L. (2011). Elementary students’ views of explanation, argumentation, and evidence, andtheir abilities to construct arguments over the school year. Journal of Research in Science
Teaching, 48(7), 793–823. https://doi.org/10.1002/tea.20430

264

Noroozi, O., Weinberger, A., Biemans, H. J. A., Mulder, M., & Chizari, M. (2012). Argumentationbased computer supported collaborative learning (ABCSCL): A synthesis of 15 years of
research. Educational Research Review, 7(2), 79–106.
https://doi.org/10.1016/j.edurev.2011.11.006
Novak, J. D., & Cañas, A. J. (2008). The theory underlying concept maps and how to construct anduse them.
Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school
science. Journal of Research in Science Teaching, 41(10), 994–1020.
https://doi.org/10.1002/tea.20035
Osborne, J. F., & Patterson, A. (2011). Scientific argument and explanation: A necessary distinction?
Science Education, 95(4), 627–638. https://doi.org/10.1002/sce.20438
Ouyang, F., Tang, Z., Cheng, M., & Chen, Z. (2023). Using an integrated discourse analysis approachto analyze a group's collaborative argumentation. Thinking Skills and Creativity, 47, 101227.
Park, J.-H., Lee, I. H., & Cooc, N. (2019). The Role of School-Level Mechanisms: How Principal
Support, Professional Learning Communities, Collective Responsibility, and Group-Level
Teacher Expectations Affect Student Achievement. Educational Administration Quarterly, 55(5),742–780. https://doi.org/10.1177/0013161X18821355
Sampson, V., & Clark, D. B. (2011). A comparison of the collaborative scientific argumentation
practices of two high and two low performing groups. Research in Science Education, 41, 63-97.
Sawyer, R. (2006). In R. Sawyer (Ed.). The Cambridge handbook of the learning sciences (pp. 187–
204). New York: Cambridge University Press Analysing collaborative discourse.
Stapleton, P., & Wu, Y. (Amy). (2015). Assessing the quality of arguments in students’ persuasive
writing: A case study analyzing the relationship between surface structure and substance.
Journal of English for Academic Purposes, 17, 12–23. https://doi.org/10.1016/j.jeap.2014.11.006
Stegmann, K., Wecker, C., & Weinberger, A. (2007). Collaborative argumentation and cognitive
processing-An empirical study in a computer-supported collaborative learning environment.
Stein, B. S., & Bransford, J. D. (1979). Constraints on effective elaboration: Effects of precision andsubject generation. Journal of Verbal Learning and Verbal Behavior, 18(6), 769-777.
Tan, S. H. J., & Chen, W. (2022). Peer feedback to support collaborative knowledge improvement:
What kind of feedback feed-forward? Computers & Education, 187,
https://doi.org/10.1016/j.compedu.2022.104467
Toulmin, S. E. (2003). The uses of argument. Cambridge university press.
Valero Haro, A., Noroozi, O., Biemans, H., & Mulder, M. (2022). Argumentation Competence:
Students’ argumentation knowledge, behavior and attitude and their relationships with domainspecific knowledge acquisition. Journal of Constructivist Psychology, 35(1), 123-145.
Vuopala, E., Näykki, P., Isohätälä, J., & Järvelä, S. (2019). Knowledge co-construction activities andtask-related monitoring in scripted collaborative learning. Learning, culture and social
interaction, 21, 234-249.
Wang, Q. (2009). Design and evaluation of a collaborative learning environment. Computers &Education, 53, 1138–1146. https://doi.org/10.1016/j.compedu.2009. 05.023.
Walker, J. P., Van Duzor, A. G., & Lower, M. A. (2019). Facilitating Argumentation in the Laboratory:
The Challenges of Claim Change and Justification by Theory. Journal of Chemical Education,
96(3), 435–444. https://doi.org/10.1021/acs.jchemed.8b00745
Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction
in computer-supported collaborative learning. Computers & education, 46(1), 71-95.
Weinberger, A., Stegmann, K., & Fischer, F. (2007). Knowledge convergence in collaborative
learning: Concepts and assessment. Learning & Instruction, 17, 416–426.
https://doi.org/10.1016/j.learninstruc.2007.03.007
Yustiawan, Y., Maharani, W., & Gozali, A. A. (2015). Degree Centrality for Social Network with Opsahl
Method. Procedia Computer Science, 59, 419–426. https://doi.org/10.1016/j.procs.2015.07.559
Zhang, J., Scardamalia, M., Reeve, R., & Messina, R. (2009). Designs for Collective Cognitive
Responsibility in Knowledge-Building Communities. Journal of the Learning Sciences, 18(1), 7–
44. https://doi.org/10.1080/10508400802581676

265

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Discussion Support Agent System to Promote
Equalization of Speech among Participants
Ryunosuke NISHIMURAa*, Risa IHARADAa, Yuya SUGAMOTOb, Yutaka ISHIIb,
Toshio MOCHIZUKIc & Hironori EGIaa
Department of Informatics, The University of Electro-Communications, Japanb
Department of Systems Engineering, Okayama Prefectural University, Japanc
School of Network and Information, Senshu University, Japan* r.nishimura@uec.ac.jp
Abstract: Reducing the imbalance of speech length in discussions is expected to
improve decision making and collaborative learning. In this paper, we propose afacilitation system based on the cumulative percentage of utterances in discussions.
We surveyed the effectiveness of this system on the collaborative learning from theperspective of standard deviation of utterance length and percentage of utterances
throughout discussions. we collected 20-minutes discussion data under the topic “Whatis a thoughtful consideration?” and “What is happiness?” from seven groups of threeparticipants. Each group had two discussions, one with the system and one without thesystem. The results revealed that the facilitation system decreased subjective andobjective imbalance of speech length. Additionally, turn-taking was active in the groupwhich facilitation by the system was effective.
Keywords: Discussion support agent system, dialog analysis, facilitation, collaborative
learning, percentage of utterances

1. Introduction
Recently, small group discussions are used in collaborative learning to avoid social loafing(Latané et al., 1979). Especially, researchers have focused on supporting collaborative
learning with technology (DeSanctis & Gallupe, 1987; Alavi, 1994; Jermann tr al., 2001). In
collaborative learning, persons are required active participation because persons discussdifferent ideas and consider these ideas carefully.
However, the imbalance of speech length may occur because of status of the
discussion such as the relationship and communication skills of participants. In addition,
decision making in discussions are affected by talkativeness (DiMicco et al., 2007). Thissuggests that the approach to support collaborative learning and increase the productivity ofthe discussion by equalizing utterance length. Ishikawa et al (2019) shows that participantsin discussion can learn co-regulation strategy using agent facilitation based on thepercentage of utterances up to 30 seconds before facilitation. On the other hand, this systemdoesn’t have enough effects to equalize the percentage of utterances. If a person speaks
much in 30 seconds before the facilitation, that person speak much throughout thediscussion.
In this study, we proposed a system for facilitation using the cumulative percentage
of utterances in discussions. To consider percentage of utterances throughout discussions,
our system can facilitate discussions with less influence from the imbalance of utterances ina short time interval. From the above, we focus on two research questions (RQ):
RQ1: Can the facilitation method using the cumulative percentage of utterances throughout
the discussion equalize the percentage of utterances?
RQ2: If the method is effective (or not effective), how the system reduced (or increased)utterance imbalance?

266

2. Related works
Commonly methods to reduce the number of utterances is that to visualize the imbalance ofparticipation. Bachour et al (2010) proposed the system for avoiding over andunderparticipation to indicate how much the individual has spoken. In this study, Matrix ofLEDs indicate person’s participation levels based on the number of speaks. Similarly, Kim etal (2008) visualized the participation as a figure consisting of a circle representing theimbalance of the utterances and some lines representing the amount of each participant's
speech. In addition, Bergsrtom & Karahalios (2007) examined the expression of theConversation Clock which visualize prosodic features such as a period of silence andvolume of speech. There are also ways to express discussion structure using metaphorssuch as trees and plants (Tausch et al., 2014; Praharaj et al., 2019).
In these methods, all participants can check the discussion status such as the
speech imbalance displayed on the table. However, this may be uncomfortable for the
reluctant speaker because everyone knows that he or she is not contributing to thediscussion. Therefore, we believe that it is suitable to encourage the participation of the
speakers by facilitating them while the agents maintain the audio features. One suchapproach is using robots (Miyake et al., 2012). However, it may be difficult to prepare robots
for actual discussions because they are expensive. In anticipation of such application toactual discussions, the facilitation agent system proposed by Ishikawa et al (2019) wasused. The facilitation system can be used in a variety of settings as long as there aredisplays and polarizers to project the agents, microphones, and computers to run thesystem.

3. A Facilitation system based on the percentage of utterances throughout
discussions
3.1 Overview of the facilitation system
Figure 1 shows the system configuration. Our system consists of two parts: discussion
analysis part and agent control part. The discussion analysis part calculates the speech
imbalance based on the speech collected by the wearable speech collection device andsends a request to the agent control part according to the degree of the imbalance. The
agent control part operates the holographic agent using request sent by the discussion
analysis part.
single board computerᶅ Correcting audio frommicrophoneᶆ Sending data

Discussion analysis part

Server PC

Send request

ᶇ Calculating the cumulative percentage ofutterancesᶈ Deciding facilitation method

ᶉ Control agent
based on request

Discussion with the agent

Agent control part267

Figure 1. The overview of the facilitation system

3.2 Calculating the cumulative percentage of utterances
The cumulative percentage of utterances is the rate of each participant's speech when the
total time of all participants' speech from the start of discussion to the time of the calculationis set to 100%. Silent periods are not included in the calculation of the percentage. For
instance, if three participants speak for 2, 3, and 5 minutes respectively in a 10-minutediscussion, the cumulative percentage of utterances is 20%, 30%, and 50%, respectively.

3.3 Criteria for agent facilitation
The holographic agent facilitates discussions when the system detects that a speech
imbalance is occurring in a discussion. In general, when the number of participants in a
discussion is n, if the cumulative percentage of utterances in all participants is 100/n%, it canbe considered that there is no imbalance in the speech of each participant. However, it isunlikely that the cumulative speech rate of all the discussion participants coincides.
Therefore, we set 100/n% ± (100/n - 100/(n+1))% as the acceptable range, and the situationin which there is a participant who is out of this acceptable range as the biased state. For
example, for a three-person discussion, the acceptable range would be 25% to 41%.
The agent suppresses the speech of the participants whose cumulative percentage ofutterances is higher than the acceptable range, and encourages the speech of theparticipants whose cumulative percentage of utterances is lower than the acceptable range.
When the cumulative percentage of utterances of all participants is within the acceptable
range, the agent does not approach them, but listens or observes them. Therefore, thefollowing three facilitations are performed by the agent system. Decisions on agent behavior
based on the cumulative percentage of utterances were made every minute.
z

If there is one or more discussion participants whose cumulative percentage ofutterances is lower than the acceptable range and there is no discussion participant
whose cumulative percentage of utterances is higher than the acceptable range.

In this case, there is a imbalance in speech because there is a participant who does not
speak much. Therefore, the agent asks the participant to "Why don’t you express some
opinions?" or "Why don’t you explore the opinions so far?" with voice.
z

If there is one or more discussion participants whose cumulative percentage ofutterances is higher than the acceptable range and there is no discussion participant
whose cumulative percentage of utterances is lower than the acceptable range.

In this case, there is a imbalance in speech because there is a participant who speaks much.
Therefore, the agent asks the participant to "Why don't you ask others?" or "Why don't you
explore other people's opinions?” with voice.
z

If there is one or more discussion participants whose cumulative percentage ofutterances is higher than the acceptable range and lower than the acceptable range.

In this case, there is a imbalance in speech because there are both a participant who speaks
much and speaks little. In this case, the above two cases are alternated.

4. Experiment
An experiment was conducted to evaluate the effectiveness of the facilitation system. Eachparticipant had two discussions, one is the experimental discussion using the system, andanother is the control discussion without the system. The order of the two discussions was268

random. Two discussion topics were set as "What is thoughtful consideration?" and “What is
happiness?”. Discussion topics are based on a course of study in Japan (MEXT, 2020).
These discussions were attended by three graduate and undergraduate students of ascience and engineering university. Each discussion had three participants and lasted for 20
minutes.
After discussion, we conducted a questionnaire to evaluate the system. In the systemevaluation questionnaire, participants were asked to describe how they felt the systemfacilitation. Figure 2 shows the discussion with the system.

Figure 2. A discussion with the agent

5. Results
5.1 RQ1: Can the facilitation method using the cumulative percentage of utterances
throughout the discussion equalize the percentage of utterances?
We examined the effects of the facilitation system on equalization of speech from theviewpoint of the standard deviation of the cumulative percentage of utterances. This
standard deviation represents the degree of speech imbalance among the participants. Thatis, if the standard deviation in the experimental discussion is smaller than the standard
deviation in the control discussion, the system should promote equalization of speech
among participants. Table 1 shows the standard deviation of the cumulative percentage ofutterances in seven groups.
Table 1. Standard deviation of the cumulative percentage of utterances in control discussion(without the system) and experimental discussion (with the system)
Standard deviation of the cumulative percentage of utterancesGroup No.
Control discussion
Experimental discussion
1
15.44
2.23
2
13.22
4.63
3
13.52
6.36
4
17.58
13.40
5
25.45
15.43
6
23.85
18.06
7
20.66
19.56

5.2 RQ2: If the method is effective (or not effective), how the system reduced (orincreased) utterance imbalance?
From Table 1, the method should be effective to promote equalization speech among
participants. However, the degree of mitigation of the speech imbalance by the facilitation
system was different. Therefore, we divided the seven groups into two groups: a large effect269

group (group 1-3) and a small effect group (group 4-7). Additionally, we examined theprocess of equalization in the large and small effect groups. Figure 3 shows the percentage
of utterances per minute in the experimental discussion of Group 3 and Figure 4 shows thatof Group 7. Group 3 had the highest number of the facilitation in the large effect group andGroup 7 had the largest the standard deviation in the small effect group.
The percentage of utterances is recalculated every minute in Figure 3 and Figure 4. For
example, Figure 3 shows that participant 1 spoke longer than other participants in theinterval from 60 to 120 seconds.
Cumulative percentage
of utterances (%)

Participant 1
Participant 2
Participant 3 ●●ʁWhy don't you ask others?

Discussion time (seconds)○ʁWhy don't you explore otherpeople's opinions?
☆ʁWhy don’t you explore theopinions so far?
★ʁWhy don’t you express
some opinions?

Figure 3. The cumulative percentage of utterances per minute in the experimental
discussion of Group 3 in large effect group
Cumulative percentage
of utterances (%)

Participant 1
Participant 2
Participant 3●ʁWhy don't you ask others?
○ʁWhy don't you explore otherpeople's opinions?
☆ʁWhy don’t you explore theopinions so far?
★ʁWhy don’t you express
some opinions?

Discussion time (seconds)

Figure 4. The cumulative percentage of utterances per minute in the experimental
discussion of Group 7 in small effect group

6. Discussion
Table 1 shows standard deviation of the cumulative percentage of utterances decreased for
all groups to a certain degree. Therefore, RQ1 is partially shown. In addition, Figure 3 and 4
suggest that turn-taking differs between the large and small effect groups. For example,
each cumulative percentage of utterances per minute in Group 3 is equalized and cross
many times after the instructions in both the first and second halves of the discussion. Thismeans speaking turn rotated fast and equally in Group 3. On the other hand, eachcumulative percentage of utterances per minute in Group 7 is not equalized and cross few
times despite being facilitated by the agent system. Similar results were observed in thequestionnaire. Participants in Group 3 answered that they were aware of speech imbalance
due to the facilitation, while participants in Group 7 answered that they ignored it due to thedifficulty of following the facilitation. Therefore, awareness of turn-taking is the key toanswering RQ2: in order to equalize speech, it is necessary for participants to be aware ofspeech imbalance through facilitation and to correct it actively by turn-taking. These results
imply the effectiveness of facilitation may be evaluated by observing the turn-taking after thefacilitation.

7. Conclusion
In this paper, we proposed the facilitation system based on the cumulative percentage ofutterances in discussions. Here, the experiment yields two important results: this system iscontributed to promote equalization of speech among participants and turn-taking is a key toequalize cumulative percentage of utterances. Our study gives a quantitative perspective of
the facilitation.

270

However, this study has some limitations. In this study, we mainly analyzed large
effect group and small effect group from the graph of cumulative percentage of utterances.
We need to further explore factors that may make the difference in the effectiveness of the
facilitation system such as personality characteristic.

Acknowledgements
This work has been partly supported by the Grants-in-Aid for Scientific Research (NOs.
21K02752, 21K18527 and 22K02951) by MEXT (Ministry of Education, Culture, Sports,
Science and Technology) in Japan

References
Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS quarterly,
159-174.
Bachour, K., Kaplan, F., & Dillenbourg, P. (2010). An interactive table for supporting participation
balance in face-to-face collaborative learning. IEEE Transactions on Learning
Technologies, 3(3), 203-213.
Bergstrom, T., & Karahalios, K. (2007, January). Conversation Clock: Visualizing audio patterns in colocated groups. In 2007 40th Annual Hawaii International Conference on System Sciences(HICSS'07) (pp. 78-78). IEEE.
DeSanctis, G., & Gallupe, R. B. (1987). A foundation for the study of group decision support
systems. Management science, 33(5), 589-609.
DiMicco, J. M., Hollenbach, K. J., Pandolfo, A., & Bender, W. (2007). The impact of increased
awareness while face-to-face. Human–Computer Interaction, 22(1-2), 47-96.
Ishikawa, N., Okazawa, T., & Egi, H. (2019). DiAna-AD: dialog analysis for adjusting duration duringface-to-face collaborative discussion. In Collaboration Technologies and Social Computing: 25th
International Conference, CRIWG+ CollabTech 2019, Kyoto, Japan, September 4–6, 2019,
Proceedings 25 (pp. 212-221). Springer International Publishing.
Jermann, P., Soller, A., & Muehlenbrock, M. (2001). From mirroring to guiding: A review of the state ofart technology for supporting collaborative learning. In European Perspectives on ComputerSupported Collaborative Learning (pp. 324-331).
Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008, November). Meeting mediator: enhancing
group collaboration using sociometric feedback. In Proceedings of the 2008 ACM conference on
Computer supported cooperative work (pp. 457-466).
Latané, B., Williams, K., & Harkins, S. (1979). Many hands make light the work: The causes andconsequences of social loafing. Journal of personality and social psychology, 37(6), 822.
MEXT (Ministry of Education, Culture, Sports, Science and Technology). (2020, October). English
Translation of the Revised a Course of Study at Junior High Schools Revised in 29th year of the
Heisei period (provisional translation). Retrieved from https://www.mext.go.jp/content/20201008mxt_kyoiku02-000005242_1.pdf.
Miyake, N., Okita, S., & Rose, C. (2012). Robot facilitation as dynamic support for collaborative
learning.
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2019). Group Coach for Co-locatedCollaboration. In Transforming Learning with Meaningful Technologies: 14th European
Conference on Technology Enhanced Learning, EC-TEL 2019, Delft, The Netherlands,
September 16–19, 2019, Proceedings 14 (pp. 732-736). Springer International Publishing.
Tausch, S., Hausen, D., Kosan, I., Raltchev, A., & Hussmann, H. (2014, October). Groupgarden:
supporting brainstorming through a metaphorical group mirror on table or wall. In Proceedings of
the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (pp. 541550).

271

" !   !" " "   #" !
#" !"$  #" ! #"   
                                                           
7;A75  40-7-:1 C-?10- 6<  '1.9=:98;.8/<12;;<=-A2;<8,87;<:=,<*,869*7287:8+8<+.1*>28:68-.5<1*< 9:8>2-.; * 5.*:7270 .7>2:876.7< .71*7,270 5.*:7.:; 68<2>*<287 <8 5.*:7 +A
27-=,2709..:*?*:.7.;; $*:<2,=5*:5A27-=,2709..:7.;;<1:8=01:8+8<+.1*>28:1*;
*7 269*,< 87 .-=,*<287  '1.:./8:. ?. >.:2/2.- * :8+8< +.1*>28: 68-.5 8:2.7<.- <827-=,. 9..:7.;; 27,5=-270 27-=,270 * ;.7;. 8/ ,869.<2<287 <1:8=01 5870<.:6
787>.:+*527<.:*,<287; '1.:.;=5<;8/<12;;<=-A,87/2:6.-<1.$..:7.;;27-=,<287*7--.687;<:*<.-<1.>*52-2<A*5<18=012<>*:2.--.9.7-27087<1.6.7<*5;<*<.  $..:5.*:72705.*:7270,869*7287:8+8<787>.:+*5,866=72,*<287  
+5@4@4101B18;<91:@;25:2;>9-@5;:@1/4:;8;3E@41?/4;;8/;:/1<@C-?8-A:/4105:-<-: ;:?1=A1:@8E-2;>9;2@1-/45:3-:081->:5:37:;C:-?181->:5:34-?.1/;91
9;>1/;99;: :-005@5;:->-<505:/>1-?15:@41019-:02;>181->:5:3C-?;.?1>B10
0A>5:3@41%* <-:0195/ ;C1B1>@412;>9;281->:5:3?A<<;>@5:181->:5:35?
0;95:-@10.E?A<<;>@2>;9-81->:1>C4;5?9;>17:;C81031-.81-.;A@@4181->:5:30;9-5:
@4-:@4181->:1>C4;5?.1@@1>?A5@10@;@>-:?21>>5:37:;C81031-:02-/585@-@5:3A:01>?@-:05:3 ;C1B1>@41>11D5?@??53:525/-:@81->:5:3<11>81->:5:35:C45/47:;C810315?
/;:?;850-@10-:0/;:25>910@4>;A34/;88-.;>-@5;:C5@4/8-??9-@1? )45?81->:5:3
>1<>1?1:@10.E-/@5B181->:5:35?28;A>5?45:35:?/4;;81:B5>;:91:@? )41>12;>1-8@4;A34
&11>"1->:5:35?:;@/;:0A/@105:;:?5@1181->:5:31:B5>;:91:@?5@5?8595@10@;/>1-@5:3-?5:381<1>?;:81->:5:31:B5>;:91:@ ;C1B1>&11>"1->:5:34-?-<;?5@5B159<-/@:;@;:8E
;:81->:5:31221/@5B1:1??.A@-8?;;:81->:1><?E/4;8;3E#->01>1@-8  )41>12;>1
C1-59@;/;:?@>A/@-:181->:5:31:B5>;:91:@@4-@.>5:3?-.;A@-<;?5@5B1<?E/4;8;35/-8
1221/@;2&11>"1->:5:3.EA?5:3-81->:5:3?A<<;>@>;.;@-:05:B1?@53-@1@41<;??5.585@E;2
5:0A/5:3-?1:?1;2/-9->-01>51<11>:1??@4>;A34@41>;.;@  ##   #  &""!#%!!%$'1?1->/44-?.11:/;:0A/@10;:81->:5:3?A<<;>@>;.;@? !-?454->-?4;C10@4-@
A:8571;@41>9105-81->:5:3?A<<;>@>;.;@?C45/44-B1<4E?5/-85@E/-:-/451B19;>1
:-@A>-8-:0-A@41:@5//;99A:5/-@5;:C5@481->:1>? 1:/15@5?1D<1/@10@;>10A/1:13-@5B1
19;@5;:??A/4-?81->:5:35:45.5@5;: :-?@A0EA?5:3-/;9<A@1>-?-/;9<-:5;:5:-
81->:5:31:B5>;:91:@!-?-51@-8  01B18;<10-9A8@5-31:@81->:5:31:B5>;:91:@5:-/;;<1>-@5B181->:5:31:B5>;:91:@@4-@?A<<;>@?81->:5:3.EA?5:3@41/;9<-:5;:-?15@41>-
<11>;>@1-/41>01<1:05:3;:@4181->:1>??5@A-@5;: 9A8@5-31:@81->:5:31:B5>;:91:@
C-?01B18;<105:-/;;<1>-@5B181->:5:31:B5>;:91:@ 4-:  <;5:@10;A@@4-@-

272

/;9<-:5;:>;.;@@4-@.14-B1?-?-@1-/41>9-E35B1@4159<>1??5;:;2.15:3C-@/410 H
;C1B1>#->01>1@-8 
?A331?@10@4-@@41<>1?1:/1;2-<11>/-:<>;0A/1<;?5@5B1
1221/@? :31:1>-8C4581<>1B5;A??@A051?;:<11>81->:5:3-:081->:5:3?A<<;>@.E>;.;@?
4-B1?A331?@10@415>A?12A8:1??@41E4-B1:;@/;:?501>10@4121185:3;2.15:39;:5@;>10
@4-@-@1-/41>8571<;?5@5;:.>5:3? )41>12;>1@45??@A0E-59?@;/;:?@>A/@->;.;@.14-B5;>
9;018@4-@5:0A/1?<11>:1??;>51:@10@;C->0@41/;:?@>A/@5;:;2>;.;@?@4-@1D45.5@;:8E-<<>;<>5-@1<11><;?5@5;:?5:>1?<;:?1@;B->5;A??5@A-@5;:?   %# $!%'%!  ! '# !#%! $A91>;A??@A051?4-B11D-95:10@41<;?5@5B11221/@?;25:@>5:?5/9;@5B-@5;:;:81->:5:3 '13->05:3-A@;:;9E5:0A/10.14-B5;>-801/5?5;:?>18-@10@;5:@>5:?5/9;@5B-@5;:,;7;E-9-
1@-8 ?@-@10@4-@;@41>?5:@1:@5;:1?@59-@5;:?5:28A1:/1;:1?;C:.14-B5;>-8
01/5?5;:? %?-C-
2;A:0@4-@@41.14-B5;>;2@41@->31@5:0A/1?5:@1:@5;:1?@59-@5;:
-:@4>;<;9;><45?95?A?10@;?A<1>59<;?1@41859.9;B191:@?;2@41>;.;@.14-B5;>;:@41
9;B191:@?;24A9-:859.?-:05:@1:@5;:1?@59-@5;:2>;9@41@->31@?.14-B5;>5?<1>2;>910
.E-:@4>;<;9;><45F5:3@41@->31@ %A>?@A0E-59?@;.;;?@81->:1>-A@;:;9E@4>;A34
5:@1:@5;:-85:21>1:/12>;9->;.;@?.14-B5;>-:02;?@1>5:@>5:?5/9;@5B-@5;:A?5:3:;:B1>.-8
-:@4>;<;9;><45//A1? )41>;.;@?.14-B5;>9;0181D/8A01?85:3A5?@5/5:2;>9-@5;:-:0>1851?
?;818E;::;:B1>.-8/A1?5:/8A05:3:;:?1:?1A@@1>-:/1?  !#%%#$%&)! &%! !# $$)#!!%!%! +14-B11D<1>591:@-88E/8->52510@4191:@-82-/@;>?@4-@/;:?@5@A@1@41&11>:1??;281->:1>?
0A>5:381->:5:3-:0/>1-@10-<?E/4;8;35/-89;018C5@4-451>->/45/-8?@>A/@A>153A>1
!;75 )4191-:5:3;21-/42-/@;>5?85?@105:)-.81 )4145341>@411B-8A-@5;:
B-8A1;21-/4@4145341>@4159<>1??5;:;2@41<?E/4;8;35/-82-/@;>?@4-@1D5?@-@@41@;<-:0
@419;>1&11>:1??5?218@5:@411:0 
53A>1  &?E/4;8;35/-89;018;221185:3<11>
)-.81 -/418191:@;2@41<?E/4;8;35/-89;018-:05@?91-:5:3
#!" ""#""#$##"($"##""#&#($"!$"####"#!$"#&!#("#(#"""" $%
($! $#("!(##(#&($"$!#( "##"##$$#.$
"&!"""&#!$##+"!
($$!"##!+" "###""$!!#"
:-:-@@19<@@;5:0A/1&11>:1??.E>;.;@-/@5;:?-//;>05:3@;@41/>1-@10<?E/4;8;35/-8
9;018C15:B1?@53-@10@41>18-@5;:?45<.1@C11:>;.;@-/@5;:?-:0@41<?E/4;8;35/-89;018-:0.-?10;:@41;.@-5:10>1?A8@?C1/;:0A/@10-?4;>@@1>9?@A0E;:5:0A/5:3&11>
:1?? H;C1B1>:;?@>;:3>1?A8@?C1>1;.@-5:102;>5:0A/5:3&11>:1??.-?10;:@41273

?1:?1;2/;9<1@5@5;:-:0?1:?1;2/;9<1@5@5;:C45/4->1?1B1>18E31:1>-@10
5:?@-:@-:1;A?8E?A331?@5:3@41:1102;>-8;:3@1>9?@A0E 




 


)45?8;:3@1>9?@A0E-59?@;5:0A/1-?1:?1;2/;9<1@5@5;:-:0<11>:1??.A5805:3;:@41
25:05:3?;2?4;>@@1>9?@A051? %C5:3@;0-@-8595@-@5;:?@41<->@5/5<-:@?5:/8A010;:19-81
6A:5;>4534?/4;;8?@A01:@C5@4.-?5/81->:5:3-.585@51?-:02;A>A:5B1>?5@E?@A01:@?@C;91:
-:0@C;C;91:/;99A@5:3@;);7E; )45??@A0E2;/A?1?;:@41>1?A8@?;2-:1D<1>591:@
C5@46A:5;>4534?/4;;8?@A01:@ )41>;.;@.14-B5;>-:0<?E/4;8;35/-89;018A?10C1>1
.-?10;:<>1B5;A?>1?1->/4!;75
C45/4810A?@;/;:0A/@-8;:3@1>95:@1>-/@5;:
59<>1??5;:1B-8A-@5;:1D<1>591:@C5@46A:5;>4534?/4;;8?@A01:@?C5@405B1>?1
01B18;<91:@-8-.585@51?5:);7E; '121>>5:3@;!-:0-1@-8 
C1/>1-@101<5?;01?
?4;C5:3>;.;@.14-B5;>/;>>1?<;:05:3@;?5@A-@5;:?@4-@C1>1-??A910@;5:0A/1-?1:?1;2
/;9<1@5@5;:.-?10;:@410-@-;.@-5:105:<>1B5;A?1D<1>591:@? 005@5;:-88E@41>;.;@C-?5:?@>A/@10@;<1>2;>99-@419-@5/-881->:5:35:/;:6A:/@5;:C5@45@ )411D<1>591:@C-?
/;:0A/@10@4>11@591?-C1172;>@C;C117?2;>;:14;A><1>0-E>121>>5:3@;#->01>1@-8 
-:0!-:0-1@-8 
C4;?@A05108;:3@1>95:@1>-/@5;:? 53A>1 588A?@>-@1?@41
1D<1>591:@-828;C-:0)-.81 <>1?1:@?@4101@-58? 
53A>1 D<1>591:@-828;C)-.81 )4101@-58?;2@411D<1>591:@#
#!$##!$$##"'!#
!"# "!
"!$#!#!/)"')#%##"#$()
!""!#"1720"##$(!"""""#$(""")#!#!!"/$"""##0
#!#"##"#"6%"/&..0(#!"$#
!"# "!
!"#"""*&%#"#"""*%#"#"#"""*%#"#.# $"#! "#!#(#!#'!##""""!"""3 "##33 !.""3)
&""#!)!""", "##-, !.""-*%$#("#!"&##!%$""#$( "!
 $"#!"$!%("#!#!!")"##!##!!"""3##%""33!3"&"#!"$#
!"5#54
:-005@5;:@;@411D<1>591:@-828;C5:@1>B51C?C1>1/;:0A/@10-@@411:0;21-/40-E@;
3-@41>@41<->@5/5<-:@?59<>1??5;:?;2@41>;.;@-:0@415>81->:5:3 )41?@A0E2;/A?10;:
5:05B50A-881->:5:35:-:181->:5:31:B5>;:91:@C5@4@41?A.61/@?-:01D<1>591:@1>5:

274

?1<->-@1?<-/1?05B5010.E-:59<A8?1?/>11: );9-5:@-5:-?1:?1;2?;85@A010A>5:3@41
?@A0E81->:1>?>1/15B105:?@>A/@5;:?2>;9-@45>0<1>?;:-:0C;>710-8;:1-:?C1>5:3
1D1>/5?1?;:8;;?181-2<-<1>C4581.15:39;:5@;>10A?5:3-C1->-.81/-91>- )41-/@5;:?
;2@41>;.;@C1>1<>1?1:@10A?5:3@41+;F91@4;0.-?10;:@411<5?;01?/>1-@10 

53A>1 (/419-@5/05-3>-9;2@411D<1>591:@-:01:B5>;:91:@








 %! $"%( *# $$+ * $! !"%%! +
)413>-<4?588A?@>-@1@41@>-:?5@5;:;2@4159<>1??5;:?;2<11>:1??-:0/;9<1@5@5B1:1??
>1?<1/@5B18E )41B1>@5/-8-D5?5:05/-@1?@41?/;>1;.@-5:10A?5:3@41*(91@4;0/;:B1>@10@;-<1>21/@?/;>1;2-:0@414;>5F;:@-8-D5?5:05/-@1?@41:A9.1>;21D<1>591:@?
<1>2;>910 )411D<1>591:@-80-E?->15:05/-@10.E>1085:1?5:253A>1 53A>19<>1??5;:1B-8A-@5;:>1?A8@?2;>&11>:1??"-:0(1:?1;2/;9<1@5@5;:'
53A>15:05/-@1?@4-@@41G&11>:1??H59<>1??5;:?/;>1>19-5:104534@4>;A34;A@-88
?1??5;:?-:0@411221/@;:G?1:?1;2/;9<1@5@5;:HC-?A:/81-> ;C1B1>G&11>:1??H?/;>1
C-?:;@-.8E8;C1>-@@41@4-:0@4@591?/;5:/505:3C5@4<->@5/5<-:@?1D<>1??5:3/;:2A?5;:
-.;A@>;.;@-/@5;:?0A>5:35:@1>B51C? 2@1>@411D<1>591:@<->@5/5<-:@?91:@5;:1021185:3
9;>1?5958->@;@415><11>??A331?@5:3@41<;@1:@5-85:0A/@5;:;2G<11>:1??H0A>5:3@41
?@A0E >-<4?5:05/-@15:/>1-?5:359<>1??5;:?;2G<11>:1??H-:0G/;9<1@5@5B1:1??HC5@4
1-/41D<1>591:@-80-E )45??A331?@?-<;?5@5B1/;>>18-@5;:.1@C11:G/;9<1@5@5B1:1??H-:0
G&11>:1??H5:/>1-?1;B1>@591@4;A34@41/;::1/@5;:@4>;A34;A@@410-EC-?:;@/;:?5?@1:@8E?@>;:3  !%'%! %!# %%$

)45??1/@5;:01?/>5.1?@41/4-:31?5:@41<->@5/5<-:@?IC5885:3:1??@;81->:9-@419-@5/?
;.?1>B10@4>;A34;A@@411D<1>591:@ :@41<>1?A>B1E<->@5/5<-:@?C1>1-?710@;>-@1@415>
C5885:3:1??@;81->:9-@419-@5/?;:- <;5:@?/-81 )415:5@5-8?/;>1C-?1534@;:- <;5:@?/-81 ;C1B1>5:@418-?@0-EI?5:@1>B51C@41?@A01:@?>1<;>@10@4-@@41E@1:010@;
35B1A<C5@4;A@?;8B5:3C>5@@1:-:?C1>?5:@411->8E0-E? 1:/1-?-:5:5@5-8?5@A-@5;:
C5885:3:1??@;81->:9-@419-@5/?01<1:0?;:@41<>;.819.15:3-00>1??10 )412;88;C5:3
01?/>5.1?@41/;:@1:@;2@41=A1?@5;::-5>12110.-/7?1/@5;:2>;9@418-?@?1??5;:;2@41
1D<1>591:@ '!!.,#))$$")%'$!"(#$,
),(*#)$($!+)%'$!"(($(!. ,(%%.)))'$$),( #)$" 
53A>19<>1??5;:?-2@1>@41 @4?1??5;:

275


)45??A331?@?@4-@@41?@A01:@?C1>19;>19;@5B-@10@;81->:9-@419-@5/?@4>;A34;A@@41
1D<1>591:@ :-005@5;:@;/;:25>9@415>>1281/@5;:?@4>;A34;A@@411D<1>591:@C12;/A?10
;:@41/;:@1:@?;2@412110.-/7/;8A9:5:@4125:-8?A>B1E )41-/@A-8/;:@1:@5?<>1?1:@10
.18;C 
*()$)# ))'$$)(,'($")#)),$*!$#.()'$.*"#(*)$#"#
"#' *)#$,!))'$$)(#)!$#,!!,)*"#( +'$,#)))$'$$)( $!+#$"%*))$#!%'$!"(($""$'*#)#$' 53A>1;:@1:@?;2@415:-8'1?1->/4;991:@?(1/@5;:)41?1>1?A8@??A331?@@41-2;>191:@5;:10?A331?@5;:;25:/>1-?109;@5B-@5;:5??@>;:31>    !!%"#$$! $:@45?1D<1>591:@@41<->@5/5<-:@?C1>1-?710@;-:?C1>=A1?@5;:?-.;A@@415>59<>1??5;:?
;2@41>;.;@5:-=A1?@5;::-5>1?A>B1E-2@1>1-/41D<1>591:@-:05:-:5:@1>B51C?A>B1E-2@1>
1-/41D<1>591:@-80-E )41>1?A8@?>1B1-810@4-@-2@1>>1<1-@105:@1>-/@5;:?@41<->@5/5<-:@?
.13-:@;6A031@41>;.;@-?-4A9-:.15:3 18;C->1@41>1?A8@?;2@4125>?@-:02;A>@4
5:@1>B51C?-:0@41>1?A8@?;2@415:@1>B51C;:@41@45>00-E-2@1>@41252@45:@1>B51C5:C45/4
/4-:31?C1>1<->@5/A8->8E1B501:@ %%!*($')'()'$$)""+'.%%. .))(,.,(!))!(%%$#))))($#)")($($ $,+',(+'.
%%.,#)'$$)+"))# 
0! )$($!+) #&*()$#$#"$')"*()# #($!+) )# #$) 53A>19<>1??5;:?-2@1>@4125>?@?1??5;:'()'$*#$%%!*((")$$#.#*!) ),(().,'(.#",!!$# /($#'$*#$%%!*(!$$ ! '#$#".( )("(,((.#!!
$# )')",("''((  )')"(,(" ##"''(( "$)$#*)$*!#)*#'()#),!! 
53A>19<>1??5;:?-2@1>@412;A>@4?1??5;:
()(*)*((($ )'$$)(()')#)$!$$ "$'#"$'! *"#)'!!  -%'"#)'*"# ( ()(*)( ,((*'%'()))'$$),("$+#)'()*)#$,)("(#$'"!
##)*'! .( 53A>1-E :@1>B51C?


)41>1?A8@?;2@415:@1>B51C?5:05/-@1@4-@2;>@4125>?@@591@4181->:1>?.13-:@;?11@41>;.;@-?-4A9-:.15:3?;91@45:3@4-@4-0:;@;//A>>10.12;>1@41?1/;:00-E;2@>-5:5:3 )41>1?A8@?;2@41=A1?@5;::-5>1?A>B1E5:05/-@10@4-@@41>;.;@.13-:@;3A1??C4-@5@
C-:@10@;?-E@;81->:1>?  
)412;88;C5:3@C;<;5:@?->159<;>@-:@5:21>1:/1?2>;9@41>1?A8@?;2@45?1D<1>591:@ • )41>15?-<>;<;>@5;:-8>18-@5;:?45<.1@C11:@41?1:?1;2/;9<1@5@5;:-:0&11>:1??
>-@5:3?;B1>@41/;A>?1;2-0-E • :0A/@5;:;25:@1:@5;:1?@59-@5;:.E<>1?1:@-@5;:;2:;:B1>.-85:2;>9-@5;:276 #!"!#%! %! $"%( %$ $!!"%%!  # $$
% $!'#% !&#$! )
)41>1?A8@?;2@45?1D<1>591:@5:05/-@1@4-@@41>1C-?:;59<>;B191:@5:1-/4;2@41
/;9<1@5@5B1:1??-:0<11>:1??>-@5:3?@4>;A34;A@@411:@5>10-E1D<1>591:@ ;C1B1>
@41>1?A8@?;2@411D<1>591:@5:05/-@10@4-@@41@C;2-/@;>?59<>;B10<>;<;>@5;:-88E )45?
?A331?@?@4-@@41>;.;@?:;:B1>.-85:2;>9-@5;:9-E5:0A/1-?1:?1;2/;9<1@5@5;:-?C188
-?<11>:1??4;C1B1>5@-8?;?A331?@?@41<;??5.585@E;2-01/>1-?15:@411B-8A-@5;:;2@41>;.;@-?0-E?<-??10 )45?/;A80.10A1@;@411221/@?;20-E?C5@/45:3@41/-A?1;2C45/4
>19-5:?A:/81-> )41>12;>1@411221/@?;2?C5@/45:30-E??4;A80.12A>@41>5:B1?@53-@10   &%! ! % %!  $%%! )#$ %%! !! '# !#%! )41=A1?@5;::-5>1-:05:@1>B51C?A>B1E>1?A8@?5:05/-@10-:5:/>1-?15:<->@5/5<-:@?I
9;@5B-@5;:@;81->:9-@419-@5/?@4>;A34;A@@411D<1>591:@ )45?5:/>1-?19-E:;@.1?;818E
-@@>5.A@-.81@;59<>;B10?1821225/-/E@4>;A34>1<1-@1081->:5:3-?<>;.819?;2B->E5:3
05225/A8@EC1>1/;:?5?@1:@8E<>1?1:@10C5@4?;91>19-5:5:3A:-:?C1>10C5@45:@41@5918595@
5:1-/4?1??5;: 005@5;:-88E53A>1?-:001<5/@-3>;C5:3<1>/1<@5;:;2@41>;.;@-?
4A9-:?A331?@5:3-@4;A34@<>;/1??-75:@;-:@4>;<;9;><45?9.-?10;:%?-C- )41>1?A8@?5:05/-@10@4-@@41>;.;@5:21>?5@?5:@1:@5;:?@4>;A34@411D45.5@109;B191:@? #;>1;B1>@41>;.;@?.14-B5;>?@->@?2>;9@4181->:1>?1D1>/5?1>18-@105:2;>9-@5;:
<;@1:@5-88E@>5331>5:3-:-/@5B1?@>-@13E-:001/5?5;:9-75:3>18-@10@;@41-A@;:;9E
/;9<;:1:@;25:@>5:?5/9;@5B-@5;: )45?5:/>1-?15:-A@;:;9E9-E81-0@;1:4-:/105:@>5:?5/
9;@5B-@5;: 005@5;:-88E81->:1>??182-225>9-@5;:5:/>1-?10C41:@41E.1851B10@41E/;A80
?;8B1<>;.819?@41:1D@@5912A>@41>?A<<;>@5:3@415>5:@>5:?5/9;@5B-@5;: 


  




!*:-.:" (.5*:-. =A,4.& &,15.2.7&  '1.$..:869*7287$:80:*67@9.:2.7<2*5 .*:7270#998:<=72<A &# 8=:7*58/ .2;=:.&<=-2.;*7-%.,:.*<287
-=,*<287   
4212:8*;121*:*    .*:7270?2<1%8+8</8:71*7,27070*0.6.7< 866=72<A/8:778>*<287
8/-=,*<287*7-5.*:7270<1:8=01869=<.:;*7-,866=72,*<2877.<?8:4;   
*;*2'   855*+8:*<287*68700.7<;27*(2:<=*5855*+8:*<2>. .*:72707>2:876.7< 7/8:6*<287$:8,.;;270&8,2.<A8/*9*7       
1*7'    7<.0:*<2872- .*:7270869*7287&A;<.6 $:8, #/ <1(85  99 
  
A*62)848A*6* %A=32#68:2  !8-.5+*;.-*7*5A;2;8/*,<287-.,2;2879:8,.;;27
,855*+8:*<2>.<*;4+*;.-8727<.7<287.;<26*<287 '1.':*7;*,<287;87=7-*6.7<*5;8/5.,<:872,;866=72,*<287;*7-869=<.:&,2.7,.;*9*7.;..-2<287   C 
2:8<*4*#;*?*  8?:<2/2,2*57<.5520.7,.&18=5-.7<1:89868:912B.-D27-270;*7-9952,*<287;8/=6*77<1:89868:912,<<2<=-.D :<2/2,2*57<.5520.7,.    C  
84287-* )8;126*;*'*?*<;=32 '*<;=78:2!*<;=2  87;<:=,<2878/*,869*7287:8+8<+.1*>28:68-.5<1*<27-=,.;9..:524.+.1*>28:-=:2705.*:7270 '1.*9*7.;.&8,2.<A/8:
:<2/2,2*57<.5520.7,.
'*4*A=42*7-* '*4*A=422:*78 *72.5*<87*7-2:8;12;120=:8  $*:<2,29*<2878/7<.:*,<2>.=6*782-%8+8<;27=6*7&8,2.<AD9952,*<287<88:.207 *70=*0.
-=,*<287D8=:7*58/<1.%8+8<2,;&8,2.<A8/*9*7  C '*4*A=42*7-* 2:8;12;120=:8'8:=;12-*  $;A,185802,*5>*5=*<287877<.:*,<287;+.<?..7$.895.*7-%8+8<8=:7*58/<1.%8+8<2,;&8,2.<A8/*9*7      277

6WXG\ RQ 7KH 'HYHORSPHQW RI&RPSXWDWLRQDO 7KLQNLQJ 'HFRPSRVLWLRQ
6WUDWHJLHV IRU 6HQLRU 3ULPDU\ 6WXGHQWV
0HQJWDR /, <D[LQ *8$1
*XDQJ &+(1)DFXOW\ RI (GXFDWLRQ %HLMLQJ 1RUPDO 8QLYHUVLW\ &KLQD
JXDQJ#EQXHGXFQ$EVWUDFW $V DQ LPSRUWDQW FRPSRQHQW RI &RPSXWDWLRQDO 7KLQNLQJ WKH GHFRPSRVLWLRQVWUDWHJ\ KDV DWWUDFWHG WKH DWWHQWLRQ RI UHVHDUFKHUV EXW IHZ VWXGLHV KDYH H[SORUHG KRZ
WKH GHFRPSRVLWLRQ VWUDWHJ\ LWVHOI LV IRUPHG DQG GHYHORSHG ,Q WKLV VWXG\ ZH FDUU\ RXW
WKHPDWLFEDVHG SURJUDPPLQJ WHDFKLQJ IRU VHQLRU SULPDU\ VFKRRO VWXGHQWV 7KH UHVHDUFKLV GLYLGHG LQWR IRXU VWDJHV SUHWHVW H[HUFLVH SRVWWHVW DQG PLJUDWLRQ :H XVH
0LFURJHQHWLF 'HVLJQ WR UHSHDWHGO\ DQG GHHSO\ REVHUYH VWXGHQWV SURJUDPPLQJ OHDUQLQJDFWLYLWLHV LQ IRXU VWDJHV DQG FRQGXFW LQWHUYLHZV DQG UHFRUGV VR DV WR DQDO\]H WKHGHYHORSPHQW SDWK DQG PLJUDWLRQ RI VWXGHQWV GHFRPSRVLWLRQ VWUDWHJLHV 7KH UHVHDUFKUHVXOWV VKRZ WKDW WKH GHFRPSRVLWLRQ DELOLW\ RI VHQLRU SULPDU\ VFKRRO VWXGHQWV FDQ EH
LPSURYHG LQ SURJUDPPLQJ OHDUQLQJ DQG WKH RYHUDOO SHUIRUPDQFH LV WKDW DGYDQFHG
VWUDWHJLHV JUDGXDOO\ UHSODFH ORZOHYHO VWUDWHJLHV +RZHYHU WKLV SURFHVV YDULHV IURPSHUVRQ WR SHUVRQ DQG LV QRW D OLQHDU GHYHORSPHQW DQG WKHUH PD\ EH RWKHU LQIOXHQFLQJIDFWRUV WKDW OHDG WR WKH UHWXUQ RI ORZOHYHO VWUDWHJLHV.H\ZRUGV &RPSXWDWLRQDO 7KLQNLQJ GHFRPSRVLWLRQ VWUDWHJLHV SURJUDPPLQJ SULPDU\ VWXGHQWV

 ,QWURGXFWLRQ,Q WKH VW FHQWXU\ &RPSXWDWLRQDO 7KLQNLQJ &7 KDV EHFRPH D SUHUHTXLVLWH VNLOO IRU LQGLYLGXDOVWR FRSH ZLWK D ZLGH UDQJH RI QHZ FKDOOHQJHV DQG LV LQIOXHQFLQJ YDULRXV ILHOGV UHODWHG WR WKHQDWXUDO DQG VRFLDO VFLHQFHV &7 KDV JDLQHG ZRUOGZLGH DWWHQWLRQ DV D NH\ OLWHUDF\ IRU VXVWDLQLQJQDWLRQDO FRPSHWLWLYHQHVV DQG KDV EHHQ DGRSWHG DV D JHQHULF VNLOO WR EH OHDUQHG E\ DOO VWXGHQWV
LQ PDQ\ QDWLRQDO LQLWLDWLYHV FXUULFXOD VWDQGDUGV DQG RWKHU SROLF\ GRFXPHQWV 6LQFH WKH
LQWURGXFWLRQ RI &7 WKHUH KDV EHHQ D ORW RI IRFXV RQ KRZ WR GHYHORS WKLV VNLOO +VX HW DO  ,Q WHUPV RI GHYHORSPHQWDO DSSURDFKHV +URPNRYLþ  DQG :LQJ  VXJJHVW
SURJUDPPLQJ DV WKH PDLQ VWUDWHJ\ IRU GHYHORSLQJ &7 LQ VFKRROV 6FKHUHU HW DO  DOVR IRXQG
WKURXJK PHWDDQDO\VLV WKDW SURJUDPPLQJ LV D SULPDU\ PHWKRG IRU GHYHORSLQJ VWXGHQWV &7
1HYHUWKHOHVV D ODUJH DPRXQW RI UHVHDUFK KDV IRFXVHG RQ WKH VWUDWHJLHV DQG HIIHFWV RIGHYHORSLQJ &7 DQG WKHUH LV D ODFN RI UHVHDUFK RQ VWXGHQWV VWUDWHJ\ GHYHORSPHQW DQG
EHKDYLRXU GXULQJ SURJUDPPLQJ
%DVHG RQ WKH DERYH EDFNJURXQG WKLV VWXG\ H[SORUHG WKH GHYHORSPHQWDO SDWWHUQV RIFRJQLWLYH VWUDWHJLHV GXULQJ SURJUDPPLQJ LQ XSSHU SULPDU\ VFKRRO VWXGHQWV )RFXVLQJ RQ
GHFRPSRVLWLRQ VWUDWHJLHV ZKLFK DUH WKH PRVW GLIILFXOW WR EH PDVWHUHG 6HOE\  DQG WKH
PRVW IXQGDPHQWDO 5LMNH HW DO  LQ &7 ZH XVHG D 0LFURJHQHUDWLRQ 'HVLJQ WR GHVFULEH WKHGHYHORSPHQWDO SDWKZD\V DQG GHJUHH RI WUDQVIHU RI VWXGHQWV GHFRPSRVLWLRQ VWUDWHJLHV GXULQJSURJUDPPLQJ DFWLYLWLHV 2Q WKH RQH KDQG ZH IRFXV RQ WKH GHYHORSPHQWDO SURFHVV RIGHFRPSRVLWLRQ VWUDWHJLHV LQ SULPDU\ VFKRRO FKLOGUHQ V &7 RQ WKH RWKHU KDQG ZH SURYLGH
VXSSRUW IRU IXWXUH LQVLJKWV LQWR DVSHFWV RI FRPSXWDWLRQDO WKLQNLQJ DQG LWV LQWHUQDO VWUXFWXUH

 /LWHUDWXUH 5HYLHZ

278

  &RPSXWDWLRQDO 7KLQNLQJ&7 LV D WHUP ILUVW XVHG E\ 3DSHUW  WR GHVFULEH WKH WKRXJKW SURFHVV RI IRUPXODWLQJSUREOHPV DQG WKHLU VROXWLRQV LQ D ZD\ WKDW FDQ EH H[HFXWHG E\ D FRPSXWHU +H IRXQG WKDWOHDUQLQJ ZLWK D SURJUDPPLQJ ODQJXDJH KHOSHG VWXGHQWV WR VLPXODWH WKH DOJRULWKPLF ORJLF RI D
FRPSXWHU ZKLFK LQ WXUQ LQIOXHQFHG WKH ZD\ WKH\ WKRXJKW 3DSHUW  &7 LV FRQVLGHUHG WR EH
WKH SV\FKRORJLFDO EDVLV IRU SUREOHPVROYLQJ GHVLJQLQJ V\VWHPV DQG XQGHUVWDQGLQJ KXPDQEHKDYLRXU DQG FDQ KHOS SHRSOH VLPSOLI\ D GLIILFXOW SUREOHP LQWR PXOWLSOH SUREOHPV WKDW ZH NQRZKRZ WR VROYH :LQJ   7KLV SURFHVV QRW RQO\ SUHSDUHV VWXGHQWV IRU VWXG\ LQ WKH ILHOG RIFRPSXWHU VFLHQFH EXW DOVR SURYLGHV WKHP ZLWK WKH WRROV DQG VNLOOV WR DSSURDFK DQG VROYH
SUREOHPV LQ GLIIHUHQW DUHDV RI NQRZOHGJH :HUQHU HW DO  
 'HFRPSRVLWLRQ VWUDWHJ\$V RQH RI WKH IRXQGDWLRQV RI &7 GHFRPSRVLWLRQ VWUDWHJLHV FDQ GHFRQVWUXFW FRPSOH[ SUREOHPVLQWR PXOWLSOH HDVLO\ VROYDEOH VXESUREOHPV - :LQJ   7KHUHIRUH GHFRPSRVLWLRQ VWUDWHJLHVDUH FUXFLDO IRU VROYLQJ FRPSOH[ DQG ODUJH UHDOZRUOG SUREOHPV 5LMNH HW DO  
7VDL HW DO  DUJXH WKDW GHFRPSRVLWLRQ DQG DEVWUDFWLRQ DUH WZR EDVLF DQG FULWLFDOIDFWRUV UHTXLUHG IRU WKH GHYHORSPHQW RI &7 DQG VXJJHVWHG WKDW GHFRPSRVLWLRQ DQG DEVWUDFWLRQ
DUH WKH VWDUWLQJ SRLQW IRU WHDFKLQJ EDVHG RQ WKH GHYHORSPHQWDO VHTXHQFH RI &7 VNLOOV'HFRPSRVLWLRQ LV SDUWLFXODUO\ LPSRUWDQW IRU WKH GHYHORSPHQW RI &7 VNLOOV IRU EHJLQQLQJSURJUDPPHUV 'HFRPSRVLWLRQ KDV DOZD\V EHHQ HPSKDVL]HG DV D SDUW RI &7 DV D ZKROH EXW IHZ
VWXGLHV KDYH UHYHDOHG VSHFLILF GHWDLOV DERXW GHFRPSRVLWLRQ LWVHOI 5LFK HW DO   $OWKRXJKUHVHDUFKHUV KDYH GHILQHG IRXU OHYHOV RI VWUDWHJLHV IURP  LQ WHUPV RI DVVHVVPHQW FULWHULD IRU
WKHLU GHYHORSPHQW ZLWK GLIIHUHQW FRQFHSWXDO DQG GHVFULSWLYH GHILQLWLRQV $YLOD HW DO   LW
GRHV QRW H[SODLQ KRZ VWXGHQWV 'LVDJJUHJDWLRQ VWUDWHJLHV GHYHORS LQ WKH SURFHVV RI &7
GHYHORSPHQW DQG ODFNV GHSWK DQG JXLGDQFH IRU SUDFWLFH
 9LVXDO SURJUDPPLQJ
9LVXDOLVDWLRQ PHDQV H[SUHVVLQJ DEVWUDFW FRQWHQW LQ DQ LQWXLWLYH ZD\ 5LMNH HW DO   8QOLNH
WH[WXDO SURJUDPPLQJ VXFK DV 3\WKRQ DQG -DYD YLVXDO SURJUDPPLQJ WRROV SURYLGH VWXGHQWV ZLWKWKH RSSRUWXQLW\ WR OHDUQ FRPSXWHU SURJUDPPLQJ FRQFHSWV WKURXJK YLVXDO IHHGEDFNEDVHG
OHDUQLQJ UHGXFLQJ WKH FRJQLWLYH ORDG SODFHG RQ VWXGHQWV E\ FRPSOH[ SURJUDPPLQJ JUDPPDU/\H .RK   DQG DOORZLQJ VWXGHQWV WR IRFXV RQ WKH ORJLF DQG VWUXFWXUH LQYROYHG LQ
SURJUDPPLQJ UDWKHU WKDQ ZRUU\LQJ DERXW WKH PHFKDQLFV RI ZULWLQJ SURJUDPV - 0 :LQJ  
1XPHURXV VWXGLHV KDYH VKRZQ WKDW LW PDNHV VHQVH WR GHYHORS GHFRPSRVLWLRQ VWUDWHJLHV IURPWKH HOHPHQWDU\ VFKRRO OHYHO RQ WKH RQH KDQG SURJUDPPLQJ GHYHORSV &7 %XLWUDJR )OyUH] HW DO  2Q WKH RWKHU KDQG FKLOGUHQ DOUHDG\ KDYH WKH FRJQLWLYH OHYHO WR XQGHUVWDQG EDVLFFRPSXWHU SURJUDPPLQJ FRQFHSWV E\ WKH DJH RI IRXU %HUV HW DO  'XULQJ SURJUDPPLQJ VWXGHQWV DUH H[SRVHG WR &7 XVLQJ FRPSXWHU VFLHQFH FRQFHSWV VXFKDV DEVWUDFWLRQ GHFRPSRVLWLRQ DQG JHQHUDOL]DWLRQ WR VROYH SUREOHPV 3URJUDPPLQJ HGXFDWLRQVKRXOG IRFXV RQ KRZ VWXGHQWV FDQ EUHDN GRZQ DQG FDWHJRUL]H FRPSOH[ SUREOHPV WR VROYH WKHPHIILFLHQWO\ 6iH]/ySH] HW DO   7KH %ORFNFRGLQJ SURJUDPPLQJ WRRO XVHG LQ WKLV VWXG\ LV
WKH 'LVFRYHU\ (GXFDWLRQ &RGLQJ &XUULFXOXP '(&&  ZKLFK VXSSRUWV VWXGHQWV LQ DOORFDWLQJOLPLWHG FRJQLWLYH UHVRXUFHV WR WKH PRELOL]DWLRQ RI SUREOHPVROYLQJ VWUDWHJLHV IRU VWXGHQWV ZKRDUH QHZ WR SURJUDPPLQJ 5HVHDUFK TXHVWLRQV
3LDJHW EHOLHYHV WKDW FKLOGUHQ V FRJQLWLYH VWUDWHJ\ GHYHORSPHQW SDWWHUQ KDV IRXU VWDJHV IURP ORZ
WR KLJK ,Q &7 GHYHORSPHQW WKHUH LV YDULDELOLW\ LQ WKH GHYHORSPHQW RI GLIIHUHQW HOHPHQWV LQ WKHIRXU VWDJHV (YHQ LI WKH\ KDYH EHHQ WUDLQHG E\ WHDFKHUV LQ DGYDQFHG VWUDWHJLHV IRU HDFKHOHPHQW WKH\ ZLOO EH LQ WKH QRVWUDWHJ\ VWDJH RU WKH ORZVWUDWHJ\ VWDJH DQG ZLOO QRW EH DEOH WR

279

XVH DGYDQFHG VWUDWHJLHV FRPIRUWDEO\ GXH WR WKH ODFN RI FRJQLWLYH UHVRXUFHV )RFXVLQJ RQ &7
FDQ KHOS VWXGHQWV EH DEOH WR VHOHFW DQG DSSO\ DSSURSULDWH VWUDWHJLHV DQG WRROV WR EHWWHU
FRQFHSWXDOL]H DQDO\]H DQG VROYH FRPSOH[ SUREOHPV +RZHYHU PRVW UHVHDUFK KDV IRFXVHG RQ
LVVXHV UHODWHG WR OHDUQLQJ VNLOOV FRQFHSWV DQG SUDFWLFHV (]HDPX]LH /HXQJ  *URYHU
3HD   DQG WKH VSHFLILF SURFHVVHV LQYROYHG LQ WKH GHYHORSPHQW RI VWXGHQWV &7 QHHG WR EHIXUWKHU H[SORUHG 7KLV VWXG\ IRFXVHV RQ RQH RI WKH PRVW IXQGDPHQWDO DQG FULWLFDO VWDUWLQJ SRLQWV
LQ WKH GHYHORSPHQW RI &7 GHFRPSRVLWLRQ VWUDWHJLHV %\ LPSOHPHQWLQJ LQVWUXFWLRQDO DFWLYLWLHVWKDW IDFLOLWDWH WKH GHYHORSPHQW RI &7 GHFRPSRVLWLRQ VWUDWHJLHV LQ HOHPHQWDU\ FODVVURRPV DQG
WUDFNLQJ VWXGHQWV EHKDYLRUV DQG FKDUDFWHULVWLFV LQ WKH SURFHVV WKLV VWXG\ SURSRVHV WR DGGUHVVWKH IROORZLQJ TXHVWLRQV
54 +RZ ZHOO GR VHQLRU SULPDU\ VWXGHQWV XVH GHFRPSRVLWLRQ VWUDWHJLHV DW GLIIHUHQW
VWDJHV"54 :KDW DUH WKH GHYHORSPHQWDO SDWKZD\V RI GHFRPSRVLWLRQ VWUDWHJLHV IRU VWXGHQWV LQ
VHQLRU SULPDU\ JUDGHV DW GLIIHUHQW VWDJHV RI WKH VDPH SURJUDPPLQJ DFWLYLW\"
54 7R ZKDW H[WHQW GR VHQLRU SULPDU\ VFKRRO VWXGHQWV WUDQVIHU GLIIHUHQW VWUDWHJLHV ZLWKLQWKH VDPH SURJUDPPLQJ DFWLYLW\" 5HVHDUFK 0HWKRGRORJ\:H H[SORUHG WKH GHYHORSPHQW RI &7 GHFRPSRVLWLRQ VWUDWHJLHV DQG SDWKZD\V LQ SULPDU\ VFKRROVWXGHQWV D PLFURLQFLGHQFH GHVLJQ ZDV XVHG WR FROOHFW DQG FRGH UHOHYDQW GDWD WKURXJKFODVVURRP EHKDYLRXUDO REVHUYDWLRQV DQG SRVWFODVVURRP LQWHUYLHZV 6XEMHFWV RI VWXG\
7KH VXEMHFWV RI WKH VWXG\ ZHUH WKJUDGH VWXGHQWV LQ D SULPDU\ VFKRRO LQ %HLMLQJ 7KURXJK SUHH[SHULPHQWDO REVHUYDWLRQV LQWHUYLHZV DQG GDWD DQDO\VHV  VXEMHFWV ZHUH ILQDOO\ LGHQWLILHG
'XULQJ WKH LPSOHPHQWDWLRQ RI WKH DFWLYLW\  VWXGHQWV GLG QRW SDUWLFLSDWH LQ WKH FRPSOHWH
H[SHULPHQWDO SURFHVV IRU SHUVRQDO UHDVRQV VR  YDOLG GDWD ZHUH UHFHLYHG DIWHU WKH WHDFKLQJDFWLYLW\ 7KH VDPSOH RI  FRQVLVWHG RI  ER\V DQG  JLUOV DOO DJHG  \HDUV 0  6' 
 6WXG\ GHVLJQ
0LFURJHQHWLF 'HVLJQ LV D UHVHDUFK PHWKRG WR VWXG\ WKH FRJQLWLYH DQG EHKDYLRUDO GHYHORSPHQW
SURFHVV RI FKLOGUHQ ZKLFK LV ZLGHO\ XVHG LQ WKH UHVHDUFK RI FRJQLWLYH VWUDWHJ\ GHYHORSPHQW RI\RXQJ FKLOGUHQ WR KLJK VFKRRO VWXGHQWV 6LHJOHU &URZOH\   7KURXJK LWHUDWLYH DQG LQGHSWK REVHUYDWLRQ DQG DQDO\VLV RI FKDQJHV LQ EHKDYLRU WKURXJKRXW WKH SURFHVV 0LFURJHQHWLF'HVLJQ FDQ SURYLGH GHWDLOHG LQIRUPDWLRQ DERXW WKH VRXUFHV SDWKZD\V YHORFLW\ DQG H[WHQW RIFRJQLWLYH FKDQJHV DV ZHOO DV WKH GLYHUVLW\ RI FKDQJH SDWWHUQV 7KHUH DUH VHYHUDO UHDVRQV 
REVHUYDWLRQV FRYHU WKH HQWLUH LQWHUYDO IURP WKH RQVHW RI FKDQJH WR UHODWLYH VWDELOLW\  WKH
IUHTXHQF\ RI REVHUYDWLRQV LV KLJKO\ FRQVLVWHQW ZLWK WKH UDWH RI FKDQJH RI WKH SKHQRPHQRQ DQG WKH REVHUYHG EHKDYLRU LV ILQHO\ LWHUDWLYHO\ DQDO\]HG LQ LWHUDWLYH H[SHULPHQWV WR IDFLOLWDWH
VSHFXODWLRQ RQ WKH SURFHVVHV WKDW SURGXFH TXDOLWDWLYH DQG TXDQWLWDWLYH FKDQJHV ,Q VKRUW WKHKLJK GHQVLW\ RI ILQHJUDLQHG GDWD SURYLGHV D JXDUDQWHH RI WKH YDOLGLW\ RI WKLV DSSURDFK7UDGLWLRQDO SUH DQG SRVWWHVW H[SHULPHQWV KDYH D ORZ GHQVLW\ RI REVHUYDWLRQ RI WKH SURFHVVRI EHKDYLRXUDO FKDQJH WKURXJKRXW WKH VWXG\ DQG GR QRW DOORZ IRU D VRSKLVWLFDWHG DQDO\VLV RIVWUDWHJ\ GHYHORSPHQW EHKDYLRXU GXULQJ WKLV SURFHVV 7KLV FDQ EH ZHOO DYRLGHG E\ XVLQJ D
0LFURJHQHWLF 'HVLJQ WR REVHUYH VWXGHQWV EHKDYLRXU DQG WKLQNLQJ LQ WKH FODVVURRP GXULQJ WKHFRXUVH RI WKH OHVVRQ :KHQ HSLVRGLF EHKDYLRXUV ZHUH PRUH DPELJXRXV RU DEVHQW D FDWHJRULFDOEDVLV IRU VWUDWHJ\ DVVHVVPHQW ZDV REWDLQHG E\ SURPSWO\ DVNLQJ VWXGHQWV KRZ WKH\ GHVLJQHG WKH
SURFHGXUH

280

 5HVHDUFK SURFHVV
7KH H[SHULPHQW FRQVLVWHG RI WZR SDUWV D SUHH[SHULPHQW DQG D IRUPDO H[SHULPHQW $OO VL[VHVVLRQV ZHUH WDXJKW E\ WKH VDPH UHVHDUFKHU DQG ODVWHG WZR DQG D KDOI ZHHNV ZLWK HDFKVHVVLRQ ODVWLQJ  PLQXWHV 7KH VL[ FODVV SHULRGV ZHUH FHQWUHG DURXQG VL[ WKHPDWLF DFWLYLWLHVWKDW ZHUH WDXJKW DV &7 DFWLYLWLHV 0RYHPHQW 6LPSOH ,QSXW 'LIIHUHQW .LQGV RI ,QSXW %XWWRQV&RPPDQGV 6HTXHQFH $QLPDWLRQ DQG &RQGLWLRQDO (YHQWV (DFK WKHPH KDV WKUHH GLIIHUHQW
SURJUDPPLQJ DFWLYLWLHV XQGHU LW DQG VWXGHQWV QHHG WR FRPSOHWH WKHVH WKUHH SURJUDPPLQJDFWLYLWLHV DQG SHUIRUP D SURJUDPPLQJ FKDOOHQJH LQ HDFK OHVVRQ ZKLFK WRJHWKHU FRQVWLWXWH WKH
SUHWHVW SUDFWLFH SRVWWHVW DQG PLJUDWLRQ SKDVHV RI WKH H[SHULPHQW,Q WKH SUHH[SHULPHQW 7RSLF $FWLYLW\  ZDV XVHG WR VFUHHQ VWXGHQWV ZLWK ]HUR IRXQGDWLRQ$IWHU FRPSOHWLQJ HDFK WDVN VWXGHQWV ZHUH WHVWHG RQ WKHLU GHFRPSRVLWLRQ VNLOOV$OO  WKHPDWLF DFWLYLWLHV LQ WKH IRUPDO H[SHULPHQW FRQVLVW RI  OHDUQLQJ WDVNV ZKLFK DUH XVHGWR DVVHVV VWXGHQWV &7 VWUDWHJ\ GHYHORSPHQW LQ WKH SUHWHVW SUDFWLFH DQG SRVWWHVW SKDVHV ,Q
DGGLWLRQ WR WKLV VWXGHQWV ZHUH UHTXLUHG WR XQGHUWDNH WKH FRPSOHWLRQ RI D SURJUDPPLQJ SURMHFW
DV D PLJUDWLRQ WHVW 'DWD FROOHFWLRQ'DWD ZHUH FROOHFWHG WKURXJK FODVVURRP REVHUYDWLRQV DQG LQWHUYLHZV &ODVVURRP REVHUYDWLRQVZHUH FRQGXFWHG WR WUDFN DQG UHFRUG VWXGHQW EHKDYLRXU DQG GHYHORSPHQW LQ WKH FODVVURRP,QWHUYLHZV ZHUH FRQGXFWHG LQ WKH IRUP RI TXHVWLRQV DVNHG E\ WKH WHDFKHU DQG VKDUHG E\ WKH
VWXGHQWV LQ WKH FODVVURRP DQG ZHUH XVHG IRU IXUWKHU DQDO\VLV DQG UHILQHPHQW ,QWHUYLHZ RXWOLQHV
ZHUH SUHSDUHG E\ WKH UHVHDUFKHU EDVHG RQ WKH OLWHUDWXUH DQG ZKDW ZDV IRFXVHG RQ LQ WKLV VWXG\
DQG WKH LQWHUYLHZV LQFOXGHG VWXGHQWV WKRXJKWV RQ SURJUDPPLQJ DV ZHOO DV WKHLU IHHOLQJV DERXWOHDUQLQJ GXULQJ WKH IRXU SKDVHV'DWD ZHUH FRGHG DFFRUGLQJ WR WKH GHFRPSRVLWLRQ VWUDWHJ\ OHYHO FRGLQJ WDEOH %DVHG RQ D
GHILQLWLRQ RI GHFRPSRVLWLRQ VWUDWHJ\ GHYHORSPHQW OHYHOV SURSRVHG E\ $YLOD HW DO   ZH WRRN
LQWR DFFRXQW VSHFLILF SURJUDPPLQJ VLWXDWLRQV DV ZHOO DV WKHRULHV UHODWHG WR FRJQLWLYH VWUDWHJLHVDQG LQYLWHG WZR UHVHDUFKHUV LQ WKH ILHOG RI &7 WR FODVVLI\ VWXGHQWV GHFRPSRVLWLRQ VWUDWHJLHV LQWR
IRXU OHYHOV 7KH 'HFRPSRVLWLRQ 6WUDWHJ\ +LHUDUFK\ VFDOH LV VKRZQ LQ WKH DSSHQGL[ 5HVHDUFK ILQGLQJV
7KH UHVXOWV RI WKH VWXG\ ZLOO GHSLFW WKH FKDQJH URXWHV DQG GHYHORSPHQW RI GHFRPSRVLWLRQVWUDWHJLHV LQ VHQLRU SULPDU\ VFKRRO VWXGHQWV GXULQJ WKH FRPSOHWLRQ RI SURJUDPPLQJ WDVNV LQ WKHIROORZLQJ WKUHH ZD\V 6WXGHQWV RYHUDOO XVH RI GHFRPSRVLWLRQ VWUDWHJLHV DW GLIIHUHQW VWDJHV$V VKRZQ LQ )LJXUH  WKH 5RPD QXPHUDOV IURP VPDOO WR ODUJH UHSUHVHQW WKH IRXU VWDJHV RI SUH
WHVW SUDFWLFH SRVWWHVW DQG PLJUDWLRQ ,Q WKH SUHWHVW VWDJH VWXGHQWV PDLQ GHFRPSRVLWLRQVWUDWHJ\ LV WKH OHYHO  VWUDWHJ\ DQG WKH\ KDUGO\ XVH WKH DGYDQFHG VWUDWHJ\ LQ WKH SUDFWLFH
VWDJH OHYHO  VWUDWHJLHV DUH WKH VWXGHQWV PDLQ VWUDWHJLHV DQG OHYHO  VWUDWHJLHV EHJLQ WR
DSSHDU LQ WKH SRVWWHVW VWDJH OHYHO  DQG OHYHO  VWUDWHJLHV EHFRPH WKH PDLQ VWUDWHJLHV DQG
OHYHO  VWUDWHJLHV KDYH EHHQ FRPSOHWHO\ UHSODFHG LQ WKH PLJUDWLRQ VWDJH WKH XVH RI OHYHO VWUDWHJLHV UHDFKHV  DQG WKH XVH RI OHYHO  VWUDWHJLHV LV DV KLJK DV 281

)LJXUH  7KH XVH RI GHFRPSRVLWLRQ VWUDWHJLHV DW GLIIHUHQW VWDJHV 'HYHORSPHQW RI VWXGHQWVDFWLYLWLHV 3DWKZD\V

GHFRPSRVLWLRQVWUDWHJLHVLQSURJUDPPLQJ$V VKRZQ LQ )LJXUH  WKH OLQH IURP OLJKW EOXH WR GDUN EOXH UHSUHVHQWV WKH IRXUOHYHOGHFRPSRVLWLRQ VWUDWHJ\ IURP ORZ WR DGYDQFHGƋHJ WKH OLJKWHVW EOXH OLQH UHSUHVHQWV WKH OHYHOGHFRPSRVLWLRQ VWUDWHJ\  7KH URXWH DQDO\VLV RI WKH VWUDWHJLHV DGRSWHG LQ IRXU GLIIHUHQW VWDJHV RIWKH VDPH SURJUDPPLQJ DFWLYLW\ VKRZV WKDW WKH GHYHORSPHQW WUHQG RI GLIIHUHQW OHYHOV RIGHFRPSRVLWLRQ VWUDWHJLHV LQ GLIIHUHQW WKHPDWLF DFWLYLWLHV LV EDVLFDOO\ WKH VDPH ZKLFK LVPDQLIHVWHG DV WKH JUDGXDO UHSODFHPHQW RI ORZOHYHO VWUDWHJLHV E\ KLJKOHYHO VWUDWHJLHV)LJXUH  7KH RYHUDOO OLQH RI GHYHORSPHQW RI WKH GHFRPSRVLWLRQ VWUDWHJ\ XQGHU GLIIHUHQW
WKHPDWLF DFWLYLWLHV 7UDQVIHU RI VWXGHQWV
H[WHQW

GHFRPSRVLWLRQ VWUDWHJLHV LQ SURJUDPPLQJ DFWLYLWLHV)LJXUH  VKRZV D URDGPDS IRU LQGLYLGXDOV WR DGRSW VWUDWHJLHV DW GLIIHUHQW VWDJHV RI SURJUDPPLQJDFWLYLW\ ༃ UHSUHVHQWV WKH OHYHO DQG WKH QXPEHU RQ WKH KRUL]RQWDO OLQH EHWZHHQ WKH WZR OHYHOVUHSUHVHQWV WKH QXPEHU RI SHRSOH ZKRVH URXWH FKDQJHV EHWZHHQ WKH WZR VWUDWHJ\ OHYHOV 7KH

282

UHVXOWV VKRZ WKDW EHIRUH EHLQJ WDXJKW SURJUDPPLQJ DOO VWXGHQWV LQLWLDOO\ XVHG RQO\ ORZOHYHOGHFRPSRVLWLRQ VWUDWHJLHV DQG WKH PDMRULW\ RI VWXGHQWV XVHG ]HUROHYHO VWUDWHJLHV ,Q PRVW
FDVHV WKH GHFRPSRVLWLRQ VWUDWHJLHV XVHG E\ VWXGHQWV IROORZ D FRQWLQXRXV GHYHORSPHQW IURPORZOHYHO WR KLJKOHYHO VWUDWHJLHV EXW WKHUH ZLOO EH D EULHI UHJUHVVLRQ IURP KLJKOHYHO VWUDWHJLHV WRORZOHYHO VWUDWHJLHV)LJXUH  7KH GHJUHH RI LQGLYLGXDO PLJUDWLRQ RI VWUDWHJLHV LQ GLIIHUHQW DFWLYLWLHV 'LVFXVVLRQ DQG FRQFOXVLRQV
7KH SXUSRVH RI WKLV VWXG\ LV WR LQYHVWLJDWH WKH GHYHORSPHQWDO SDWKZD\V DQG VWUDWHJ\ WUDQVIHU RI&7 GHFRPSRVLWLRQ VWUDWHJLHV RI VWXGHQWV LQ WKH XSSHU SULPDU\ JUDGHV DQG ZH FDQ VXPPDULVH
WKUHH PDLQ ILQGLQJV ,LUVWO\ VWXGHQWV GHFRPSRVLWLRQ VWUDWHJLHV FDQ EH GHYHORSHG WKURXJK WRSLF
EDVHG SURJUDPPLQJ OHDUQLQJ DQG SURJUDPPLQJ LQVWUXFWLRQ LQ WKH XSSHU SULPDU\ JUDGHV FDQ EH
DQ HIIHFWLYH ZD\ WR GHYHORS VWXGHQWV GHFRPSRVLWLRQ VNLOOV 7KLV UHVXOW UHDIILUPV %HUV HWDO  DQG 5LMNH HW DO  WKDW LW LV IHDVLEOH DQG PHDQLQJIXO WR GHYHORS VWXGHQWVGHFRPSRVLWLRQ VNLOOV E\ WHDFKLQJ SURJUDPPLQJ IURP WKH SULPDU\ OHYHO 6HFRQGO\ VWXGHQWVGHFRPSRVLWLRQ VWUDWHJLHV DUH QRW OLQHDU EXW D G\QDPLF SURFHVV RI GHYHORSPHQW DQG WKHUH PD\
EH RWKHU IDFWRUV WKDW OHDG WR WKH UHFXUUHQFH RI ORZOHYHO VWUDWHJLHV 7KLV UHVXOW LV VLPLODU WR WKH
ILQGLQJV RI 6LHJOHU 6WHUQ  RQ WKH GHYHORSPHQW RI PDWKHPDWLFDO VWUDWHJLHV LQ VHFRQG
JUDGHUV ,Q DGGLWLRQ VWXGHQWV GHFRPSRVLWLRQ VWUDWHJ\ GHYHORSPHQW LV D FRPSOH[ SURFHVV DQG
VWXGHQWV RZQ H[SHULHQFHV WHDFKHUV LQWHUYHQWLRQV DQG GLIIHUHQFHV LQ LQVWUXFWLRQDO FRQWHQW PD\
DIIHFW WKH GHYHORSPHQWDO WUHQGV DQG WKH GHJUHH RI WUDQVIHU RI VWXGHQWV GHFRPSRVLWLRQVWUDWHJLHV ,Q WKH SURFHVV RI JXLGLQJ VWXGHQWV WR OHDUQ SURJUDPPLQJ WHDFKHUV QHHG WR SURYLGHHIIHFWLYH JXLGDQFH WR KHOS VWXGHQWV GHFRPSRVH SUREOHPV DQG WDVNV $W WKH VDPH WLPH WKURXJKWKH FUHDWLRQ RI VSHFLILF SUREOHP VLWXDWLRQV DQG SUDFWLFH RSSRUWXQLWLHV WHDFKHUV FDQ WUDQVIHU WKH
GHFRPSRVLWLRQ VWUDWHJ\ WHDFKLQJ FRQWHQW WR RWKHU GLVFLSOLQHV DQG PDNH HIIHFWLYH DGMXVWPHQWV WR
WKH FXUULFXOXP GHVLJQ VR DV WR LPSURYH VWXGHQWV GHFRPSRVLWLRQ VWUDWHJLHV LQ PXOWLSOH ZD\V

3UHIHUHQFH%HUV 0 8 )ODQQHU\ / .D]DNRII ( 5 6XOOLYDQ $   &RPSXWDWLRQDO WKLQNLQJ DQG
WLQNHULQJ ([SORUDWLRQ RI DQ HDUO\ FKLOGKRRG URERWLFV FXUULFXOXP &RPSXWHUV (GXFDWLRQ ± KWWSVGRLRUJMFRPSHGX

283

%XLWUDJR )OyUH] ) &DVDOODV 5 +HUQiQGH] 0 5H\HV $ 5HVWUHSR 6 'DQLHV *  &KDQJLQJ D *HQHUDWLRQ¶V :D\ RI 7KLQNLQJ 7HDFKLQJ &RPSXWDWLRQDO 7KLQNLQJ 7KURXJK
3URJUDPPLQJ 5HYLHZ RI (GXFDWLRQDO 5HVHDUFK    ±KWWSVGRLRUJ(]HDPX]LH 1 2 /HXQJ - 6 &   &RPSXWDWLRQDO 7KLQNLQJ 7KURXJK DQ (PSLULFDO/HQV $ 6\VWHPDWLF 5HYLHZ RI /LWHUDWXUH -RXUQDO RI (GXFDWLRQDO &RPSXWLQJ 5HVHDUFK   ± KWWSVGRLRUJ
*URYHU 6 3HD 5   &RPSXWDWLRQDO 7KLQNLQJ LQ .± $ 5HYLHZ RI WKH 6WDWH RI WKH)LHOG (GXFDWLRQDO 5HVHDUFKHU    ±
KWWSVGRLRUJ;
+URPNRYLþ -   &RQWULEXWLQJ WR *HQHUDO (GXFDWLRQ E\ 7HDFKLQJ ,QIRUPDWLFV ,Q 5 7
0LWWHUPHLU (G  ,QIRUPDWLFV (GXFDWLRQ ± 7KH %ULGJH EHWZHHQ 8VLQJ DQG 8QGHUVWDQGLQJ&RPSXWHUV 9RO  SS ±  6SULQJHU %HUOLQ +HLGHOEHUJKWWSVGRLRUJB
+VX 7& &KDQJ 6& +XQJ <7   +RZ WR OHDUQ DQG KRZ WR WHDFK FRPSXWDWLRQDOWKLQNLQJ 6XJJHVWLRQV EDVHG RQ D UHYLHZ RI WKH OLWHUDWXUH &RPSXWHUV (GXFDWLRQ ± KWWSVGRLRUJMFRPSHGX
.DIDL < % 3URFWRU &   $ 5HYDOXDWLRQ RI &RPSXWDWLRQDO 7KLQNLQJ LQ .± (GXFDWLRQ
0RYLQJ 7RZDUG &RPSXWDWLRQDO /LWHUDFLHV (GXFDWLRQDO 5HVHDUFKHU    ±KWWSVGRLRUJ;
/\H 6 < .RK - + /   5HYLHZ RQ WHDFKLQJ DQG OHDUQLQJ RI FRPSXWDWLRQDO WKLQNLQJWKURXJK SURJUDPPLQJ :KDW LV QH[W IRU ." &RPSXWHUV LQ +XPDQ %HKDYLRU  ±
KWWSVGRLRUJMFKE2WHUR $YLOD & )RVV / %RUGLQL $ 6LPRQH 'HEDFFR 0 'D &RVWD &DYDOKHLUR 6 $  (YDOXDWLRQ 5XEULF IRU &RPSXWDWLRQDO 7KLQNLQJ &RQFHSWV  ,((( WK ,QWHUQDWLRQDO&RQIHUHQFH RQ $GYDQFHG /HDUQLQJ 7HFKQRORJLHV ,&$/7  ±KWWSVGRLRUJ,&$/7
3DSHUW 6   0LQGVWRUPV &KLOGUHQ FRPSXWHUV DQG SRZHUIXO LGHDV 5HSULQW  +DUYHVWHU
3UHVV
3DSHUW 6   0LQGVWRUPV &KLOGUHQ FRPSXWHUV DQG SRZHUIXO LGHDVKWWSVFQELQJFRPDFDGHPLFSURILOH"LG GFDFDEDH HQFRG
HG  Y SDSHUBSUHYLHZ PNW ]KFQ
5LFK 3 - (JDQ * (OOVZRUWK -   $ )UDPHZRUN IRU 'HFRPSRVLWLRQ LQ &RPSXWDWLRQDO
7KLQNLQJ 3URFHHGLQJV RI WKH  $&0 &RQIHUHQFH RQ ,QQRYDWLRQ DQG 7HFKQRORJ\ LQ&RPSXWHU 6FLHQFH (GXFDWLRQ ± KWWSVGRLRUJ
5LMNH : - %ROOHQ / (\VLQN 7 + 6 7ROERRP - / -   &RPSXWDWLRQDO 7KLQNLQJ LQ
3ULPDU\ 6FKRRO $Q ([DPLQDWLRQ RI $EVWUDFWLRQ DQG 'HFRPSRVLWLRQ LQ 'LIIHUHQW $JH*URXSV ,QIRUPDWLFV LQ (GXFDWLRQ    ± KWWSVGRLRUJLQIHGX
6iH]/ySH] -0 5RPiQ*RQ]iOH] 0 9i]TXH]&DQR (   9LVXDO SURJUDPPLQJODQJXDJHV LQWHJUDWHG DFURVV WKH FXUULFXOXP LQ HOHPHQWDU\ VFKRRO $ WZR \HDU FDVH VWXG\
XVLQJ ³6FUDWFK´ LQ ILYH VFKRROV &RPSXWHUV (GXFDWLRQ  ±KWWSVGRLRUJMFRPSHGX
6FKHUHU 5 6LGGLT ) 6iQFKH] 9LYHURV %   7KH FRJQLWLYH EHQHILWV RI OHDUQLQJFRPSXWHU SURJUDPPLQJ $ PHWDDQDO\VLV RI WUDQVIHU HIIHFWV -RXUQDO RI (GXFDWLRQDO
3V\FKRORJ\    ± KWWSVGRLRUJHGX
6HOE\ & &   5HODWLRQVKLSV &RPSXWDWLRQDO WKLQNLQJ SHGDJRJ\ RI SURJUDPPLQJ DQG%ORRP¶V 7D[RQRP\ 3URFHHGLQJV RI WKH :RUNVKRS LQ 3ULPDU\ DQG 6HFRQGDU\ &RPSXWLQJ(GXFDWLRQ ± KWWSVGRLRUJ
6LHJOHU 5 6 6WHUQ (   &RQVFLRXV DQG XQFRQVFLRXV VWUDWHJ\ GLVFRYHULHV $PLFURJHQHWLF DQDO\VLV -RXUQDO RI ([SHULPHQWDO 3V\FKRORJ\ *HQHUDO    ±
KWWSVGRLRUJ
7VDL 0- /LDQJ -& /HH 6 :< +VX &<   6WUXFWXUDO 9DOLGDWLRQ IRU WKH'HYHORSPHQWDO 0RGHO RI &RPSXWDWLRQDO 7KLQNLQJ -RXUQDO RI (GXFDWLRQDO &RPSXWLQJ
5HVHDUFK    ± KWWSVGRLRUJ

284

:HUQHU / 'HQQHU - &DPSH 6 .DZDPRWR ' &   7KH IDLU\ SHUIRUPDQFH
DVVHVVPHQW 0HDVXULQJ FRPSXWDWLRQDO WKLQNLQJ LQ PLGGOH VFKRRO 3URFHHGLQJV RI WKHUG $&0 7HFKQLFDO 6\PSRVLXP RQ &RPSXWHU 6FLHQFH (GXFDWLRQ ±KWWSVGRLRUJ
:LQJ -   &RPSXWDWLRQDO WKLQNLQJ DQG WKLQNLQJ DERXW FRPSXWLQJ  ,((( ,QWHUQDWLRQDO
6\PSRVLXP RQ 3DUDOOHO DQG 'LVWULEXWHG 3URFHVVLQJ ±
KWWSVGRLRUJ,3'36
:LQJ - 0   &RPSXWDWLRQDO WKLQNLQJ &RPPXQLFDWLRQV RI WKH $&0    ±
KWWSVGRLRUJ

$SSHQGL[
7DEOH  'HFRPSRVLWLRQ 6WUDWHJ\ +LHUDUFK\*UDGH/HYHO 6WUDWHJ\'HILQLWLRQ
6SHFLILF GHVFULSWLRQ'XULQJ WKH DQDO\VLV RI WKH
6XEMHFWLYH MXGJHPHQW WKH SURFHVV RI LGHQWLI\LQJ WKH
SURJUDPPLQJ WDVN SURJUDPPLQJ WDVN SUREOHP DQG WKXV FRPSOHWLQJ WKH SURJUDPPLQJ E\
UHOLHV RQ VXEMHFWLYH MXGJHPHQW YLUWXH RI DQ DXWRQRPRXV XQGHUVWDQGLQJ RI WKH
DQG PD\ QRW DFKLHYH WKH
SURJUDPPLQJ WDVN LQ ZKLFK HUURUV PD\ RFFXUSURJUDPPLQJ REMHFWLYHV
0LVVLQJ REMHFWV 2EMHFWV QHHGHG LQ WKH SURJUDPPLQJ
DUH QRW DGGHG WR WKH VFHQH 5HVXOWLQJ LQ PLVVLQJ FRGH
0LVVLQJ FRGH 0LVVLQJ FRGH LQ WKH SURJUDPPLQJ
SURFHVV GXH WR D ODFN RI V\VWHPDWLF GHFRPSRVLWLRQ RI
SURJUDPPLQJ WDVNV

7DVN GHFRPSRVLWLRQ WKH DELOLW\ WR GHFRPSRVH WKH/HYHO 2EMHFWLYH GHFRPSRVLWLRQ RI
6WUDWHJLHV SURJUDPPLQJ WDVNV EDVHG RQ PDLQ VXEWDVNV RI D SURJUDPPLQJ WDVN
2EMHFW GHFRPSRVLWLRQ WKH DELOLW\ WR SURJUDP DFWLRQ
VXEMHFWLYH MXGJHPHQW XVLQJFRPELQDWLRQV RI H[LVWLQJ FRGH FRGHV IRU HDFK REMHFW EDVHG RQ WKH REMHFWV WKDW DSSHDU
LQ WKH SURJUDPPLQJWR FRPSOHWH SURJUDPPLQJ
WDVNV/HYHO %H DEOH WR FRPSOHWHO\ DQDO\VH 6XEWDVN FRPELQDWLRQ WKH DELOLW\ WR EUHDN GRZQ WKH
RYHUDOO JRDO RI D SURJUDPPLQJ WDVN LQWR VHYHUDO WDVN
6WUDWHJLHV WKH REMHFWLYHV RI D WDVN LQ
EORFNV DQG FRPELQH WKHP XVLQJ H[LVWLQJ FRGH WR
SURJUDPPLQJ DQDO\VH WKHFRPSOHWH WKH SURJUDPPLQJUHODWLRQVKLSV EHWZHHQ VXESUREOHPV DQG XVH FRPELQDWLRQV 6HTXHQWLDO GHFRPSRVLWLRQ SUREOHP GHFRPSRVLWLRQ LQ
WKH RUGHU RI WKH SUREOHPV WKDW FDQ EH SURJUDPPHG IRURI SUHYLRXVO\ OHDUQHG FRGH
D SURJUDPPLQJ WDVN
EORFNV WR FRPSOHWH&RQGLWLRQDO GHFRPSRVLWLRQ WKH DELOLW\ WR GHFRPSRVH
SURJUDPPLQJ WDVNVSURJUDPPLQJ WDVNV DFFRUGLQJ WR WKH GLIIHUHQW VLWXDWLRQVRI WKH SURJUDPPLQJ SUREOHP/HYHO 7KH SURFHVV RI GHFRPSRVLQJ $XWRQRPRXV GHFRPSRVLWLRQ WKH DELOLW\ WRDXWRQRPRXVO\ LGHQWLI\ WKH PDLQ SUREOHPV LQ D
6WUDWHJLHV WDVN REMHFWLYHV HQDEOHV D
SURJUDPPLQJ WDVN RI WKHLU RZQ GHVLJQ DQG WKH
V\VWHPDWLF EUHDNGRZQ RIUHODWLRQVKLSV EHWZHHQ WKH YDULRXV VXESUREOHPVSURJUDPPLQJ REMHFWLYHV LQWR
2SWLPDO VXESUREOHP FRPELQDWLRQ 7KH DELOLW\ WR
VPDOOHU SUREOHPV LQ D
IRUP DQ RSWLPDO FRPELQDWLRQ RI VXESUREOHPV LQFRPELQDWLRQ RI SURFHVV DQG
FRQWHQW ZKLFK DUH HIIHFWLYHO\ SURJUDPPLQJ LQWR DQ HIILFLHQW VROXWLRQ WR WKH HQWLUHSURJUDPPLQJ WDVNFRPELQHG WR FRPSOHWH WKH
SURJUDPPLQJ WDVN

285

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Remembering the knowledge of experts
and novices in computer-supported
collaborative learning: A multinomial
processing tree approach
Oktay ÜLKERa*, Daniel BODEMERa
University of Duisburg Essen, Germany*oktay.uelker@uni-due.deaAbstract: Accurately assessing learning partners’ knowledge profiles improves
collaborative learning. Group awareness tools facilitate constructing such social
context knowledge during learning and the formation of learning partner models
retrievable afterwards. In this experimental study (N = 70), we investigated potential
schema effects in partner modeling: Participants were first provided with descriptionsof two learning partners (expert vs. novice of an area) and with their knowledge profiles
consisting of knowledge levels (high vs. low) regarding certain topics of this area. In amemory test, participants had to remember the specific knowledge levels of bothpartners. Partner modeling was analyzed with multinomial processing tree models, asthese models disentangle memory and guessing, which are often confounded whenschema effects on memory are examined. High and low knowledge levels were equallywell remembered for both partners. However, participants showed metacognitive
biases, expecting their memory to be better for high knowledge levels. Additionally,knowledge profile estimations revealed that while novices’ knowledge was estimatedaccurately, experts’ knowledge was overestimated. We discuss the results andpotential benefits of using multinomial processing tree models in the learning sciences
for and beyond analyzing schema effects in partner modeling.
Keywords: Computer-supported collaborative learning, Group awareness tools,
Partner modeling, Schemas, Multinomial processing tree modeling

1. Introduction
Collaborative learning is an omnipresent component of education in nearly all formal andinformal settings. However, effective social learning relies on learners being aware of theirlearning partners’ knowledge profiles (i.e., what they know or do not know about a learning
topic). If they are, targeted questions and explanations can be adapted to the learning partners
and facilitate knowledge construction (Erkens & Bodemer, 2019; Webb, 1989). Such cognitive
group awareness, i.e., the perception of competence or knowledge levels (Janssen &Bodemer, 2013), can be enhanced in computer-supported collaborative learning (CSCL)
settings: Group awareness tools (GATs) collect, transform, and present relevant knowledgerelated characteristics of learning partners (Bodemer et al., 2018) and learners can more
effortlessly perceive and use relevant information about their partners’ knowledge. However,
especially in (long-term) repeated collaboration, it is crucial to memorize knowledge-related
information about learning partners: Does the partner know a lot about a topic and thus canbe asked questions, or does the partner have low knowledge and might need explanations?
Indeed, these specific partner modeling (PM) processes, i.e., estimating others’ knowledge(Dillenbourg et al., 2016), can have beneficial effects on learning (Sangin et al., 2011).
Heterogeneous groups (regarding expertise) sometimes outperform homogeneousgroups because of useful interactions between learners, such as asking questions to an expert

286and providing explanations to a novice (Webb, 1989). In some cases, a learner might learn
collaboratively with both: A high knowledge partner (expert) and a low knowledge partner(novice) regarding an area (e.g., animals). Social schemas (e.g., “animal experts”) can help
us navigate through the world, but they can also be dangerous as they can be inaccurate(Schnaubert & Bodemer, 2022). It is reasonable to assume that an expert has rather highknowledge in many different content-specific topics. However, it is also possible that the expert
has low knowledge in some topics. Schemas carry the risk of overestimating partnercompetence and corresponding biases. In an experimental study, we explore potential
schema effects and investigate whether PM processes differ for expert and novice partners.
While, for example, Erkens and Bodemer (2019) found no differences in PM for high and low
knowledge partners, we further disentangle PM processes and use analysis methods, i.e.
multinomial processing tree (MPT) models, which are novel in CSCL research, but often used
to analyze schema effects in source memory or person memory research.

1.1 Expected partner modeling advantage for schema-inconsistent information
Source memory and person memory research suggest an inconsistency-effect: Information
which is inconsistent with a schema (here: partner expertise) is better remembered thanschema-consistent information (e.g., Bell et al., 2015). For example, it is better remembered
when trustworthy appearing persons cheat or untrustworthy appearing persons cooperate(Mieth et al., 2021). Here, we test whether such results can be transferred to PM processes
when specific knowledge levels of experts and novices must be remembered. Biased
modeling of learning partners’ knowledge levels could hinder efficient knowledge exchange
processes in collaborative learning, such as asking questions about the right topics or giving
explanations adjusted to partners’ low-level topics. Based on the inconsistency-effect, we
expect that memory for low knowledge levels of an expert should be more accurate thanmemory for high knowledge levels (H1a) and memory for high knowledge levels of a novice
should be more accurate than memory for low knowledge levels (H1b).

1.2 Expected metacognitive beliefs of higher schema-consistent modeling accuracy
Beyond actual memory processes, we explore participants’ metacognitive beliefs regarding
their PM accuracy, as these beliefs themselves can influence behavior (Schaper & Bayen,2023). In contrast to the inconsistency-effect observed in actual memory performance, whenpeople judge their own memory performances, schema-consistent effects can be found:People assume their memory to be better for schema-consistent information than for schemainconsistent information. For example, Mieth et al. (2021) have shown that while cooperation
by untrustworthy people and cheating by trustworthy people is better remembered, people
assume their memory to be better for the opposite. As these metacognitive illusions can leadto poor regulation and control processes, we considered metacognitive assumptions inaddition to actual memory performances. Based on metacognitive illusions in source memory
and person memory, we anticipate schema-consistent assumptions of learners: Participants’
metacognitive assumptions of their own PM accuracy for an expert should be higher for highknowledge than for low knowledge (H2a), and for a novice partner, assumptions of own PM
accuracy should be higher for low knowledge than for high knowledge (H2b).

1.3 Exploratory analysis and study aim
We will also exploratively examine global PM accuracy: We tested the accuracy of perceived
knowledge profiles regarding the amount of high and low knowledge topics of both partners.
This study investigates potential schema effects in PM processes using MPT models,accurately assessing (schema driven) memory for different knowledge levels by disentangling
memory and guessing processes (Bröder & Meiser, 2007). The novel use of MPT models in
PM processes and combining paradigms used in different research fields can enrich CSCL
research. This consequently enables a holistic view on long-term effects of tool usage in CSCL
to better find instructions beneficial for learners and theory driven implications for tool design.

287

2. Methods
N = 70 participants took part in the experimental study. Some descriptive data and dependent
variables have some missing data, resulting in different degrees of freedom. The age range(N = 67) was between 18 and 40 years (M = 22.81, SD = 4.40), and the sample was a student
sample (N = 69, 94.20% students, 49 female, 20 male). We used a 2 × 2 within-subject design
with the factors partner schema induced by providing expertise labels and descriptions (expert
vs. novice) and partner knowledge level for specific learning topics (high vs. low).
The procedure is illustrated in Figure 1. Participants were first informed that two
persons would be presented with information about their knowledge of certain animals. Also,
participants would (allegedly) have to read texts about certain animals, learn collaboratively
with the presented persons, and finally take a knowledge test. Next, descriptions about thepartners (expert vs. novice) followed. Both were described as a 24-year-old man: While theexpert was described as a zoology student with a passion for animals and who is seen as anexpert regarding animals, the novice was described as a history student without pets, who isseen a novice regarding animals. Both partners’ presentation order was counterbalanced.
Schemaactivation

Partner model
formation

Descriptions oflearning partners(expert and novice)

Display of knowledge
profiles (high and low
levels)

Metacognition
Self-assessedassumptions ofpartner modelingaccuracy

Partner model
retrieval

Knowledge
profile retrieval

Retrieval of specificknowledge levels ofboth partners

Estimations ofknowledge profiles ofboth partners

Figure 1. Schematic illustration of the key phases of the experiment.

In the partner model formation phase, participants were informed that both partners
conducted a test and that their knowledge levels about specific animals would be presented(high vs. low). In a norming study, N = 14 participants rated 160 animals regarding theirknowledge about them. We chose 80 animals with moderate ratings. From this pool, 20
animals were ascribed to the expert and 20 to the novice under different randomization
conditions. On separate pages for each partner, a GAT presented knowledge profiles withknowledge levels about 20 animals in a table. Animals were displayed on the right side with
their according knowledge level on the left side, with high levels shown in green boxes andlow levels in white boxes (Figure 2a). Participants had 3 to 5 minutes to view the knowledge
profile of each partner. While the expert’s knowledge profile consisted of 12 high and 8 low
knowledge levels, the novice’s pattern was reversed to ensure credible schemas.
Knowledge level

AnimalOtter
Buffalo
Hawk

(a)

Sloth

high

low

new

Penguin

high

low

new

Buffalo

high

low

new

(b)

Figure 2. Schematic illustrations of the partner model formation (a) and partner model retrieval phases (b).

Participants next provided metacognitive assumptions of their perceived modelingaccuracy. On scales from 0% to 100%, participants indicated for each of the 4 categories(expert high, expert low, novice high, novice low) for how many of the remembered topics thecorrect knowledge level (high, low) was memorized. After providing demographic information,
the partner model retrieval phase started (Figure 2b). Here, for each partner the 20 animals of
the partner model formation phase were presented intermixed with 20 new animals. On
separate pages for each partner, participants had to remember the information of the GAT in
the learning phase: Participants indicated whether each animal was associated with highknowledge, low knowledge, or was new. On the final page, participants provided knowledge
profile estimations (global PM): For both partners, the proportion of high and low knowledge
levels had to be indicated by sharing 100% between both categories (“Please indicate for whatpercentage of the animals the expert and novice had high and low knowledge levels”).

288

3. Results
3.1 Multinomial processing tree models: Schema effects on partner modeling (H1)
When memory for context information of items (here, high vs. low knowledge for certain topics)
is assessed, especially in contexts with schemas (expert vs. novice partners), analyses basedon hit and false-alarm rates can lead to the conclusion that schema-consistent information canbe memorized better due to guessing biases (Bröder & Meiser, 2007). Multinomial processing
tree (MPT) models offer a solution: Based on observed category frequencies (e.g., number of“high” responses to high knowledge topics), these models disentangle different memory andguessing processes and enable probability estimations of these processes (for an overview of
MPT models, see Erdfelder et al., 2009). Here, we adapted the two-high threshold MPT modelof source monitoring (Bayen et al., 1996) for our study purposes (Figure 3) to assess (schema
effects on) PM in collaborative learning. The model illustrates that combinations of different
cognitive processes can lead to the same response in the partner model retrieval phase. For
example, the response “high” to an actual high knowledge topic may be based on correct
memory processes (DHigh × dHigh) or guessing processes, e.g., DHigh × (1 – dHigh) × a.
diDi

1 − diTopic with original
knowledge level ib
1 − Di1−bia“High”

1−a

“Low”

g

“High”

1−g

“Low”
“New”

Figure 3. Multinomial processing tree model of source monitoring (Bayen et al., 1996), adapted for our study purposes with two
sets (for the expert and novice). Rectangle on the left side represents the type of presented topic in the partner model retrieval
phase. Index i denotes the associated knowledge level regarding that topic, i ‫{ א‬high, low}. Rectangles on the right side represent
the possible answers of participants. Letters along the links represent probabilities of cognitive processes: D = Recognizing thata topic was old (i.e., presented before); d = Remembering the associated knowledge level of an old topic (specific partnermodeling); b = Guessing that a topic was old; a, g = Guessing that a recognized (a) or unrecognized (g) topic was associatedwith a high level. The processing tree for new topics follows a similar principle (for an example, see Bayen et al., 1996).

To use our model, an identifiable base model with certain parameter restrictions mustfirst be found. None of the identifiable base models (see Bayen et al., 1996, Figure 4) fit ourdata, models 5: G²(2) > 31.76, p < .001, model 4: G²(4) = 277.18, p < .001. We thus had tofind a base model which fits the data with restrictions across both conditions. A model with therestrictions DLow_Expert = DNew_Expert; DLow_Novice = DNew_Novice; dHigh_Expert = dLow_Expert = dHigh_Novice =dLow_Novice shows good fit, G²(1) = 1.19, p = .276. The restriction of the PM-parameters (d = .76,
SE = 0.03) inherently means that specific PM does not differ within and between both partners:
High and low knowledge of experts and novices seem to be equally well modelled.
Consequently, the hypotheses H1a and H1b cannot be supported by the data: Our findings
do not provide evidence for an inconsistency effect in PM. Participants demonstrated relatively
accurate PM abilities with a 76% probability of remembering the knowledge levels.

3.2 Schema effects on metacognitive assumptions of partner modeling accuracy (H2)A 2 × 2 repeated-measures ANOVA (descriptive data in Figure 4a) revealed no significant
main effect of partner schema on metacognitive assumptions, F(1, 67) = 0.85, p = .360,
ηp² = .01. There was, however, a significant main effect of knowledge level, F(1, 67) = 56.74,
p < .001, ηp² = .46. Although actual memory performances between high and low knowledge
levels did not differ (see section 3.1), participants assumed that they remembered highknowledge better than low knowledge. The ANOVA further revealed a significant interaction
effect, F(1, 67) = 13.32, p < .001, ηp² = .17 (all follow-up pairwise comparisons were significant,t(67) > 2.78, pHolm < .009): Participants assumed better memory for high knowledge comparedto low knowledge both for the expert (contradicting H2a) and the novice (supporting H2b), but
the difference was greater for novice partners than for expert partners.

289

Perceived knowledge
profilesMetacognitive
assumptions ofmodeling accuracy

Knowledge
level
High
Low

.60
.40
.20
.00

ExpertNovicePartner schema
(a)

1.00
.80

Knowledge
level

.60

High
Low

.40
.20

.00

ExpertNovicePartner schema(b)

Figure 4. (a) Mean metacognitive assumptions of own memory accuracy, all SE = 0.03, and (b) perceived knowledge profiles asestimated percentages of high and low knowledge topics (SEExpert = 0.05, SENovice = 0.01). Note that the expert’s knowledge profile
consisted of 60% high knowledge topics and the novice’s knowledge profile consisted of 40% high knowledge topics.

3.3 Global partner modeling accuracy: Perceived knowledge profiles (explorative)
We further tested global PM accuracy (Figure 4b). One sample t-tests revealed that theexpert’s knowledge was overestimated: Mean estimations of the number of topics with highknowledge were higher than 60%, t(65) = 3.08, p = .003, d = 0.38. However, the proportion ofhigh and low knowledge topics of the novice was accurately estimated, t(65) = 1.00, p = .319,
d = 0.12. The novice’s knowledge was only descriptively overestimated.

4. Discussion
Group awareness tools (GATs) provide knowledge related information about learning partners
to support effective CSCL processes. Sometimes, knowledge related information needs to beremembered in retrospect (partner modeling, PM), both in short- and long-term collaborations.
We investigated whether learners model knowledge of experts and novices differently,
predicting a memory advantage for unexpected information (e.g., knowledge gaps of experts).
However, we could not observe such an inconsistency-effect in PM: High and low knowledge
were equally well encoded for expert and novice partners (contradicting H1). For educational
practice, this implies that in short-term collaboration—for example in heterogeneous groups
with high and low knowledge students—such PM biases do not impair collaborative learning
processes but allow for asking and answering questions properly adapted to the learning
partners. However, other studies found the inconsistency effect. We can thus assume that
there may be conditions and moderating variables affecting the results. For example,
Ehrenberg and Klauer (2005) found the inconsistency-effect primarily in contexts with higher
cognitive load during encoding and longer retention-intervals, which are given in manyeducational settings. Consequently, manipulations of collaboration duration could be
employed to test whether the inconsistency effect is generally absent in PM, or whether our
results are limited to short-term collaborative learning situations.
We further considered metacognitive assumptions, as wrong metacognitive
assumptions can also influence control processes and study behavior (Schaper et al., 2023).
Here, learners generally assumed memory for high knowledge to be better than for low
knowledge (supporting H2a, but contradicting H2b). When students mistakenly assume thatthey would remember high partner knowledge better than low knowledge, they might ask only
about their partners’ knowledge gaps, even though asking for high knowledge topics would
give them the opportunity to immediately access the best explanations. Metacognitive prompts
can enhance students’ metacognitive awareness when collaborating (Teng, 2022). Integrating
prompts into GATs can assist students recognizing the metacognitive illusion of alleged bettermodeling of high levels, thereby countering the illusion and fostering better help-seeking.
Exploratively examining knowledge profile estimations (global PM) revealed that
novices’ knowledge was estimated accurately, which is desirable, as both, over- andunderestimations of novices by more knowledgeable partners can impair novices’ learning(Wittwer et al., 2008). Experts’ knowledge, however, was overestimated, which might pose a290danger: Students might over-trust experts and their explanations, without adequately verifying
their produced learning contents. Again, metacognitive prompts or cues—delivered throughguidance from educators or integrated into GATs—could potentially make learners aware ofpotential overestimations and aid them in critically questioning their assessments.
From a methodological perspective, MPT models can disentangle memory andguessing processes and thus offer diverse applications in the field of learning sciences. They
have the potential to reveal cognitive biases beyond (potential) schema effects in PM: For
example, learners might project their own knowledge onto peers when failing to remember
their knowledge (Nickerson, 1999). MPT models could test this assumption by assessing
whether the guessing parameters of the model differ based on students’ prior knowledge.
Accurately disentangling different PM processes and biases can help to derive sound
implications for instructors and tool designers. While this study was a first attempt to use MPTmodels to test the impact of collaboration tools on PM processes, further drawing on methods,
paradigms, and findings of research areas such as source memory holds the potential toenrich CSCL research, ultimately benefiting learners.

Acknowledgements
The authors thank Jamie-Lee Tripke for data collection in the course of her Bachelor’s Thesis.

References
Bayen, U. J., Murnane, K., & Erdfelder, E. (1996). Source Discrimination, Item Detection, and Multinomial Models of Source
Monitoring. Journal of experimental psychology: Learning, Memory, and Cognition, 22(1), 197–215.
https://doi.org/10.1037/0278-7393.22.1.197
Bell, R., Mieth, L., & Buchner, A. (2015). Appearance-based first impressions and person memory. Journal of Experimental
Psychology: Learning, Memory, and Cognition, 41(2), 456–472. https://doi.org/10.1037/xlm0000034
Bodemer, D., Janssen, J., & Schnaubert, L. (2018). Group Awareness Tools for Computer-Supported Collaborative Learning. In
F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), International Handbook of the Learning Sciences(pp. 351–358). Routledge. https://doi.org/10.4324/9781315617572-34
Bröder, A., & Meiser, T. (2007). Measuring Source Memory. Zeitschrift Für Psychologie / Journal of Psychology, 215(1), 52–60.
https://doi.org/10.1027/0044-3409.215.1.52
Dillenbourg, P., Lemaignan, S., Sangin, M., Nova, N., & Molinari, G. (2016). The symmetry of partner modeling. International
Journal of Computer-Supported Collaborative Learning, 11(2), 227–253. https://doi.org/10.1007/s11412-016-9235-5
Ehrenberg, K., & Klauer, K. C. (2005). Flexible use of source information: Processing components of the inconsistency effect inpersonmemory.
Journalof
Experimental
Social
Psychology,
41(4),
369–387.
https://doi.org/10.1016/j.jesp.2004.08.001
Erdfelder, E., Auer, T.-S., Hilbig, B. E., Aßfalg, A., Moshagen, M., & Nadarevic, L. (2009). Multinomial Processing Tree Models:
A Review of the Literature. Zeitschrift Für Psychologie / Journal of Psychology, 217(3), 108–124.
https://doi.org/10.1027/0044-3409.217.3.108
Erkens, M., & Bodemer, D. (2019). Improving collaborative learning: Guiding knowledge exchange through the provision of
information about learning partners and learning contents. Computers & Education, 128, 452–472.
https://doi.org/10.1016/j.compedu.2018.10.009
Janssen, J., & Bodemer, D. (2013). Coordinated Computer-Supported Collaborative Learning: Awareness and Awareness Tools.
Educational Psychologist, 48(1), 40–55. https://doi.org/10.1080/00461520.2012.749153
Mieth, L., Schaper, M. L., Kuhlmann, B. G., & Bell, R. (2021). Memory and metamemory for social interactions: Evidence for ametamemory expectancy illusion. Memory & Cognition, 49(1), 14–31. https://doi.org/10.3758/s13421-020-01071-z
Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know: Imputing one's own knowledge toothers. Psychological bulletin, 125(6), 737–759. https://doi.org/10.1037/0033-2909.125.6.737
Sangin, M., Molinari, G., Nüssli, M.-A., & Dillenbourg, P. (2011). Facilitating peer knowledge modeling: Effects of a knowledge
awareness tool on collaborative learning outcomes and processes. Computers in Human Behavior, 27(3), 1059–1067.
https://doi.org/10.1016/j.chb.2010.05.032
Schaper, M. L., Bayen, U. J., & Hey, C. V. (2023). Remedying the Metamemory Expectancy Illusion in Source Monitoring: Are
there Effects on Restudy Choices and Source Memory?. Metacognition and Learning, 18(1), 55–80.
https://doi.org/10.1007/s11409-022-09312-z
Schnaubert, L., & Bodemer, D. (2022). Group awareness and regulation in computer-supported collaborative learning.
International Journal of Computer-Supported Collaborative Learning, 17(1), 11–38. https://doi.org/10.1007/s11412022-09361-1
Teng, M. F. (2022). Effects of individual and group metacognitive prompts on tertiary-level students’ metacognitive awarenessand writing outcomes. The Asia-Pacific Education Researcher, 31(5), 601–612. https://doi.org/10.1007/s40299-02100611-8
Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Educational Research, 13(1), 21–39.
https://doi.org/10.1016/0883-0355(89)90014-1
Wittwer, J., Nückles, M., & Renkl, A. (2008). Is underestimation less detrimental than overestimation? The impact of experts’beliefs about a layperson’s knowledge on learning and question asking. Instructional Science, 36(1), 27–52.
https://doi.org/10.1007/s11251-007-9021-x

291

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring Group Formation Strategies in
Computer-Supported Collaborative
Learning: A Systematic Review
Jiamin TANG, Huihan ZHOU, Yajing TAN & Guang CHEN*
Faculty of Education, Beijing Normal University, China*guang@bnu.edu.cn
Abstract: In this study, three independent reviewers conducted a systematic review ofarticles on group formation strategies in computer-supported collaborative learning(CSCL) and synthesized and structuralized the involved grouping attributes andevaluation indicators with some other features. The study aims to contribute to efficientand effective group formation in educational practices in CSCL environments.
Keywords: Group formation strategies, computer-supported collaborative learning,systematic review

1. Introduction
With the proliferation of information and communication technologies (ICTs) and the farreaching pandemic, CSCL has further developed in education, whose benefits whenappropriately applied have been validated by substantial studies (Costaguta, 2015).
The group composition affects the learning outcomes and the group development
lifecycle, which are crucial in fostering an effective and efficient CSCL setting (Krouska et al.,2019). Thus, exploring effective grouping strategies in the CSCL environment and theirimpact on CSCL becomes practically necessary.
Previous studies have focused on 2004-2018 articles about grouping strategies,
while literature reviews evaluating the effects of those strategies are insufficient.
Furthermore, although online learning has been integrated into pedagogical practices with
unprecedented depth and scope since 2019 due to the pandemic, the group formation
strategies in CSCL from 2019-2023 have not been deeply examined. Therefore, to fill thegaps and compare trends before and after the pandemic, the researchers searched articles
on group formation strategies in CSCL in the past 12 years. Due to the time limitation andthe concerns for comprehensiveness and credibility, Web of Science was chosen as the sole
database for search. Hence, our research questions are as follows:
Q1: What group formation strategies have been adopted in the CSCL environment?
Q2: What has been the influence of those strategies on CSCL?

2. Method
In this study, the English-language journal articles or conference articles of group formation
strategy in Computer-Supported Collaborative Learning that published from 2012 to 2023
were searched in the Web of Science database, and The Preferred Reporting Items for
Systematic Reviews and Meta-Analyses (PRISMA) framework and snowballing were theprimary guide for this study. The three reviewers worked collectively to discuss the criteria ofarticles' inclusion and exclusion and independently coded articles, reaching a high level ofconsistency (Kendall’s W = 0.81), and 17 articles were finally included. For the 17 final
included articles, this study analyzed the basic information, counted the effect of groupformation strategy, and used Theme analysis method analyzed the group formation strategy(Virginia et al., 2006).

292

3. Result and Discussion
3.1 Descriptive Information of the Literature
As for the time distribution (Fig 1. a), the number of published articles remained stable from2012 to 2019 and showed a growing trend from 2019 to 2022. As for the country distribution(Fig 1. b), Chinese researchers paid more attention to research in this field. In terms of the
distribution of the education stage (Fig 1. c), the research on group formation strategies in
CSCL was mainly focused on higher education (82%). As for the subject distribution (Fig 1.
d), group formation strategies in CSCL mainly focused on STEM disciplines (72%). As forthe sample size, the number of participants presented a state of polarization, with thenumber of participants less than 50 and more than 200 accounting for the majority.

(a)

(b)
(c)
(d)
Figure 1. The descriptive information graphs of the literatures

3.2 Group Formation Strategy
The initial codes had been distilled from the group formation strategies of the included
articles and combined into an initial thematic web map (shown in Figure 2. a), describing the
involved strategies from eight thematic dimensions, such as "Student Learning Performance”
or "Personality." These themes are interdependent with links between them, such as "Spatial
Competence" in "Intelligence" and "Gender" in "Demography Variable" (Zhong et al., 2023).
Although differences and commonalities coexist in these eight themes, they were all retained
temporarily before the sequent reviews.

漏a漐

(b)

(c)

Figure 2. The thematic web map
Next, the individual themes and the corresponding codes under the themes wereverified and proofread, which formed Figure 2. b. The researcher incorporated “Personality"and "Intelligence" into the theme "Intrinsic Attributes of Students"; while “Learning Interest”
and “Learning Style” and “Learning Performance” were included in the theme "Learning
Variables". In addition, those highly similar codes, such as “Academic Performance” and“Learning Performance”, were merged into “Student Learning Performance”.
The final thematic network diagram was developed after the themes’ definitions andthe ultimate optimization (Figure 2. c). The researchers found that those strategies could be
divided into explicit attributes and implicit attributes according to the difficulty of theirmeasurement. Explicit attributes are easy to obtain and measure, while implicit attributes

293require specific measuring tools to obtain results which may have more options and maychange with different measuring tools. Additionally, the demography variable belonging to
explicit attributes can be mixed with the learning variable or the intrinsic attributes students
belonging to implicit attributes together as group formation strategy rather than just alone.

3.3 The Evaluation of Group Formation Strategy
Most of the studies evaluated group formation strategies with different involved attributes
using a mixture of indicators, some researchers reaching a consensus conclusion on theimpact of the same group formation attributes, and some not. For example, As for the
influence of gender as a group formation attribute, the researchers reached a consensus
conclusion on the role of gender composition in groups regarding its influence on learners'
attitudes. In addition, no consensus conclusion has been reached on the influence oflearning style and achievement level as grouping attributes, which requires further research.
Detailed information can be obtained from figure 3.

Figure 3. The Involved Attributes of Group Formation Strategies and Evaluation Indicators

4. Conclusion
Based on a review of group formation strategies in the context of computer-supported
collaborative learning from 2012 to 2023, this study constructed a theme map displaying theclassification of the attributes involved in group formation strategies, analyzed the influenceof the group formation strategies. The limitation of this study is that only one database wasconsulted for the data source of this study. Therefore, the data source should be expanded
to include more literature to provide a broader and more holistic perspective in futurestudies.

References
Costaguta, R. (2015). Algorithms and Machine Learning Techniques in Collaborative Group
Formation. Lecture Notes in Computer Science, 249–258.
Krouska, A., Troussas, C., & Virvou, M. (2019). Applying genetic algorithms for student grouping incollaborative learning: A synthetic literature review. Intelligent Decision Technologies, 13(4),395–406.
Virginia Braun & Victoria Clarke.(2006).Using thematic analysis in psychology. Qualitative Researchin Psychology(2).
Zhong, Z., Wang, J., Deng, Y., Jin, S., Feng, S., & Li, R. (2023). Effects of external scripts
incorporating capabilities, roles and tasks on IVE’s collaborative learning. Education andInformation Technologies.

294

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Comparative Analysis on the Effects of
Cognitive Tools in Data Inquiry Cultivation
Hui ZHANG*, Bian WU, Yi-Ling HU & Yu-Jie XU
Department of Education Information Technology, East China normal University, China*51214108043@stu.ecnu.edu.cn
Abstract: The advent of the big data era has elevated the importance ofunderstanding, analyzing, and mining data, necessitating data science education withcognitive tools aiding data inquiry. However, limited appraisals exist, especiallyregarding diverse tools' impact on data inquiry cognitive aspects. This study addresses
this void by examining cognitive tool impact on data inquiry. The study collecteddiscourse and questionnaire data generated during online collaborative tasks.
Epistemic network analysis and difference testing unveiled cognitive pattern disparities,
learning motivation, cognitive load, and self-efficacy variations between groups. Blockbased group displayed robust cognitive connections in data understanding andpreparation. Text-based group focused on modeling and optimization iterations.
Motivation and load didn't differ significantly, yet block-based group showed higher selfefficacy. Study concludes by acknowledging limitations and suggesting future research
directions.

Keywords: Data inquiry, cognitive tools, CSCL, human-computer interaction

1. Introduction
Data inquiry is the process of extracting meaningful and valuable information from data,
including the identification of understandable patterns and relationships, as well as theconstruction of representative data models. In recent years, many researchers have provided
various perspectives on the connotative structure of data inquiry. Most of them perceive datainquiry as a collection of data skills, encompassing elements such as data collection, preprocessing, analysis, model building, and evaluation (Donoho 2017).
Various cognitive tools have emerged to support data practice and learning. These
tools can be categorized into three types: block-based, text-based programming, and menuselected (Bart et al. 2020). Block-based tools, such as RapidMiner and IBM SPSS Modeler,
offer encapsulated modular systems and user-friendly graphical interfaces, enabling quick
prototyping and validation of predictive models. They simplify the entire analysis process andprovide instant gratification. On the other hand, text-based programming tools like RStudioand Python offer greater flexibility and scalability. They allow users to explore algorithmic
details, create customized program algorithms, and offer system openness.
Block-based cognitive tools offer a potential resolution to the educational challenge ofdata inquiry. Their algorithmic intricacies are concealed, enabling learners to concentrate on
configuring and adapting predefined algorithms and parameters. By seamlessly connecting
modules through drag-and-drop actions, learners swiftly execute data inquiry, with a shallow
learning curve fostering confidence and problem-solving belief. Modular encapsulationdelegates computational tasks to cognitive tools, enabling learners to focus on problem solving,
expanding mental capacity and reducing cognitive load (Javadpour 2022). This phenomenon
is well-established in computational thinking development. Additionally, graphical cognitive
tools deliver instant interactive feedback via module interconnections, prompting immediate
data-inquiry knowledge reflection, potentially heightening motivation to learn.
However, there have been no studies exploring the differences in the effects of blockbased and traditional text-based cognitive tools on learners' data inquiry abilities. At the sametime, researchers have pointed out that future studies should explore whether certain data
science practices are influenced by tool affordance (Jiang and Kahn 2020). In order to better295

design teaching and learning based on block-based data science cognitive tools, clarify how
cognitive tools affect learning processes and knowledge construction, this study proposes thefollowing research questions:
1) What are the differences in the frequency distribution of data inquiry cognitiveelements between the block-based and the text-based tools in online collaboration activities?
2) What are the differences in the cognitive patterns of data inquiry between the blockbased and the text-based tools in online collaboration activities?
3) Are there any differences in learning motivation, cognitive load, and self-efficacy
between the block-based and the text-based tools?

2. Method
2.1 Research Context and ParticipantsThis empirical study was conducted during the Fall Semester of 2022-2023 at acomprehensive university in eastern China, focusing on the "Educational Data Mining" course.
Designed for senior college students pursuing careers in education, the study participants
were selected from two classes within the same university department, totaling 44 individuals- 10 males and 34 females. All participants had not received prior data mining instruction.
Collaborative learning was facilitated in groups of 3-4 learners. One class (16 students, 5
groups, average age 21) used text-based cognitive tools, while the other class (28 students,
9 groups, average age 23) employed block-based cognitive tools.

2.2 Research Process
The experimental procedure is illustrated in Figure 1. The study spanned 18 weeks,comprising a 14-week instructional phase dedicated to learners' acquisition of data inquiry
skills. Both classes were taught exactly the same by a veteran instructor with over a decadeof teaching experience. Subsequently, participants undertook data analysis projects during
the final 4 weeks. Upon course completion, all students were required to completequestionnaires assessing cognitive load, learning motivation, self-efficacy, and satisfaction.
The questionnaire completion process typically lasted around 10 minutes.

Figure 1. The flowchart of research experiment.
The two student cohorts employed distinct cognitive tools – block-based and text-based– for their data inquiry tasks. The block-based group utilized the RapidMiner platform, a robust
data mining tool characterized by its visual workflow interface, facilitating intuitive design,
execution, and assessment of diverse data mining tasks. Conversely, the text-based groupleveraged the RStudio platform, an interactive development environment based on the R
programming language. This platform encompasses a variety of functions, including a code
editor and data viewer, facilitating efficient organization and management of code and data.

296

2.3 Data Collection
In the project's final 4 weeks, each group conducted 5 online discussions lasting 1 to 3 hours,
recorded for analysis. The study employed adapted percentage scored questionnaires forlearning motivation (6 items, Cronbach's α = 0.848), cognitive load (2 items, Cronbach's α =0.836), and self-efficacy (3 items, Cronbach's α = 0.843). Learning motivation drew from Lau& Lee (2008), measuring knowledge mastery and grades on a five-point scale. Cognitive load,
adapted from Leppink et al. (2013), assessed internal and external load using a nine-point
scale. Self-efficacy, from Tsai et al. (2020), focused on data inquiry self-efficacy with a fivepoint Likert scale.

2.4 Code Framework
A coding framework was applied to analyze cognitive elements in data inquiry for frequency
and patterns. Adapted from Wirth & Hipp (2000), a recognized data mining model, the initialframework had 6 codes. Grounded theory guided manual analysis of pre-experimentconversation data, refining the framework. The final version contained 7 codes: planning,understanding, preparation, modeling, evaluation, technical inquiries, and tangential topics.
For code breakdown, meanings, and case examples, consult Table 1.
To ensure coding framework reliability, two coders independently processed arandomly selected 15% of session data, defining content analysis units as uninterrupted
student presentations. Results showed substantial agreement, Kappa score at 0.784. Coders
discussed discrepancies to enhance shared understanding and scheme interpretation.
Remaining data (8,615 sentences) was then independently coded, resulting in 5,986 coded
data points.
Table 1. Collaborative Conversation Data Coding Framework
Code
Plan

UnderstandPreparation

Modeling

Evaluation

Meaning
Develop plans to achieve thegoals. Review models andprocesses, issues that arise, andidentify areas for improvement in
subsequent work.
Understand project goals andrequirements, understand andinfer data context and types,
hidden trends, anomalies, patterns
in data, explore data distribution,correlations and trends.
Fixing data problems, converting
data formats, merging multiple
data sets, extracting new featuresfrom data, transforming features,
selecting some data rows andfeatures.
Find, search, introduce, and filter
algorithmic models that can beused, build models using cognitive
tools, understand the meaning ofmodel parameters, and change
parameters to improve model
performance.
Select evaluation metrics toassessmodel
performance,
calculate selected metrics using
software manipulation or code
programming, and understand

297

Cases
For the next step, we may have to continue toextract the features. I'm missing a descriptive
statistic for each type of question, such as thetotal value of the mcss type and its averagevalue, and time.
One is to determine whether he answered thequestion or not, and the other is to calculate thethreshold value. These are the two problems
that need to be solved in order to go ahead andmake predictions based on these.
We need to standardize the test dataset.
I roughly sieve a few features, you guys look at.
We can count the type of operation. We canalso add other columns, such as how manytimes to enter, how many times to exit, howmany times to draw.
Can you use generalized linear regression, but
we're not a continuum here.
There should be a template from when you
wrote the logistic regression before. Take it andchange it.
I adjusted the tree model, because the previous
1500 may overfitting, so I set it to 1000, may be
a little better.
I see logistic regression can convert
probabilistic output to labeled output, it can also
output precision confusion matrix, exact value.
How to use kappa for logistic regression?
AUC is still 0.3536, Kappa is still 0, very low.

Technology
problem
Irrelevant
topic

what the evaluation results
represent.
Only address technical issues thatarise during interaction with
RStudio and RapidMiner.
Distractions, or project related
emotional expressions.

If kappa is equal to 0, it means that the two
judgment results are caused by chance.
No matter how I access it, I cannot access itscore content. So, I'm stuck here, stuck for a longtime, stuck me for almost four or five hours.
R language is too disgusting.
I'm infected COVID-19 too.

2.5 Data Analysis
To address research question one, chi-square tests examined differences in frequency
distribution across the seven dimensions between the two groups. For research question two,
ENA method generated cognitive process patterns. Unit of analysis was cognitive tool typeand group number, with section size set at 4. For research question three, descriptive statistics
were initially performed. Data normality was assessed using Shapiro-Wilk, andhomoscedasticity with Levene's method. Normally distributed data underwent independent
sample t-tests, while non-normally distributed data were analyzed using Mann-Whitney U tests.

3. Results
3.1 What Are the Differences in the Frequency Distribution of Data Inquiry Cognitive
Elements Between the Block-Based and the Text-Based Tools?
A total of 3062 codes were generated in the Block-based group, and 3148 codes weregenerated in the text-based group. Specifically, the block-based group had a significantly
higher cognitive frequency than the text-based group in planning (χ2 = 11.239, p < 0.01) andunderstand (χ2 = 164.512, p < 0.01). Code frequencies for modeling (χ2 = 102.044, p < 0.01),
evaluation (χ2 = 88.256, p < 0.01), and irrelevant topics (χ2 = 83.195, p < 0.01) weresignificantly higher in the text-based group. See Table 2 for more detailed results.
Table 2. The Frequency Distribution of Cognitive Elements of Data Inquiry
Code
Plan
UnderstandPreparation
Modeling
Evaluation
Technology problem
Irrelevant problem
Total

block-based
(F)
(%)
274
8.95%
652
21.29%
1239
40.46%
268
8.75%
273
8.92%
147
4.80%
209
6.83%
3062
100%

text-based
(F)
(%)
190
6.04%
259
8.23%
1160
36.85%
514
16.33%
499
15.85%
117
3.72%
409
12.99%
3148
100%

χ2

p

11.239164.5120.034
102.04488.256
1.92
83.195

0.001
0
0.866
0
0
0.169
0

3.2 What Are the Differences in the Cognitive Patterns of Data Inquiry Between the
Block-Based and the Text-Based Tools?
The connection lines between the nodes represent the relationship between the data inquiry
elements, and the thickness of the connection line reflects the number of times the two nodes
appear simultaneously in the students' utterances. By observing the ENA diagrams andcomparison plot of two groups, as shown in Figure 2, we found that the block-based group's
plan-understand, understand-preparation are highly correlated. In the text-based group,
evaluation frequently co-occurs with preparation, modeling, and technical problem. In addition
to this, preparation is also more highly relevant to modeling and technical problem.

298

Figure 2. Comparison plot of block-based group and text-based group (left), ENA
network of block-based group (middle), ENA network of text-based group (right).

3.3 Are There Any Differences in Learning Motivation, Cognitive Load, and SelfEfficacy Between the Block-Based and the Text-Based Tools?
The Block-based group exhibited a mean learning motivation of 76.000, while the text-based
group had 72.667. T-test results (t = 0.814, p = 0.427 > 0.05) indicated no significant difference
between them. Comparing cognitive load, the block-based group averaged 81.482, and thetext-based, 83.704. Independent sample t-test (t = -0.787, p = 0.438 > 0.05) revealed bothgroups had higher cognitive load, but insignificantly different. However, self-efficacy showedsignificance, U = 66.000, p = 0.047 < 0.05, with block-based group at 72.889, and text-based
group at 63.111 mean. See Table 3 for detailed results.
Table 3. Difference Test of Learning Motivation, Cognitive Load and Self-Efficacy

Learning motivation
Cognitive load
Self-efficacy

Block-based
MeanSD
76.00014.97181.482
7.987
72.889
7.753

Text-based
MeanSD
72.6677.473
83.7047.753
63.11114.224T/U

p-value

0.814 (T)-0.787 (T)66.000 (U)

0.427
0.438
0.047

4. Discussion and Conclusion
This study extensively examined how distinct cognitive tools influence students' data inquiry
skills by analyzing data inquiry patterns within block-based and text-based groups. Both tooltypes exhibit distinct merits in enhancing data comprehension and modeling.

4.1 Block-Based Cognitive Tools Make It Easier for Beginners to Enter Data Inquiry
The block-based group exhibited significantly higher cognitive frequencies for planning andunderstanding than the text-based group. Furthermore, ENA network connections, particularly
between understand and preparation, were markedly stronger in the block-based group. Upon
comprehending task objectives and data meanings, the block-based group strategized
converting complex datasets into a streamlined format using the pivot module operator in
RapidMiner. This transformation enhanced data comprehension, revealing trends likeincomplete question responses, which the original dataset obscured. As understanding
deepened, the group refined their inquiry plans. This iterative process, underpinned by solid
data contextualization, facilitates pattern identification (Wilkerson, Lanouette, and Shareff
2021), emphasizing data inquiry's importance. Conversely, the text-based group encounteredcoding challenges, causing disruption between planning, understanding, and preparation. This
resulted in decreased self-efficacy. Similar findings were noted by Price and Barnes (2015) incomputational thinking comparisons. Novices, facing difficulties, may cease autonomous
learning after few attempts (Tawfik, Payne, and Olney 2022).

299

4.2 Text-Based Group Used External Resources for In-Depth ModelingText-based group exhibited significantly higher frequencies of modeling and evaluation thanblock-based group. In-depth analysis revealed text-based group's model selection andparameter challenges, necessitating external resource usage (blogs, papers). This processenhanced their model understanding and internal models, aiding subsequent featurescreening goals, as supported by Oh & Oh (2011). In contrast, block-based group relied on
tool-provided modules within its interface, missing external learning opportunities. Althoughattempting more models, they lacked direction and understanding. Instructors must ensure
resource exploration beyond tool for less reliance on blind attempts.
Cognitive load and learning motivation didn't significantly differ. However, working
memory constraints and simultaneous module identification hinder effective attention
allocation to modeling and optimization. Text-based group efficiently treats multiline text
codes, reducing element interactivity (Chen, Kalyuga, and Sweller 2017) and cognitive load.
Encapsulating multiple modules into one may ease resource consumption.

Acknowledgements
This paper was supported by the East China Normal University graduate student international
conference special fund.

References
Bart, Austin Cory et al. 2020. “Design and Evaluation of a Block-Based Environment with a Data Science
Context.” IEEE Transactions on Emerging Topics in Computing 8(1): 182–92.
Chen, Ouhao, Slava Kalyuga, and John Sweller. 2017. “The Expertise Reversal Effect Is a Variant of
the More General Element Interactivity Effect.” Educational Psychology Review 29(2): 393–405.
Donoho, David. 2017. “50 Years of Data Science.” Journal of Computational and Graphical Statistics
26(4): 745–66.
Javadpour, Leili. 2022. “Using RapidMiner for Executing Queries and Visualization in a Traditional
Database Course.” Journal of Education for Business 97(4): 247–52.
Jiang, Shiyan, and Jennifer Kahn. 2020. “Data Wrangling Practices and Collaborative Interactions with
Aggregated Data.” International Journal of Computer-Supported Collaborative Learning 15(3):257–81.
Lau, Kit-Ling., and John C. K. Lee. 2008. “Validation of a Chinese Achievement Goal Orientation
Questionnaire.” British Journal of Educational Psychology 78(2): 331–53.
Leppink, Jimmie et al. 2013. “Development of an Instrument for Measuring Different Types of Cognitive
Load.” Behavior Research Methods 45(4): 1058–72.
Oh, Phil Seok, and Sung Jin Oh. 2011. “What Teachers of Science Need to Know about Models: An
Overview.” International Journal of Science Education 33(8): 1109–30.
Price, Thomas W., and Tiffany Barnes. 2015. “Comparing Textual and Block Interfaces in a Novice
Programming Environment.” In Proceedings of the Eleventh Annual International Conference on
International Computing Education Research, Omaha Nebraska USA: ACM, 91–99.
https://dl.acm.org/doi/10.1145/2787622.2787712 (October 12, 2022).
Tawfik, Andrew A., Linda Payne, and Andrew M. Olney. 2022. “Scaffolding Computational Thinking
Through Block Coding: A Learner Experience Design Study.” Technology, Knowledge andLearning. https://link.springer.com/10.1007/s10758-022-09636-4 (May 19, 2023).
Tsai, Chia-Lin, Moon-Heum Cho, Rose Marra, and Demei Shen. 2020. “The Self-Efficacy Questionnaire
for Online Learning (SeQoL).” Distance Education 41(4): 472–89.
Wilkerson, Michelle Hoda, Kathryn Lanouette, and Rebecca L. Shareff. 2021. “Exploring Variability
during Data Preparation: A Way to Connect Data, Chance, and Context When Working with
Complex Public Datasets.” Mathematical Thinking and Learning: 1–19.
Wirth, Rüdiger, and Jochen Hipp. 2000. “CRISP-DM: Towards a Standard Process Model for Data
Mining.” In Proceedings of the 4th International Conference on the Practical Applications ofKnowledge Discovery and Data Mining, Manchester, 29–39.

300

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Argumentative Knowledge Construction
and Certainty Navigation: A Comparison
between Individual and Group Work
Wenli Chena, Eng Eng Nga*, Su Guoa, Su Junzhua, Xinyi Lib, Siew Cheng Aileen Chaia
and Qianru Lyuaa
National Institute of Education, Nanyang Technological University, SingaporebBeijing Normal University, China*caryn.ng@nie.edu.sg
Abstract: This study investigated (the extent to which levels of certainty impacted theargumentative knowledge construction in individual work and group work.
Argumentative knowledge construction has been characterized into simple claims,
grounds, qualifiers, counterarguments, and integrated replies to illustrate thecomponents of argumentation and nature of resolving conflicts in argumentation wherecertainty levels have been divided into uncertain, neutral, and certain. Findings showedthat individual and group work differed significantly in terms of levels of certainty forsimple arguments and counterarguments. Study implications were discussed.
Keywords: Knowledge construction, collaborative argumentation, individual writing,group work, argumentative writing

1. Introduction and Literature Review
In Computer-Supported Collaborative Argumentation (CSCA) research, the argumentativeknowledge construction model seeks to understand how social interactions and argumentation
processes contribute to knowledge construction and learning (Nguyen & Young, 2022).
Through engaging in social interactions and argumentation with others, individuals actively
navigate uncertainty by constructing their understanding of a topic, negotiating meaning, andrefining their ideas. Existing research on the area of knowledge construction has thus far
centered on facilitating knowledge construction through placing the individual in the context ofthe community around him or her and building on discourse moves and reasoning to constructarguments (Nguyen & Young, 2022). Usage of words indicating certainty has been an area of
focus in social interactions as they navigate between advancing their own knowledge andresolving conflicts with others (Tabak & Baumgartner, 2004; Tan et al., 2016). Uncertainty inargumentation in navigating conflict resolution is also demonstrated by using more hedging
words (“possibly”, "maybe", and “potentially” etc.) to question and evaluate each other’s claims
and evidence (Martinez-Maldonado et al., 2013). However, hedges such as "may" or"perhaps" show a tentativeness to commit i.e., uncertainty. Word such as "actually", showedstronger commitment. Our study investigated the extent to which levels of certainty used would
impact argumentative knowledge construction when comparing individual work and groupwork.
In individual writing, students engage with deep learning of the content, clarify theirthoughts and ideas, identify gaps in understanding, develop soft skills, and explore newperspectives on a topic (Harney et al., 2017; Latifi et al., 2020, 2021; Noroozi et al., 2022;
Storch, 2005; Valero Haro et al., 2020; Weinberger & Fischer, 2006). Collaborative writing
happens when group members contribute their perspectives and ideas to the group's
collective efforts, working together to achieve learning outcomes while honing technologicalliteracy, problem-solving, critical thinking, innovation, shared responsibility, belongingness
within the group, better mastery of the concepts, interrelationships among concepts, andintegration of new concepts with prior knowledge (Asterhan & Schwarz, 2007; (Clark &Sampson, 2007; Mayweg-Paus et al., 2021; Roberts, 2004).

301

Argumentation fosters higher-order cognitive abilities as students dissent, evaluate,
expand, and synergize claims, evidence, and counterarguments to reach conclusions (Harney
et al., 2017; Latifi et al., 2020, 2021; Valero Haro et al., 2020). Such higher-order thinking skills
involve knowledge construction, concept integration, critical thinking, and reflective judgments(Harney et al., 2017; King, 2002; Lu & Zhang, 2013). Students move from individual
work/argumentation, preparing themselves from individually constructing their arguments tocollaborative argumentation. Individuals move from personal viewpoints to resolving conflicts
and converging at a joint and shared knowledge in argumentation. Through social interactions(Hou & Wu, 2011), students co-construct knowledge as interpersonal interactions connect
academic discussions and argumentative task coordination (Hou & Wu, 2011).
The model for managing uncertainty developed by Chen et al., (2019) postulated thatlearners go through the first stage of (1) raising uncertainty, where they ask questions to clarify,
critique, and challenge existing gaps in individual knowledge about a topic before moving onto the next stage of (2) maintaining uncertainty where learners seek to find solutions andevidence along with garnering varied perspectives (Nguyen & Young, 2022). Finally, learners
reduce uncertainty by reconciling current knowledge with prior knowledge – integrating andresolving conflicts and gaps in knowledge. Although argumentation has been linked withuncertainty navigation in past studies, previous work has been focused on how uncertainty
plays a tentative and suitable role in challenging and resolving conflicting arguments inargumentative discourse (Nguyen & Young, 2022).
Therefore, this study investigated the following research question and hypothesis:
Research Question: To what extent does level of hedging indicating certainty
levels impact argumentative knowledge construction in individual work andgroup work?
Hypothesis: There was a significant difference in uncertainty using hedging forindividual and group work in argumentative knowledge construction.

2. Method
2.1 Study Design and Participants
18 Singapore Secondary grade 3 students (15 years old; 10 males; 8 females) participated inthis study. The teacher assigned students randomly to a total of 10 groups, with each groupconsisting of four students. They discussed collaboratively on an online collaborativeargumentation platform known as AppleTree. The students in the class knew one another andexperienced asynchronous group learning in other lessons. Students addressed the extent towhich they agreed and provided evidence for their claims and rebuttals/counterarguments on
a topic surrounding social issues: “Volunteering causes more harm than good. How far do you
agree?” In the individual work/ideation phase (15 minutes), students individually constructedarguments on the given topic. When students moved into group synergy/work (20 minutes),
they collaborate with one another to achieve a group consensus. Ethics approval and a signed
written informed consent was obtained from all participants before data collection in April 2023.

2.2 Coding Argumentation of Artefacts
Content analysis was used to address our research questions and hypotheses to examine thecertainty levels from students’ individual and group argumentative work. Argumentativeknowledge construction is operationalized as the average scores of these following coded
elements (see Table 1 below). Dichotomous coding was used. The presence of an element
below was coded as 1 whereas 0 was given if the element was absent.
Table 1. The Coding Scheme for Argumentation Knowledge Construction in Individual andGroup Work (Weinberger & Fischer, 2006; Weinberger et al., 2005)
Element
Description
Examples302SimpleClaim
Ground(withwarrants)
QualifierCounterargument

Integrated
Replies

Learners’position
Supporting
reason orevidence
Exceptions toclaim’s validity
Challenges toexisting claims

Integrations ofprevious points
or extension of
the argument“I agree that volunteering causes more harm than good.”“Yes, I agree because the person hosting this event wants
to try their best to support the volunteers rather than thoseneeding help.”“However, sometimes when you volunteer to help someone
the help you give may not really benefit them.”“Volunteering spreads kindness around our neighborhood,
precinct or even to further people, which could be a potential
for more people being helped by volunteers to being
volunteers to help those in need. However, some people
feel that volunteering also causes negative impacts, such as
people volunteering for financial gains and volunteering forwork resume experiences to be written on. Somevolunteering services are ineffective in helping people whichcauses more harm to the people in need than good.”“Research also shows that short-term orphanage visits cancause damage to children's development and emotional
well-being, creating unhealthy short-lived attachments andseparation anxiety. Children need constant sources oflove and support.”

Level of certainty is indicated by the number of uncertainty and certainty words usedin an argument (Nguyen & Young, 2022). Certainty and uncertainty words used by the learnerwere coded as 1 and then tabulated to compute a total score. If the participant used the same
number of certainty and uncertainty words, the level of certainty would be coded as neutral(Nguyen & Young, 2022). Examples of uncertainty words are hedges such as “can” and “maybe”. while certainty words include, “actually” and “definitely” (Nguyen & Young, 2022).

2.3 Procedure
Students went through individual work followed by group work and then back again, based on
the Spiral Model of Collaborative Knowledge Improvement (SMCKI) (Chen et al., 2021): (1)Individual ideation: students work on their own to indicate the extent to which they agreed or
disagreed with the given question with their supporting evidence; (2) Group synergy: individual
claims are merged together in a single claim; (3) Refinement: students reflected on the
feedback given to them by the othergroup, choosing to make changes if
needed; and (4) Individual achievement:
students write their own individualreflections after the whole argumentation
activity (Chen et al., 2021). For this study,
we examined students’ Individual
Ideation work, and compared theirindividual work with their group work
(Group Synergy).
The second and fourth authors
coded students’ individual writing worksheets and argumentation graphs (seeFigure 1) using the coding scheme (seeTable 1). These findings indicated sub- Figure 1. Sample of an argumentation diagram
stantial levels of agreement between the independent coders (Kappa: 82.5%). We used NVivo
version 12 and Microsoft Excel for the manual qualitative content analyses coding before
proceeding to analyze our quantitative data in IBM SPSS version 28.

303

3. Results. Discussion, and Conclusion
This study investigated the extent to which differences were observed in students’ level ofcertainty as shown in hedging used in the context of individual and group social knowledge
argumentation construction. The variables did not satisfy normality assumptions – potentially
due to a small sample size. Using Spearman rho correlations (Table 2), significant
associations (p < .01) were found between individual work qualifier and group work hedging
used, group work ground and group work simple argument, group work ground and group
work integrated replies, and group work ground and group work hedging used. It is expectedthat students’ certainty levels would decrease significantly as they move from individual work
to group work. This is in line with past research which encouraged students to use hedging to
express an openness to consider conflicting arguments against their initial position (Nguyen& Young, 2022). Nevertheless, students’ individual and group work were non-significant
across other categories of argumentation. Thus, students’ certainty navigation remained
consistent throughout argumentative knowledge construction in individual or group contexts.
Findings from one-way ANOVA (Table 3) revealed significant differences (p < .05) incertainty levels when forming simple arguments during individual work and counterarguments
at group work. Post-hoc Bonferroni’s correction revealed significant differences between
Certain and Neutral levels (p < .05, Mean Difference = .75; Standard Error = .26) when forming
simple arguments during individual work, and between Uncertain and Neutral levels (p < .05,
Mean Difference = .67; Standard Error = .18) when forming counterarguments at group work.
Navigating uncertainty self-regulatory skills and attitudes requires time to hone and build (Xuet al., 2015). Although students are encouraged to express certainty, reconciling varied
perspectives in argumentation required learners to take a tentative approach, hence, using
uncertainty words (Nguyen & Young, 2022). It is possible that learners only become moreaware of differences in synthesizing and expressing their arguments after moving on to group
work from individual work. These findings added support that students did not merely copy
and paste their individual work to group work, but rather, modified their work based on theirlearning experiences (Tsovaltzi et al., 2017). Future interventions may focus on supportinggroup interactions and scaffolding students’ argumentative work to enable smooth online
collaboration, familiarity with the challenges of communicating asynchronously (Oren et al.,2002), unshared knowledge in individual work translated into group work, levels of certainty
impacted by vicarious learning via influence from other group members (Fischer & Mandl,
2001; Weinberger & Fischer, 2006), and emotional self-control (Nguyen & Young, 2022).
Table 2. Spearman rho Findings on Certainty and Argumentative Knowledge Construction
Reflected in Individual and Group Work; IW: Individual Work; GW: Group Work; SA: Simple
Argument; Q: Qualifier; CA: Counterargument; IR: Integrated Replies; HU: Hedging Used
(Uncertain = 1, Neutral = 2, and Certain = 3)
Variables
1
2
3
4
5
6
7
8
9
10
11
IW-SA
IW-Q
.28
IW-CA
.22
.42
IW-IR
.90
.63
.81
GW-SA
.88
.59
.57
.69
GW-G
.73
.20
.44
.63 .01**
GW-Q
.49
.75
.88
.45
.49
.89
GW-CA
.50
.96
.07
.90
.06
.73
.49
GW-IR
.63
.47
.38
.21
.06 .01** .39
.63
IW-HU
.85
.07
.64
1.00
.53
.91
.37
.39
.71
GW-HU
1.00 .01** .83.21..08 .01** .84
.26
.24
.61
Table 3. ANOVA Findings on the Level of Certainty and Argumentative Knowledge
Construction as Reflected in Individual and Group Work (Uncertain = 1, Neutral = 2, andCertain = 3; df1 = 2, df2 = 15)
Variables
Certainty
M
SD
F
p

304

Individual Ideation
Simple Argument
QualifierCounterargument
Integrated Replies
Group Work
Simple Argument
GroundQualifierCounterargument
Integrated Replies

Hedging Used: Uncertain = 9, Neutral = 5, and Certain = 4
1
.78
.44
4.25
< .05*
2
1.00
.00
3
.25
.50
1
.33
.50
1.54
> .05
2
.20
.45
3
.75
.50
.56
.53
.49
> .05
1
.40
.55
2
3
.25
.50
1
.11
.33
.72
> .05
2
.40
.55
3
.25
.50
Hedging Used: Uncertain = 9, Neutral = 6, and Certain = 3
1
2.11
1.54
.07
> .05
2
1.83
1.47
3
2.00
1.00
1
2.11
1.36
.53
> .05
2
2.00
.89
3
1.33
.58
1
.11
.33
.24
> .05
2
.17
.41
3
.00
.00
1
.00
.00
6.04
< .05*
2
.67
.52
3
.33
.58
.44
.73
.36
> .05
1
.17
.41
2
3
.33
.58

References
Asterhan, C. S., & Schwarz, B. B. (2007). The effects of monological and dialogicalargumentation on concept learning in evolutionary theory. Journal of Educational
Psychology, 99(3), 626–639.
Chen, W., Tan, J. S. H., & Pi, Z. (2021). The spiral model of collaborative knowledge
improvement: an exploratory study of a networked collaborative classroom.
International Journal of Computer-Supported Collaborative Learning, 16(1), 7–35.
Chen, Y.-C., Benus, M. J., & Hernandez, J. (2019). Managing uncertainty in scientificargumentation. Science Education, 103(5), 1235– 1276.
Clark, D. B., & Sampson, V. D. (2007). Personally-seeded discussions to scaffold online
argumentation. International Journal of Science Education, 29(3), 253–277.
Fischer, F., & Mandl, H. (2001). Knowledge convergence. The role of shared external
representation tools. 8th European Conference for Research on Learning andInstruction.
Harney, O. M., Hogan, M. J., & Quinn, S. (2017). Investigating the effects of peer to peer
prompts on collaborative argumentation, consensus and perceived efficacy incollaborative learning. International Journal of Computer-Supported Collaborative
Learning, 12(3), 307–336.
Hou, H. T., & Wu, S. Y. (2011). Analyzing the social knowledge construction behavioral
patterns of an online synchronous collaborative discussion instructional activity using
an instant messaging tool: A case study. Computers and Education, 57(2), 1459–1468.
King, A. (2002). Structuring peer interaction to promote high-level cognitive processing.
Theory Into Practice, 41(1), 33–39.

305

Latifi, S., Noroozi, O., Hatami, J., & Biemans, H. J. A. (2021). How does online peer feedback
improve argumentative essay writing and learning? Innovations in Education andTeaching International, 58(2), 195–206.
Latifi, S., Noroozi, O., & Talaee, E. (2020). Worked example or scripting? Fostering students’
online argumentative peer feedback, essay writing and learning. Interactive Learning
Environments, 1–15.
Lu, J., & Zhang, Z. (2013). Assessing and supporting argumentation with online rubrics.
International Education Studies, 6(7), 66–77.
Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., & Yacef, K. (2013).
Capturing and analyzing verbal and physical collaborative learning interactions at anenriched interactive tabletop. International Journal of Computer-Supported
Collaborative Learning, 8(4), 455–485.
Mayweg-Paus, E., Zimmermann, M., Le, N. T., & Pinkwart, N. (2021). A review of technologies
for collaborative online information seeking: On the contribution of collaborativeargumentation. In Education and Information Technologies (Vol. 26, Issue 2).
Education and Information Technologies.
Nguyen, H., & Young, W. (2022). Knowledge construction and uncertainty in real worldargumentation: A text analysis approach. ACM International Conference Proceeding
Series, 34–44.
Noroozi, O., Banihashem, S. K., Taghizadeh Kerman, N., Parvaneh Akhteh Khaneh, M.,
Babayi, M., Ashrafi, H., & Biemans, H. J. A. (2022). Gender differences in students’argumentative essay writing, peer review performance and uptake in online learning
environments. Interactive Learning Environments, 1–15.
Oren, A., Mioduser, D., & Nachmias, R. (2002). The development of social climate in virtual
learning discussion groups. International Review of Research in Open and Distance
Learning, 3(1), 1–19.
Roberts, T. S. (2004). Supporting distributed problem-based learning: The use of feedback
mechanisms in online learning. In Online collaborative learning: Theory and practice(pp. 86–102). Information Science Publishing.
Schwarz, B. B., Neuman, Y., Gil, J., & Ilya, M. (2003). Construction of collective and individual
knowledge in argumentative activity. Journal of the Learning Sciences, 12(2), 219–256.
Storch, N. (2005). Collaborative writing: Product, process, and students’ reflections. Journalof Second Language Writing, 14(3), 153–173.
Tabak, I., & Baumgartner, E. (2004). The Teacher as Partner: Exploring Participant Structures,
Symmetry, and Identity Work in Scaffolding. Cognition and Instruction, 22(4), 393–429.
Tan, C., Niculae, V., Danescu-Niculescu-Mizil, C., & Lee, L. (2016). Winning Arguments.
Proceedings of the 25th International Conference on World Wide Web, 613–624.
Tsovaltzi, D., Judele, R., Puhl, T., & Weinberger, A. (2017). Leveraging social networking sites
for knowledge co-construction: Positive effects of argumentation structure, but
premature knowledge consolidation after individual preparation. Learning andInstruction, 52(3), 161–179.
Valero Haro, A., Noroozi, O., Biemans, H., & Mulder, M. (2020). Argumentation Competence:
students’ argumentation knowledge, Behavior and attitude and their relationships withdomain-specific knowledge acquisition. Journal of Constructivist Psychology, 35(1), 1–
23.
Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge
construction in computer-supported collaborative learning. Computers & Education,46(1), 71–95.
Weinberger, A., Fischer, F., & Stegmann, K. (2005). Computer-supported collaborative
learning in higher education: Scripts for argumentative knowledge construction in
distributed groups. Computer Supported Collaborative Learning 2005: The Next 10
Years!, January, 717–726.
Xu, J., Du, J., & Fan, X. (2015). Students’ groupwork management in online collaborative
learning environments. Educational Technology and Society, 18(2), 195–205.

306

ALT

307

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

ChatGPT’s Performance in Spreadsheets
Modeling Assessments based on Revised
Bloom’s Taxonomy
Michelle CHEONG
School of Computing & Information Systems, Singapore Management University
michcheong@smu.edu.sg
Abstract: ChatGPT has taken the education scene by storm and caused uneasinessamong educators. Mixed reactions were observed with some institutions banning it,
while others embracing it with caution. This paper evaluates the performance of
ChatGPT on solving spreadsheets modeling assessment questions with multiple test
items categorized according to the revised Bloom’s taxonomy, to discover the accuracyof the answers provided at each cognitive learning level. The insights obtained may be
useful for educators to design future assessment questions which focus more on
testing critical thinking skills to assess the students accordingly to achieve the intended
learning outcomes, and we propose recommended actions on how to do so. Our
proposed methodology can be applied to other course modules to achieve theirrespective insights for future assessment designs and actions.
Keywords: ChatGPT, Performance Evaluation,
Assessments, Revised Bloom’s Taxonomy

Spreadsheets

Modeling,

1. Introduction
ChatGPT (Generative Pre-trained Transformer) is a Generative AI (GAI) Large Language
Model (LLM) developed by OpenAI (OpenAI, 2022) trained to provide answers across a
myriad of domains. Released in November 2022 as free access to all users, New York City
(The Guardian, 2023) and New South Wales and Queensland in Australia (ABC News,2023) decided to ban it in schools with many other cities following suit. Such a reaction isreasonable as CNN reported that ChatGPT was able to pass the law exam and a businessschool exam (CNN Business, 2023). However, when ChatGPT was tested on three years ofSingapore’s Primary School Leaving Examinations (PSLE) for 6th grader, it failed miserably
in Mathematics and Science, and only managed a borderline pass for English (Business
Insider, 2023). Many papers supported using ChatGPT for teaching and learning, while atthe same time expressed many concerns (Alkaissi & McFarlane, 2023; Gilson et al., 2023;
Kung et al., 2023). Thus, it is not clear to many whether ChatGPT, or any other GAI like
Google Bard or Amazon Bedrock, is really a boon or bane to education.
In this paper, we review the recent literature on ChatGPT’s implications on educationand how it performed in terms of educational assessments. We then evaluate ChatGPT’s
performance in quiz questions for a university spreadsheets modeling module where thequestions with multiple test items are mapped according to the revised Bloom’s taxonomy.
With the answers provided by ChatGPT, we establish how well ChatGPT tackled technicalquestions on the different cognitive learning levels. We hope to provide educators withinsights to design future assessments on spreadsheets modeling module and propose
recommended actions on how to do so, especially for institutions which are ready toembrace ChatGPT in education. Our proposed methodology can be applied to other course
modules to achieve their respective insights for future assessment designs and actions.

308

2. Literature Review
Due to the recent rapid increase in the popularity of ChatGPT, several academic papers
were published on its implications on education and its performance in educational
assessments. Lim et al. (2023) presented four paradoxes of GAI and their implications to the
future of education. “Paradox 1: GAI is a friend yet a foe” highlighted its advantages inaccelerated learning, discovery of new knowledge, reform the design of assessments toachieve higher level, and yet concerned with misinformation and its impact on academic
integrity. “Paradox 2: GAI is capable yet dependent” discussed the need to make the bestuse of it to maximize returns, and yet be cognizant that it is dependent on the quantity andquality of prompts and its prior training. “Paradox 3: GAI is accessible yet restrictive”
discussed the need for a sustainable model to promote equitable access to all. “Paradox 4:
GAI is popular when banned” promoted its use in education rather than banning it which maylead to many negative consequences. Through the four paradoxes, they provided guidelines
for practices and directions for future research.
Pavlik who co-authored with ChatGPT (2023) studied its implications on journalism
and media. ChatGPT could provide high-quality answers and thus may pose a threat tojournalists and media professionals. He suggested educators to use it to develop course but
warned that students might use it in their own academic work compromising academic
integrity. Alkaissi and McFarlane (2023) presented two medical cases and asked ChatGPT
to provide scientific writing on specific medical and non-medical topics. The response
provided was mostly factually correct but with some errors. The references provided were
non-existent, demonstrating that it suffered from “artificial hallucination” where seemingly
realistic outputs do not correspond to any real-world inputs. Thus, they advocated using AI
output detectors in editorial processes and clear disclosures by authors. Similarly, Anderson
et al. (2023) also requested ChatGPT to write two academic papers on using AI for scientificmanuscripts and in sports medicine. They also found that the bibliographies generated were
wrong. One concerning finding was that when using GPT-2 Output Detector to assess theoriginality score for both papers, the scores increased tremendously by simple paraphrasing.
In terms of ChatGPT’s performance in educational assessments, Gilson et al. (2023)
and Kung et al. (2023) both evaluated ChatGPT’s performance in the United States Medical
Licensing Exam (USMLE). Gilson et al. (2023) used questions from question banks for Step
1 and Step 2 exams, while Kung et al. (2023) selected sample exam questions from June
2022 to test ChatGPT’s performance on answering original questions instead of retrieving
answers from the corpus which it was trained on. Both studies concluded that ChatGPT
passed the threshold of 60% and can potentially assist learners in medical education. One
interesting finding from Gilson et al. (2023) was that the accuracy of the responses
significantly decreased as the difficulty of the question increased. They measured the typesof errors committed by ChatGPT and found that it committed mostly logical errors (failure toconvert information to answer), then information errors (failure to identify key information),and very few statistical errors (arithmetic error).
With ChatGPT performing reasonably well in educational assessments, how should
future assessments be changed? Stokel-Walker (2022) suggested the possibility ofremoving essay writing altogether. Zhai (2022) suggested educators to focus on improving
students’ creative and critical thinking skills instead of general skills, and to use AI tools to
conduct subject-domain tasks. Echoing this, O’Connor who co-authored with ChatGPT(2023) recommended to use oral presentation and objective structured clinical examinations
to increase the diversity of assessments used in nursing education.
In this paper, we evaluate the accuracy performance of ChatGPT to solve technicalquestions on spreadsheets modeling and each question with multiple test items was mapped
against the revised Bloom’s taxonomy. Our work is different from previous works in two mainareas. Firstly, past works mostly evaluated ChatGPT’s performance in non-technicalmodules, while we used a technical module in spreadsheets modeling. The only other workthat used a technical module is the unpublished work by Malinka et al. (n.d.) where theyevaluated ChatGPT’s performance in computer security modules. They only shared thetypes of assessments used to test ChatGPT but did not share any details of the questions.

309

They found great variability in the correctness of ChatGPT’s answers and artificial
hallucination also occurred. Secondly, none of the previous works linked ChatGPT’s
performance on test items categorized on a cognitive learning scale, while we mapped test
item against the well-accepted revised Bloom’s taxonomy to achieve insights to advise futureassessment designs in response to recommendations by Zhai (2022) and Lim et al.(2023) tofocus more on assessing critical thinking skills.

3. Bloom’s Taxonomy and Spreadsheets Modeling Module
3.1 Bloom’s Taxonomy
Bloom’s taxonomy is a theoretical model (Bloom, 1956) intended to classify cognitive
learning from simple to complex level in a progressive manner, to aid educators in reducing
duplicate assessment items to achieve the same learning outcome. There are six levels (1)knowledge, (2) comprehension, (3) application, (4) analysis, (5) synthesis, and (6)evaluation. For each level, there are action verbs to classify assessments accordingly. To
represent the 21st century teaching and learning, Anderson and Krathwohl (2001) revised
Bloom’s taxonomy into a two-dimensional model. The six levels remained but they werechanged from noun to verb as (1) remember, (2) understand, (3) apply, (4) analyze, (5)evaluate, and (6) create. Levels 5 and 6 were swapped, and a second dimension called the“knowledge dimension” which has four levels (1) factual, (2) conceptual, (3) procedural, and(4) metacognitive, was added in addition to the original “cognitive dimension”.
One main challenge in applying Bloom’s taxonomy stems from the overlapping action
verbs in multiple levels, leading to ambiguity in its application as highlighted by Das, Mandal,
and Basu (2022). Even with the revised taxonomy, the ambiguity issue did not go away
totally. In this paper, we classify our test items to the different levels of the revised Bloom’s
taxonomy based on two module instructors’ expertise in the subject matter, and a third
instructor was consulted in case of different classification.

3.2 Spreadsheets Modeling Module
We use two quizzes created for a university level module in spreadsheets modeling toevaluate ChatGPT’s performance. The module objective is to teach students how to build
spreadsheet models from scratch and perform calculations and data analysis to support
business decision making. The learning outcomes include formulating business problems
and integrating business analysis skills to model and appraise business problems; acquiring
computer skills to perform problem analysis; and acquiring competency in spreadsheets tool.
The module covers six main topics as described in Table 1.
Table 1. Six Main Topics Covered in the Spreadsheets Modeling Module
Topic
1
2

3
4

Description
Basic modeling techniques to formulate arithmetic equations andusing Excel functions to perform basic statistical calculations.
Spreadsheet engineering skills in terms of using conditions toevaluate outcomes, create basic charts, compute intercept andslope of linear lines, create different order of polynomial functions
to represent business logic relationships to be included into themodel to seek for best answer using Goal Seek.
Financial calculations using Excel functions to understand andcompute time value of money to assess investment options andreturns.
Data lookup and linkup to handle changing inputs which affectoutputs, and to determine optimal solutions to business problems
which are subjected to constraints using Solver Add-in.

310

5

6

Monte-Carlo simulation to simulate repeated scenarios using
simulated input values from re-sampling, cumulative relative
frequency, and distribution functions, to obtain repeated output
results using Data Table, to support decision making.
Extends topic 5 to include date and time for time-based discrete event
simulation to determine performance of queue systems.

3.3 Assessment Quizzes
The module instructor designed the two quizzes for the spring term of academic year
2022/23. As the quiz questions were original, they did not exist before January 2023, andthus will not form part of the ChatGPT’s training corpus. Each quiz describes a businessscenario and contains multiple questions as listed in Table 2. Q1 to Q5 are for Quiz 1, while
Q6 to Q10 for Quiz 2. Each question contains multiple test items, and each item is mapped
to the revised Bloom’s taxonomy (BT), with the lowest at 3, and the highest at 6. It is notcommon to test students on levels 1 and 2 for university level modules. The mapping wasdone by two instructors and the inter-rater agreement between them was evaluated using
the Cohen’s kappa statistics (Hallgren, 2012), and found to be 0.902.
Table 2. Questions and Test Items from Two Quizzes and their Bloom’s Taxonomy (BT)QuestionQ1. A mall offers different discounts, Types
A to F, and allows shoppers to split their
total purchase amount into multiple
payments to enjoy as many discounts
as possible. Type A is a $10 discountwith minimum $140 spending. Type B isa $20 discount with a minimum $500
spending. Type C is a $10 discount witha minimum of $125 spending. Type D isa $45 discount with a minimum
spending of $800. Type E is a $15
discount with a minimum $800
spending. Type F is a 1.5% discountwith no minimum spending. Computethe ratio defined as discount divided byminimum spending, and compute therank of the ratio in descending orderusing a suitable Excel function. Create
an Excel spreadsheet model, called
Table 1, to determine the answer.
Q2. You plan to buy a computer for $1200
and wish to split the purchase amountinto multiple payments to use as many
discounts as possible, to enjoy the
maximum discount. Create a second
Excel spreadsheet model, called Table
2. In the first column, list the rank from
1 to 6. Then use any Excel lookup
function to retrieve the discount types
from Table 1 corresponding to the rank
position, displaying the discount, theamount to be charged to this discount,and the discount enjoyed. It is only
smart to charge the minimum amount to311

Test itemx Test arithmetic tocompute ratio.
x Testidentification andapplicationofcorrect
Excelfunction to rank
ratio.

BT
3

x

3

x

Test
identification andapplicationoflookup function
to retrieve data.
Test formulationofcomplexformulastodetermine
ifdiscount is usedandamountcharged to each
discount
considering
all

4

6

the discount if this discount is used.
Then add another column to indicate if
the discount is used using “Yes” or “No”
label. Finally, compute the amountpayable if a discount is used, and thefinal total amount payable.

xx

Q3. With the final total amount computed
earlier, you can choose to pay it overthree equal monthly instalments. Usingan annual interest rate of 2%, what
would be the present value of the final
total amount?

x

Q4. Your friend wants to buy your computer
for $1400 today. How much would you
earn today if you were to sell yourcomputer to your friend, based on thepresent value of the final total amount
you paid computed earlier?
Q5. At what annual interest rate would you
be able to earn $300 today if you wereto sell your computer to your friend for$1400, based on the present value ofthe final total amount you paid using thisnew annual interest rate?
Q6. At a clinic, patients join a single queuelining up one after another to see thedoctor. Assuming the inter-arrival timeof patients follows an exponential
distribution with mean 4 minutes. Out ofthese patients, 35% of them see thedoctor due to suspected COVID-19
infection, while the remaining 65% arenormal patients. For those who suspect
COVID-19 infection, the doctor will test
to confirm if they are indeed infected. It
was found that 70% of them are
confirmed. Due to different medical
conditions, the consultation time differs,
and they all follow normal distributions
with different means and standard
deviations. For the normal patients, themean is 5 minutes and standard
deviation is 1 minute. For the suspected
and not confirmed patients, the mean is
7 minutes and standard deviation is 2
minutes. For the suspected andconfirmed patients, the mean is 15
minutes and standard deviation is 3
minutes. Create an Excel model Table
1 with the first column to simulate thex312

discounts
collectively
toenjoy maximum
discount.
Test
amountpayableifdiscount is used.
Test computationof total amountpayable.
Test
identification andapplicationofcorrect
Excelfunction
toperform financial
calculation.
Test arithmetic tocomputeearnings.

5
3

4

3

x

Test
recursive
calculation using
Goal Seek or
Solver.

5

x

Test
data
simulation using
exponential
distributionforinter-arrival time.
Test conditional
probability
tosimulate patient
type.
Test
time
calculation.
Test IF condition
to get correct
inputsfor
simulating
consultation timeusing
normaldistributionfordifferent patient
type.

4

x

x
x

4

3
5

inter-arrival time of 50 patients, thencompute the arrival time of the patientsusing 9am as the reference startingtime in the second column. Then add a
new column to simulate if this patient issuspected COVID-19 case using “Yes”
or “No” label. If “Yes”, then add anothercolumn to simulate if this patient is aconfirmed COVID-19 case using “Yes”
or “No” label. Finally, add anothercolumn to simulate the consultation
time depending on the patient type.
Q7. Continuing with Table 1 created earlier,for each patient determine theconsultation start time, consultation end
time, the wait time, and the system time.
Update as Table 2.

Q8. While waiting in the line, it is possiblethat a non-COVID-19 patient (e.g.
Person B) may get infected by the
person standing directly in front (e.g.
Person A), and/or the person standing
directly behind (e.g. Person C), if
Person A and/or Person C areconfirmed COVID-19 cases, and theoverlap time they spend standing in line
next to Person B exceeds 30 mins.
Continuing with the Table 2 createdearlier, add two new columns todetermine the overlap time between
Person A and Person B, and overlap
time between Person B and Person C.
Finally, add a last column to indicate foreach patient, will this patient get
infected using “Yes” or “No” label.
However, if this patient is already aconfirmed COVID-19 patient, indicate“C+” instead.
Q9. Using the updated Table 2 createdearlier, compute the number of possible
new infections, and use Data Table torepeat this simulation 500 times todetermine an overall average numberof possible new infections.

x
x

x
x
x

x

x
x
Q10. The doctor thinks that he should
reduce the mean consultation time by 2

313

x

Test
queue
system concepts.
Test arithmetic tocomputestarttime, end time,
wait time system
time.
Test
queue
system concepts.
Test arithmetic tocompute overlap
time.
Test formulationofcomplexformulastodetermine new
infection status
takingintoaccount multiple
linkedconsiderations.

4

Test
identification andapplicationofcorrect
Excelfunction to countbased on test
condition.
Test Data Table
concept to repeat
simulations.
Test arithmetic tocomputeaverage.
Test Data Table
concept to repeat

4

4

4
4
6

5
3

6

minutes. He can choose to do that for
only one of the cases, normal patients,
suspected and not confirmed patients,
or suspected and confirmed patients.
Regardless of which case he decides toreduce the mean consultation time, thewait time for all patients will be reduced,
and thus lead to reducing the number ofpossible new infections. Determine thenew overall average number of possible
new infections for each case using Data
Table and conclude which case will bemost effective and explain why.

xsimulations withdifferent inputs totest
different
scenarios.
Test inference of
insights
fromresults obtained.

5

To simulate the similar answering process as real students, the questions wereposted to ChatGPT sequentially in one single continuous conversation to allow it to refer to
its previous responses to generate the next response for follow-up questions. This is an inbuilt capability that ChatGPT has, which fascinated many users.

4. ChatGPT’s Performance in Spreadsheets Modeling Assessment
We evaluate ChatGPT’s performance for the quizzes over three complete runs and computethe average accuracy at each cognitive level. While the responses from ChatGPT were notexactly the same for each run, they were rather similar albeit different mistakes committed.
The main difference comes from the way the responses were presented, which could be in
the form of step-by-step explanation or presented as table. We found that ChatGPT can
perform reasonably well at level 3 with 67% accuracy and the accuracy decreases as the
cognitive level increases. This result is similar to the work by Gilson et al. (2023) where theyreported a significant decrease in accuracy as the question difficulty increases for USMLE.
Table 3. ChatGPT’s Accuracy Performance at Different Bloom’s Taxonomy (BT) LevelBT
3 (apply)
4 (analyze)
5 (evaluate)
6 (create)# items
6953

ChatGPT Average Accuracy
67%
39%
33%
22%

At level 3 (apply), ChatGPT was able to perform arithmetic calculations reasonably
well. However, it committed mistakes when it misinterpreted the information provided in thequestion. For example, in Q1, the 1.5% discount for Type F is equivalent to a discount ratioof 0.015 but ChatGPT misinterpreted it and indicated it as N/A instead (Figure 1).

Figure 1. ChatGPT Misinterprets Information.

314

At level 4 (analyze), ChatGPT was able to analyze the problem and determine thecorrect Excel function to use but was not able to apply the function correctly. For example, in
Q3, it correctly identified using PV function, but the arguments to fill in the function were
wrong, where the [type] should be set as 1 and not ignored which means type 0 (Figure 2).

Figure 2. ChatGPT Correctly Identifies PV Function but Filled in Wrong Argument.
At level 5 (evaluate), ChatGPT was able to explain the evaluation approach, but was
not able to perform the evaluation correctly. For example, in Q9, ChatGPT can explain how
to use Data Table to perform the simulation, but the actual calculation using COUNTIFS
function was not done correctly, as it should only count patients with new “Yes” label, andnot those who are “No” and “C+” labels. (Figure 3).

Figure 3. ChatGPT can Explain Data Table Setup but Performed Wrong Calculation.
At level 6 (create), where complex formulas which involve linking several parts ofcalculations together are expected, ChatGPT failed miserably. It tends to over-simplify thecalculations leading to the wrong answers. For example, Q2 required formulas to determine
if a specific discount is used and the amount charged to each discount if that discount isused. This requires all discounts to be considered together as a whole to maximize total
discount, and not treat each discount as separate consideration.

5. Insights and Implications315

We found that ChatGPT’s responses are often self-contradictory and incoherent when
responding to technical calculation questions. It can describe the calculation process
correctly, but the actual calculation was performed incorrectly. Very often, the suggested
calculation steps are messy and cell references in the provided formulas are not referring to
the correct cells. One plausible explanation for such a behavior is that ChatGPT is alanguage model. It excels in “auto-complete” and thus performed well in essay writing.
However, when it comes to logical and coherent thinking required in complex calculations
involving linking multiple considerations, it was not able to handle.
On the other hand, we found that ChatGPT is knowledgeable in explaining Excel
functions in terms of the arguments and how to use them. And it is surprisingly good in
queuing system, probability and distribution functions concepts. It could be due to the corpus
it was trained on which is mainly based on textual content and facts.
Generative AI is here to stay and will become more powerful creating immense
impact on education. Instead of outright banning it, educators should consider how to bestuse it for course development, teaching and assessment, as echoed by many researchers
including Lim et al. (2023) and Zhai (2022). Based on the results of our study, we provide
three recommendations for educators who teach spreadsheets modeling modules. One,
allow students to use ChatGPT to answer in-class activity questions (not actual
assessments), and guide them to identify errors and limitations in the answers and tosuggest how to improve the answers. For example, request students to build a first model
based on ChatGPT’s suggested answers, and then build a second model with any identified
errors corrected. Two, ask students to formulate questions to ask ChatGPT and throughanalyzing the answers, learn to sharpen the questions to represent the exact intent of thequestions as a form of interactive learning (Rospigliosi, 2023). For example, students can
ask ChatGPT to compare two different investments options, and then sharpen the question
to ask for comparison using net present value or internal rate of return calculations. And
lastly, design assessment questions which focus on testing higher order thinking skills, withtest items from level 4 onwards (Zhai, 2022). For example, design questions that link data
lookup function with conditional test to determine the answer, instead of questions that only
require simple arithmetic calculations like computing the sum or average. Our
recommendations aim to overcome diminishing learners’ own innovative capacities andcritical thinking skills, and focus on enhancing their higher order thinking skills.

6. Conclusions and Future Research
Our study shows that ChatGPT’s performance in spreadsheets modeling assessment
questions was good up to level 3 of the revised Bloom’s taxonomy, and its accuracy
decreases as the cognitive level increases. It could explain technical facts very well andperformed reasonably well in basic arithmetic calculations. However, from level 4 onwards,
ChatGPT was unable to apply Excel functions correctly even though it has identified thecorrect Excel function; unable to evaluate complex situations even though it could explain
the evaluation approach; and unable to formulate complex formulas that link multiple
considerations as its formulations were often simplified. However, we believe that ChatGPT
and other GAI will become better and as educators we should exploit their full potential in
education while keeping in mind how to overcome the concerns on cheating, plagiarism andloss of independent thinking skills in students. This calls for a balanced approach and aparadigm shift in education.
We recommend several future research possibilities involving GAI. From the
instructors’ perspective, we can perform research in using GAI to generate assessments andassess the cognitive levels of such assessment questions to understand if GAI can generate
questions of higher order thinking skills, and how much GAI can lighten instructors’
workload. From students’ perspective, we can quantify the improvement in knowledge gainin students when using GAI as part and parcel of their learning process. From the GAI tool’s
perspective, we can consider developing and implementing customized GAI tool using LLM

316

to teach and learn technical modules such as spreadsheets modeling or Python
programming, and assess the tool’s effectiveness.

References
ABC News. (2023). Queensland to join NSW in banning access to ChatGPT in state schools. ABC
News. https://www.abc.net.au/news/2023-01-23/ queensland-to-join-nsw-in-banning-accessto/101884288
Alkaissi, H., & McFarlane, S.I. (2023). Artificial hallucinations in ChatGPT: Implications in scientificwriting. Cureus 15(2): e35179. DOI:10.7759/cureus.35179
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing:A revision of Bloom’s taxonomy of educational objectives. New York, NY: Longman.
Anderson, N., Belavy, D.L., Perle, S.M., Hendricks, S., Hespanhol, L., Verhagen, E., & Memon, A.R.
(2023). AI did not write this manuscript, or did it? Can we trick the AI text detector into generated
texts? The potential future of ChatGPT and AI in Sports & Exercise Medicine manuscript
generation. BMJ Open Sport & Exercise Medicine, 9:e001568. DOI:10.1136/bmjsem-2023001568
Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: The classification of educational
goals. Handbook I: Cognitive domain. New York, NY: David McKay Company.
Business Insider (2023). ChatGPT failed miserably in Singapore's 6th-grade tests, averaging 16% formath and 21% for science. Days later, it was getting answers right.
https://www.businessinsider.com/chatgpt-failed-singapore-sixth-grade-exams-psle-2023-2
CNN Business (2023). ChatGPT passes exams from law and business schools.
https://edition.cnn.com/2023/01/26/tech/chatgpt-passes-exams/index.html
Das, S., Mandal, S.K.D., & Basu, A. (2022). Classification of action verbs of Bloom’s Taxonomy
cognitive domain: An Empirical Study. Journal of Education, 202(4), 554-566.
doi.org/10.1177/00220574211002199
Gilson, A., Safranek, C.W., Huang, T., Socrates, V., Chi, L., Taylor, R.A., & Chartash, D. (2023). How
does ChatGPT perform on the United States Medical Licensing Examination? The implications ofLarge Language Models for medical education and knowledge assessment. JMIR Medical
Education, 9.
Hallgren, K.A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial.
Tutorials in Quantitative Methods for Psychology, 8(1), pp. 23-34.
Kung, T.H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepano, C., Madriaga, M.,
Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on
USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit
Health, 2(2): e0000198. doi.org/10.1371/journal.pdig.0000198
Lim, W.M., Gunasekara, A., Pallant, J.L., Pallant, J.I., & Pechenkina, E. (2023). Generative AI and thefuture of education: Ragnarok or reformation? A paradoxical perspective from management
educators. The International Journal of Management Education, 21, 100790. DOI:
10.1016/j.ijme.2023.100790
Malinka, K., Peresini, M., Firc, A., Hujnak, O., & Janus, F. (n.d.). On the Educational Impact of
ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?
https://arxiv.org/abs/2303.11146
O’Connor, S., & ChatGPT. (2023). Open artificial intelligence platforms in nursing education: Tools foracademic progress or abuse? Nurse Education in Practice, 66.
OpenAI. (2022). ChatGPT: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/
Pavlik, J.V. (2023). Collaborating with ChatGPT: Considering the implications of Generative Artificial
Intelligence for journalism and media education. Journalism & Mass Communication Educator,
78(1), 84-93, DOI: 10.1177/10776958221149577
Rospigliosi, P. (2023). Artificial intelligence in teaching and learning: what questions should we ask of
ChatGPT?, Interactive Learning Environments, 31:1, 1-3, DOI: 10.1080/10494820.2023.2180191
Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays — should professors worry? Nature.
https://doi.org/10.1038/d41586-022-04397-7.
The Guardian (2023). New York City school bans AI chatbot ChatGPT.
https://www.theguardian.com/us-news/2023/jan/06/new-york-city-schools-ban-ai-chatbot-chatgpt
Zhai, X. (2022). ChatGPT user experience: Implications for education.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4312418

317

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Promoting Middle School Students'
Achievement and Attitude toward Science
Learning through Sphere RecognitionBased AR Application
Ruixue LIUa*, Lijun LIANGa & Xiaodong WEIa
School of Educational Technology, Northwest Normal University, China*isnow0211@163.comaAbstract: With the advancement of Augmented Reality (AR) technology, considerable
studies have explored AR’s application in various educational fields, especially in thefield of science. This paper used a sphere recognition-based identifier to develop an
AR application that supports middle school students learning of science. A quasiexperimental method was adopted, using a pretest and posttest of science knowledge
to measure academic performance. A questionnaire was conducted to measure
students’ attitudes toward the use of AR in science lessons. The research sample
included 80 middle school students and was divided into experimental and control
groups. Students in the experimental group learned science lessons by using a sphererecognition-based AR application, while the control group learned the same material
by using a traditional method. Results showed that students who learned science
lessons in a sphere recognition-based AR application performed better in academic
achievement than that in the control group. In addition, the experimental group showed
a positive attitude toward the use of AR in science classes. This study provides
empirical evidence to explore the use of sphere recognition-based AR in scienceeducation.
Keywords: Augmented reality, academic achievement, middle school, sciencelearning, attitude

1. Introduction
Science learning is an important subject in the educational area. However, learners often
face difficulty in learning science because it involves many complex and abstract concepts(e.g. gravity, electromagnetic phenomena, etc) (Anderson & Barnett, 2013). In this way,learners may have misunderstandings in the process of understanding and analyzing
scientific phenomena due to the lack of intuitive and vivid presentation. In addition, realscientific practice faces problems such as geographical distance and security risks (Zhang &Wang, 2021). For example, previous studies have indicated that students could grasp thedescription of the phases of the moon, but they often misunderstand the interpretation of themoon phases and believe that the moon phases are formed by the occlusion of the sun(Lelliott & Rollnick, 2010; Wilgenbus & Lena, 2009). Therefore, many students may havedifficulties in science learning, such as low interest and confidence in science curricula, andlow learning achievement.
With the development of emerging technologies, various visual technologies such as
virtual reality, and augmented reality can be applied in educational fields to promotestudent’s learning experience. Augmented reality (AR) is a technology that seamlessly
connects virtual information with the real world and has the characteristics of virtual and realfusion (Azuma, 1997). It provides opportunities for learners to observe and explore 3Dmodels and support their understanding of intellectual concepts (Xu et al., 2022). With thehelp of this technology, learners can place 2D or 3D virtual information in real scenes byusing their devices to identify markers. Moreover, AR has the advantages of presenting

318

abstract scientific knowledge and supporting scientific exploration. Fleck and Simon (2013)indicated that AR is a good scaffold for learning science knowledge due to its ability toenhance sensorimotor interactions, visual guides, and realistic representation. This study
aims to investigate the effect of sphere recognition-based AR application on middle school
students learning of science. The research questions are as follows:y Compare to the traditional teaching methods, do students who learned sciencethrough using a sphere recognition-based AR application perform better inacademic achievement?
y What are the attitudes of students toward the use of sphere recognition-based AR
applications in science learning?

2. Literature Review
2.1 Science Learning
Science is a discipline concerned with the study of organisms and events in nature. It
requires people to form concepts and principles by observing and generalizing things andphenomena (Karagozlu, 2021). The purpose of science education is that it requires students
to form scientific concepts, master the nature of science, and improve their willingness toactively participate in scientific learning (National Research Council, 1996). One of the main
challenges for science education is helping learners to develop a conceptual understandingof scientific concepts (Osborne et al., 2016). However, there is a lack of physical conditions
to support learners in practicing learning tasks in a school classroom setting. To solve these
problems, researchers have attempted to integrate AR and virtual reality into the sciencecurriculum (Monita & Ikhsan, 2020). Since these technologies provide opportunities for thevisualization, simulation of scientific knowledge, and science practice.

2.2 AR in Science Learning
AR is a technology that can superimpose virtual information in the real world. It has threecharacteristics of the combination of reality and reality, real-time interaction, and 3Dregistration (Azuma, 1997). Based on AR’s characteristics, it brings new opportunities in thefield of education (Zhang et al., 2022). First, the combination of reality and reality canpresent 3D learning content for learners, which can contribute to the visualization ofknowledge and solve the problems of simulating inaccessible, invisible, and dangerous
scientific phenomena. Second, AR supports students’ real-time interaction with virtual
information in a real environment. Moreover, some studies have shown that AR plays animportant role in increasing learning motivation and engagement, as well as enhancing
learners’ creativity and imagination (Demircioglu et al., 2022; Lorusso et al., 2018).
Previous studies have applied AR in science education. For example, Liu et al.
(2023) developed an AR application for chemistry courses for middle school students level.
The results showed that AR helped increase knowledge gains and improved motivation.
Wang (2022) developed a game-based AR science learning system for middle school
students to learn electromagnetic phenomena. Results found that the proposed system has
advantages in improving interactivity, learning performance, and learning satisfaction. Tina
et al. (2019) developed a real-world-oriented smartphone AR system and they found that thesystem is an effective learning tool and improves the interest in learning science.
Overall, previous studies have demonstrated the advantages of AR in scienceeducation. Among them, marker-based (such as image-based) AR is commonly used. Other
types of AR, such as location-based and object recognition-based AR, are lessimplemented. In this study, we use an AR application based on sphere recognition to explore
its effects on science learning.

319

3. Development of a Sphere Recognition-based AR Science Learning System
3.1 System Structure Overview
In this study, a sphere recognition-based AR application was developed using Unity andVuforia engines. The process of the development of the AR application requires the use of“Scanner” software to scan spheres and then generate data files of the sphere model.
Following that, upload the data file to the database of the Vuforia engine. Next, connect thecorresponding Vuforia engine database in the Unity engine, and edit the sphere data in the
Unity engine to form the final AR application. The AR application was eventually released as
an APK and installed on students’ mobile devices. Figure 1 shows the system structure.

Figure 1. The System Structure of the AR Application.

3.2 Sphere Recognition-based AR Science Learning Application
The theme of the science curriculum is “Exploring the Earth-Moon System”. The AR
application consists of three main parts: (a) “Moon Observation”, (b) “Exploring the Moon
Phase”, and (c) “The Solar and Lunar Eclipses” module. Students can click the module
selection button to enter the interface according to the learning progress on the AR
application homepage (see Figure 2).

Figure 2. The Interface of the Homepage

320

The “Moon Observation” module includes the moon’s position in the universe, thetypes of terrain on the moon, and the characteristics, and locations of the terrain (see Figure3). Students scan the sphere using their mobile phones, and a 3D model of the moon
appears on the page. Students can interact with the 3D model, including zooming androtating. The surface of the 3D model is equipped with marker points, and students are
guided by the marking points. When students click on the markers, text and images
describing the model appear on the right side of the screen. In addition, by clicking on the
buttons at the bottom of the screen, students can get an audio or video explanation of thelunar terrain.

Figure 3. The Interface of the “Moon Observation” Module
The “Exploring the Moon Phases” module includes the types of moon phases, theevolution of moon phases, and reasons for the formation of moon phases (see Figure 4).
Students scan the sphere using their mobile phones, and a 3D model of the Earth-Moonsystem appears on the page. After the student selects the date on the left side of the page,the 3D model shows the relative positions of the Earth and the Moon, and an image of thecorresponding moon phase appears on the right side of the page. Students can click the
buttons at the bottom of the screen to get text, audio, and video explanations of the moonphase.

Figure 4. The Interface of the “Exploring the Moon Phases” Module321

In the “The Solar and Lunar Eclipses” module (see Figure 5), students learn the types
and patterns of solar and lunar eclipses and analyze the reasons for their formation. They
select an option from the drop-down menu in the upper right corner of the screen. The 3Dmodel is corresponding to the options that will be presented on the page. Students select the“Annular Solar Eclipse” option, and the 3D model on the page simulates the relative
positions of the Sun, Moon, and Earth during an annular solar eclipse, as well as the lightand shadow phenomena. Text and images of the annular solar eclipse are presented at thebottom of the page. Students can also use the buttons on the left side of the page to accessaudio and video explanations of the annular solar eclipse.

Figure 5. The Interface of the “Solar and Lunar Eclipses” Module

4. Methods
4.1 Research Design
This study used a quasi-experimental method to investigate the effect of a sphererecognition-based AR application on students’ academic performance and attitude towardscience learning. The two classes participating in the experiment were divided intoexperimental and control groups. Before the experiment, both groups of students took apretest to ensure that they had a similar knowledge level. Following that, students in theexperimental group learned science lessons by using a sphere recognition-based AR
application, while the control group learned the same content using traditional teaching
methods. After finishing the experiment, a posttest and questionnaires were administered toboth students.

4.2 Participants
The study was conducted in a public middle school in the Northwest region of China. A totalof 80 students from two classes with an average age of 13 years old participated in thisstudy. Among them, 40 students were in the experimental group and 40 students were in thecontrol group. Both classes had approximately the same proportion of boys and girls. The
science lessons were taught by the same teacher with more than two years of teachingexperience. All students had not been exposed to AR science lessons before theexperiment.

322

4.3 Measurement Tools
4.3.1 Pretest and Posttest
To test students’ science learning effect, pretest, and posttest were used to measure
students’ academic achievement before and after the learning activity. Both tests weredeveloped by research and the teacher and have a Cronbach alpha of 0.83. The pretest andposttest consist of 10 multiple-choice items (e.g., When does the lunar eclipse occur? A.
new moon B. upper moon C. lower moon D. full moon), 6 true or false items (e.g., TheaveraJHGLVWDQFHIURPWKHPRRQWRWKHHDUWKLVDERXWî຿ km. A. True B. False), and 2
short-answer items (e.g., How many stages are there in the moon phase? What is theformation principle of the moon phase?). The perfect score on both tests is 100. An expert
panel of three educators examined the validity of the tests. Both science teachers and theexpert panel reviewed the final version of the tests. We modified the inappropriate items in
the tests to ensure high reliability and validity.
4.3.2 QuestionnaireThis study used an AR Application Attitude Scale (ARAAS) developed by Kucuk et al. (2014)
to measure students’ attitudes toward the use of AR applications in science lessons. The
ARAAS adopts a 5-point Likert scale (“1”= Strongly Disagree, “5”= Strongly Agree) andconsists of 14 items with three dimensions, including “using satisfaction”, “using anxiety”,
and “using willingness”, among which six positive descriptions reveal students’ satisfaction
with using AR application, (e.g., I enjoy the AR lessons ), six negative descriptions reflect
students’ anxiety level in using AR applications (e.g., It is difficult to use AR applications),
two positive descriptions show students’ intention to use AR applications in the future (e.g., I
want AR applications in other lessons). The values of Cronbach’s alpha for the three
dimensions were 0.86, 0.82, and 0.89, respectively. The value of Cronbach's alpha for theoverall scale was 0.83, indicating that the scale is reliable.

4.4 Procedure
As shown in Figure 6, the experiment lasted for 5 weeks. In the first week, students in bothgroups took a pretest. From the second to the fourth week, the students participated inscience lessons. Each week, both groups of students took a 45-minute science lesson. An
example of a 45-minute class is shown in Table 1. During the lesson, students in theexperimental group learned science lessons using a sphere recognition-based AR
application. the teacher first conducts the lesson by introducing the topic. After that, theteacher introduces the use of AR and guides the students to use it. When finished, students
use the AR application for science learning. In the end, the teacher guides the students tosummarize and share the knowledge learned in the lesson. The control group followed thesame teaching process and used traditional teaching methods. In the fifth week, both groups
took the posttest. Meanwhile, students in the experimental group were also required tocomplete the questionnaire.

323

Figure 6. Procedure
Table 1. An Example of a 45-minute Class
Step
1
2
3
4

Activity (T for Teacher, S for Students)
T introduces the content and topics learned in this lesson or
guides S to review
T introduces and demonstrates the use of AR application andguides S to scan identifiers
S use AR applications to learn
T guides S to summarize the learning content and invites S toshare what have learned today

Time
10min
5min
20min
10min

4.5 Data Analysis
In this study, data were analyzed using SPSS software. Descriptive analyses and t-tests
were used to analyze the student’s academic achievement. In terms of attitudes toward AR
technology use, descriptive analysis was used to analyze data from students' ARAAS
questionnaires.

5. Results
5.1 Academic Achievement
The results of the pretest scores are given in Table 2. The Mean values of the pretest scores
were 47.75 for the experimental group and 47.50 for the control group respectively. The ttest results (t = 0.27, p > 0.05) showed no significant difference between the two groups,
indicating that both groups had a similar level of a priori knowledge before the learning
activity.
Table 2. T-test Result of Pretest and Posttest Scores
PretestPosttest

Group
Experimental
Control
Experimental
Control

N
40
40
40
40

M
47.75
47.50
83.70
72.33

*p < 0.01.

324

SD
4.14
4.23
4.50
6.25

t

p

0.27

0.79

9.34

0.00*

At the end of the learning activities, both groups of students took a posttest. The
Mean values of post-test scores were 83.70 for the experimental group and 72.33 for thecontrol group. The t-test results (t = 9.34, p < 0.01) showed a significant difference between
the two groups. It can be concluded that the students who use AR applications for sciencelearning had better learning performance.

5.2 Attitude Towards AR Applications
As shown in Table 3, the Mean values and SD of the overall attitudes澳for the experimentalgroup were 3.95 and 0.27, indicating that students in the experimental group gave highattitudes toward the use of AR applications in science classes. Moreover, the Mean values
and SD of satisfaction for the experimental group were 3.91 and 0.38, and the willingness forthe experimental group was 3.86 and 0.57. In addition, the Mean values and SD of anxiety
for the experimental group were 1.99 and 0.29. The results showed that students weresatisfied with using AR applications in their courses, with a low level of anxiety and theygenerally showed their intention to continue using AR applications in the course. It can beconcluded that students in the experimental group have positive attitudes toward using AR
applications in science courses.
Table 3. Descriptive Data of ARAAS
Dimension
Satisfaction
Anxiety
Willingness
General ARAAS

N
40
40
40
40

Min
3.17
1.00
3.00
3.50

Max
5.00
2.50
5.00
5.00

M
3.91
1.99
3.86
3.95

SD
0.38
0.29
0.57
0.27

6. Discussion and Conclusion
In this study, we explored the effects of a sphere recognition-based AR澳application on middle
school students’ academic performance and their attitudes toward science learning. The
results showed that students who used a sphere recognition-based AR application for
science learning had significantly higher academic achievement than that of the control
group. In addition, students in the experimental group gave positive attitudes regarding thesatisfaction and willingness to use AR applications in a science curriculum.
In terms of academic achievement, the results of the current study are consistent withprevious studies (Kalemkus & Kalemkus, 2022; Wang, 2022). In this study, the reasons forsuch an effect may be as follows. First, AR helps visualize scientific knowledge. Although
middle school students have certain abilities of cognitive and abstract, they are prone tomisunderstand complex scientific phenomena such as the moon phase. AR presents
complex and abstract scientific phenomena in the form of 3D models in the real world, andstudents can observe intuitively through real-time interaction to facilitate understanding andlearning. Second, this study uses spheres as identified objects because the external shapesof the Earth, Moon, and Sun are spherical. Compared with 2D images, the 3D model has abetter sensory effect in the way of sphere recognition. The fun and manipulability of thelearning process motivate students to maintain a high level of motivation and interest in
learning, which provides the possibility for learning more knowledge and finally effectively
improves their academic performance.
Moreover, students showed positive attitudes toward the use of AR applications,which confirmed other studies’ similar results (Celik & Ersanli, 2022; Fidan & Tuncel, 2019;
Kaya & Bicen, 2019; Karagozlu et al., 2019). They were satisfied with the use of AR
application in science lessons. Visual observation displayed by the AR helps students
understand and learn from the knowledge. They also looked forward to using AR in futurescience lessons.

325

7. Limitations and Future Research
Although the above-mentioned study provides a positive result on using AR in middle school
science learning, some limitations still exist in the current study. First, students were first
exposed to AR learning activities in this study. Therefore, the novelty of AR learning
activities may be a reason for the positive effects. In future research, pre-experiments shouldbe performed on this issue to eliminate the influence of this factor on experimental results.
Second, the experimental sample was limited to middle school students. Future studies
should consider taking other age levels into account and increasing the sample size toobtain more comprehensive and accurate data. Moreover, this study used a sphererecognition-based AR application to investigate the students’ learning effect in science
lessons. In future research, the effect of using AR applications of other recognition methods
can be examined, and comparative research can be conducted to see which recognition
method has a better effect in supporting students' learning. It is also necessary to consider
the long-term effects of the use of sphere recognition-based AR applications in middle
school science learning.

Acknowledgments
The research was supported by the National Natural Science Foundation of China [grant
number 62067009], and the 2023 Gansu Province University Youth Doctoral Support Project(Project No. 2023QB-110).

References
Anderson, J. L., & Barnett, M. (2013). Learning physics with digital game simulations in middle school
science. Journal of science education and technology, 22, 914-926.
Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtualenvironments, 6(4), 355-385.
Celik, F., & Ersanli, C. Y. (2022). The use of augmented reality in a gamified CLIL lesson andstudents’ achievements and attitudes: a quasi-experimental study. Smart Learning
Environments, 9(1), 30.
Demircioglu, T., Karakus, M., & Ucar, S. (2022). Developing students’ critical thinking skills andargumentation abilities through augmented reality–based argumentation activities in scienceclasses. Science & Education, 1-31. https://doi.org/10.1007/s11191-022-00369-5
Fidan, M., & Tuncel, M. (2019). Integrating augmented reality into problem based learning: The effects
on learning achievement and attitude in physics education. Computers & Education, 142,
103635.
Fleck, S., & Simon, G. (2013). An augmented reality environment for astronomy learning inelementary grades: An exploratory study. In M. Hachette, N. Russell (Eds), Proceedings of the25th Conference on l'Interaction Homme-Machine (pp. 14-22). ACM Press.
Kalemkus, J., & Kalemkus, F. (2022). Effect of the use of augmented reality applications on academic
achievement of student in science education: meta-analysis review. Interactive Learning
Environments, 1-18.
Karagozlu, D. (2021). Creating a sustainable education environment with augmented reality
technology. Sustainability, 13(11), 5851.
Karagozlu, D., Kosarenko, N., Efimova, O., & Zubov, V. (2019). Identifying students’ attitudes
regarding augmented reality applications in science classes. International Journal of Emerging
Technologies in Learning, 14(22), 45-55.
Kaya, O. S., & Bicen, H. (2019). Study of augmented reality applications use in education and its
effect on the academic performance. International Journal of Distance Education Technologies,
17(3), 25-36.
Kucuk, S., Yilmaz, R., Baydas, O., & Goktas, Y. (2014). Augmented reality applications attitude scale
in secondary schools: Validity and reliability study. Truth and Science, 39(176), 383-392.

326

Lelliott, A., & Rollnick, M. (2010). Big ideas: A review of astronomy education research 1974–
2008. International Journal of Science Education, 32(13), 1771-1799.
Liu, Q., Ma, J., Yu, S., Wang, Q., & Xu, S. (2023). Effects of an Augmented Reality-Based Chemistry
Experiential Application on Student Knowledge Gains, Learning Motivation, and Technology
Perception. Journal of Science Education and Technology, 32(2), 153-167.
Lorusso, M. L., Giorgetti, M., Travellini, S., Greci, L., Zangiacomi, A., Mondellini, M., Sacco, M., &Reni, G. (2018). Giok the alien: an AR-based integrated system for the empowerment ofproblem-solving, pragmatic, and social skills in pre-school children. Sensors, 18(7), 2368濁澳Monita, F. A., & Ikhsan, J. (2020). Development Virtual Reality IPA (VR-IPA) learning media forscience learning. Journal of Physics: Conference Series,1440 (1), 012103.
National Research Council. (1996). National science education standards. National Academies Press.
Osborne, J. F., Henderson, J. B., MacPherson, A., Szu, E., Wild, A., & Yao, S. Y. (2016). The
development and validation of a learning progression for argumentation in science. Journal ofresearch in science teaching, 53(6), 821-846.
Tian, K., Urata, M., Endo, M., Mouri, K., Yasuda, T., & Kato, J. (2019). Real-world-oriented
smartphone AR-supported learning system based on planetarium contents for seasonal
constellation observation. Applied Sciences, 9(17), 3508.
Wang, Y. (2022). Effects of augmented reality game-based learning on students’
engagement. International Journal of Science Education, Part B, 12(3), 254-270.
Wilgenbus, D., & Lena, P. (2009). Early science education and astronomy. In D. VallsGabaud, A.
Boksenberg (Eds), Proceedings of the International Astronomical Union (pp. 629-641).
Cambridge University Press.
Xu, W. W., Su, C. Y., Hu, Y., & Chen, C. H. (2022). Exploring the effectiveness and moderators ofaugmented reality on science learning: A meta-analysis. Journal of Science Education andTechnology, 31(5), 621-637.
Zhang, J., Li, G., Huang, Q., Feng, Q., & Luo, H. (2022). Augmented reality in K–12 education: A
systematic review and meta-analysis of the literature from 2000 to 2020. Sustainability, 14(15),9725.
Zhang, W., & Wang, Z. (2021). Theory and Practice of VR/AR in K-12 Science Education—ASystematic Review. Sustainability, 13(22), 12646.

327

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Preparation for Future Lockdowns: A
Comparison of Student LMS Activity
During and After COVID-19
Ezekiel Adriel LAGMAYa* & Maria Mercedes RODRIGOa
Ateneo Laboratory for the Learning Sciences, Ateneo de Manila University, The Philippines*ezekiel.lagmay@obf.ateneo.edu

aAbstract: We use CausalImpact analysis to compare student online activity during the
COVID-19 pandemic, when all classes were online, and post-pandemic, when classes
were offered in a mix of modes. Student activity was operationalized as activity logs
from the Canvas Learning Management System (LMS). When compared with student
activity in pandemic fully online classes, we found that, contrary to our hypotheses,
student activity in post-pandemic fully online classes decreased while activity in postpandemic fully onsite classes increased. We attribute the decrease in participation in
fully online classes to factors such as Zoom fatigue and feelings of loneliness andprolonged isolation. The increase in student online activity in fully onsite classes might
be explained by students’ general preference for face-to-face interactions. They may
have felt more connected to teachers and peers, leading to greater productivity overall.
The discourse about the transition from online learning brought on by the pandemic is
dominated by survey research. By examining student online activity in an LMS, thispaper contributes an analysis of empirical data that supports the findings of priorstudies about student perceptions of the online learning experience and theiradjustments to the next normal.
Keywords: Learning Management Systems, COVID-19 Emergency Remote
Teaching, CausalImpact

1. Introduction
In his keynote address during the International Conference for Computers in Education,
Baker (2022) stated that, despite advances in analytics and data mining, the field still lacks
theories “that could help build models robust to complexity and change. We still lackunderstanding of how change impacts models.” We have learned that models perform lesswell when the conditions under which they are applied change (see “Algorithmic Bias in
Education [Wiki],” n.d.). There are many possible causes of model degradation includingchanges in user interface design, differences in the way errors are counted, and differencesin the learning population over the years (Baker, 2022).
When the COVID-19 pandemic forced schools to close in early 2020, many of theprediction models and detectors stopped being useful because some of the attributes that
fed into them, e.g. student attendance data, were no longer being recorded or were recorded
in different ways. As we emerge from the pandemic, educational systems have retainedsome of these new policies and formats, such as a greater openness to hybrid formats. In
this next normal, can we still trust models that were built on pre-pandemic data? What value,
if any, can we derive from data collected during the pandemic?
We argue that pandemic data still offers us value because it helps us anticipate theeffects of lockdowns that may take place in the years to come. Climate change, newpandemics, civil unrest, and other factors may prompt educational systems to migrate online
temporarily and the return to face-to-face formats later on. In this paper, we examine
learning management system (LMS) activity from two time periods: during the pandemic,
when all classes were online, and when most restrictions were lifted and classes were328

offered in a variety of formats. We use CausalImpact analysis (Brodersen et al., 2015) todetermine differences in student LMS participation when all classes were fully online versus
when they were offered in fully onsite, fully online, and variants of the two modes. We
hypothesize that fully online student online participation will remain unchanged while student
LMS participation for onsite classes and mixed-mode classes will decline.

2. Related Literature
As the pandemic waned, many studies emerged trying to assess the differences in learnerperceptions of higher education teaching strategies, student course satisfaction, student
intention to continue using e-learning platforms, and factors affecting student engagement. A
study conducted at the University of Rajasthan in India (Sharma & Alvi, 2021) attempted todetermine differences in student perceptions of various teaching methods before and during
the pandemic. Using a questionnaire, they found that students had more positive opinions
about pre-pandemic, blended e-learning formats than the purely web-based learning thatthey experienced during lockdown.
These findings were corroborated by the US-based study of Corral and Fronza(2022). They found that student course satisfaction declined when courses moved online.
Student satisfaction ratings showed that satisfaction was high at the start of the pandemic.
As the pandemic extended, though, both student satisfaction and engagement waned. They
returned to normal levels when face-to-face classes resumed.
What accounts for the differences in satisfaction and engagement? Individual
differences were key factors in overall satisfaction. Students whose learning preferenceswere compatible with distance learning and with sufficient technical resources and support
services were more likely to have a more satisfying experience (Clary et al., 2022). Not all
students enjoyed this level of compatibility. Many expressed that online learning was lonely
and fostered feelings of laziness (Singh et al., 2021). Traditional learning, they said, madethem feel more productive. Many had to cope with considerable “lifeload”—the sum of allpressures a student has to confront (Hews et al., 2022). Those experiencing external
stresses engaged less than those who experienced greater well-being.
Moving forward, researchers found several types of student perceptions that
influence students’ intention to engage with distance learning again (Zacharis &Nikolopoulou, 2022). These included student perceptions of e-learning as enjoyable, that
their time and effort lead to worthwhile outcomes, the adequacy of technical and institutional
support, and the beliefs that important others such as peers and university mentors haveabout the students themselves.
The consensus from prior, recent research is that students are, in general, moresatisfied with face-to-face formats than online formats. While this finding may be consistent
with our own intuition and experience, all of these studies tended to rely on survey data.
They queried students regarding their perceptions, but they could not offer empirical data
about student online participation. This paper contributes to the discourse by analyzing
student participation as represented by LMS activity.

3. Methodology
3.1 Dataset Preprocessing
This study makes use of Canvas LMS log files collected from a university in Quezon City,
Metro Manila, Philippines during the Summer period of Academic Years (AY) 2021-2022 and2022-2023, which serve as the pre-intervention period and the post-intervention period
respectively. This is because the classes during Summer of AY 2021-2022 were all fully
online as opposed to the mixed-modality nature of Summer of AY 2022-2023. In addition,
the list of courses offered during the Summer of AY 2021-2022 and AY 2022-2023 weresimilar to each other. The Summer of AY 2021-2022 ran from June 28, 2021 to August 6,329

2021 (LS One Student Blueboard, 2021) while the Summer of AY 2022-2023 ran from June
13, 2022 to July 23, 2022 (Ateneo de Manila University Loyola Schools, 2022). The Canvas
logs were first trimmed down so that only the logs that were generated by the 6,519 students
and 737 teachers who were present in both Summer of AY 2021-2022 and Summer of AY
2022-2023 are included, and then are tallied overall per date, role type (students orteachers), and class ID, regardless of the event type each log pertains to.
In addition to the Canvas log files, this study also made use of two datasets, one for
the Summer of AY 2021-2022 and another for the Summer of AY 2022-2023, containing thelist of courses, their sections with their respective Canvas Class IDs, and the learningmodality used for the Summer of AY 2022-2023, which includes the following:z
Fully Online - All class activities were conducted online during all class days (J. Sugay,
personal communication, November 17, 2022).
z
Hybrid - Some class days were for online class activities, and others were for onsiteclass activities, for the entire class (J. Sugay, personal communication, November 17,2022).
z
Flex - All class activities included students who were onsite and online at the same time
during all class days (J. Sugay, personal communication, November 17, 2022).
z
Online+ - Most class activities were online with the exception of special activities (suchas exams, presentations, laboratory sessions, etc.) which were onsite (J. Sugay,
personal communication, November 17, 2022).
z
Fully Onsite - All class activities were onsite (face-to-face classes) during all class days.
In order to simplify the assumptions, we only included courses offered during both
Summer of AY 2021-2022 and Summer of AY 2022-2023. Furthermore, all sections of any
given course offered during Summer of AY 2022-2023 had to follow the same modality.
One of the critical decisions we had to make was the choice of pre-intervention
dataset. As mentioned earlier, we had five post-intervention cases: Fully Online, Hybrid,Flex, Online+, and Fully Onsite. The post-intervention dataset for each of these courses wascomposed of data from all classes offered in these modes. We decided that the preintervention dataset of each of these cases would be composed of the same courses offeredduring the pandemic. To illustrate, suppose that the following were the courses offeredduring the pandemic and post-pandemic periods as shown in Table 1.
Table 1. Courses Offered During the Pandemic and Post-Pandemic Periods (Illustration)
Pandemic Courses (All
Online)
A
B
C
D
E
F
G
H
I
J

Post-Pandemic Courses

Post-Pandemic Modality

A’
B’
C’
D’
E’
F’
G’
H’
I’
J’

Fully Online
Fully Online
HybridFlex
Online+Fully Onsite
Fully Onsite
Flex
HybridOnline+The pre-intervention and post-intervention datasets would therefore be as shown in
Table 2.
Table 2. Pre-Intervention and Post-Intervention Datasets Based on Table 1 (Illustration)
Post-Pandemic Modality
Fully Online
HybridFlex

Pre-Intervention Dataset
A, B
C, I
D, H

330

Post-Intervention Dataset
A’, B’
C’, I’
D’, H’

Online+Fully Onsite

E, J
F, G

E’, J’F’, G’

Note that for this analysis, we did not take into consideration the differences between
courses, nor the differences in course contents. The final breakdown of the number ofcourses and their respective number of sections/classes studied are shown in Table 3.
Table 3. Breakdown of the Number of Courses Analyzed and Their Classes
Modality
Fully Online
HybridFlex
Online+Fully Onsite

Total Courses
74
33
4
9
2

Total Classes (2021)
126
66
8
17
2

Total Classes (2022)
133
50
5
17
2

The aggregated Canvas logs and the list of courses were then split according to theacademic period. For each academic period, the list of courses was first merged into theaggregated Canvas logs. Then, the resulting dataset was again aggregated according todate, user type, and modality. The total number of classes per modality for the academic
period (see Total Classes (2021) and Total Classes (2022) columns in Table 3) was
obtained from the list of courses, and was divided from the total number of logs calculated
according to date, user type, and modality, and then discretized using a ceiling function to
get the average number of logs per class for that date, user type, and modality. Lastly, the tcolumn represents the number of days elapsed. The values for the Summer period of AY
2021-2022 are computed by subtracting each date d from August 7, 2021, the day after thelast day of Summer AY 2021-2022 (LS One Student Blueboard, 2021). The values for theSummer period of AY 2022-2023 are computed by subtracting each date d from June 13,2022, the first day of Summer AY 2022-2023 (Ateneo de Manila University Loyola Schools,2022). The t-column therefore has negative values for the Summer period of AY 2021-2022,
0 for June 13, 2022, and positive values for the Summer period of AY 2022-2023. The
datasets were then appended back together, then split into separate CSV files (10 in total)
according to user type and modality in preparation for CausalImpact analysis using R. Prior
to the analysis proper, we take one plus the absolute value of the lowest negative t-column
value (that is, pertaining to the first day of Summer period of AY 2021-2022, which is June
28, 2021) (LS One Student Blueboard, 2021), and add it to all of the t-column values in order
to conform with the requirement in CausalImpact that t-values need to be positive,
continuous, and chronological.

3.2 CausalImpact Analysis
For each modality, CausalImpact was performed with the student data as the outcome
variable and the teacher data as the predictor variable. CausalImpact is a type of causal
inference that estimates the impact of an intervention (e.g. transition from fully online classes
to a variety of learning modalities) using Bayesian structural time series models (Brodersen,
2014; Brodersen et al., 2015; Lagmay & Rodrigo, 2022). Furthermore, in contrast to someother popular causal inference methods like Interrupted Time Series or Prophet (further
explained in (Lagmay & Rodrigo, 2022) alongside other alternatives), CausalImpact offers
both better predictions and easier interpretations (Kuromiya et al., 2020), although one
limitation of it is that it is “only able to observe the outcomes under the treatment for one timeseries and under the control for the treatment for another one, but not the potential outcome
under control for the former and under treatment for the latter” (Li & Bühlmann, 2020).
CausalImpact takes in the predictor and outcome variables, as well as the pre- and postintervention time segments in order to model the relationship between the predictor andoutcome variables using the pre-intervention data, estimate the post-intervention

331

counterfactual, and give the impact of the intervention by measuring the difference between
the counterfactual and the observed post-intervention data (Lagmay & Rodrigo, 2022).
We used the teacher data as the predictor variable following the Teacher Expectancy
Effect or Pygmalion Effect which states that teacher expectations have an impact on
students’ academic performance (Szumski & Karwowski, 2019). A lengthier discussion forour rationale is available in (Lagmay & Rodrigo, 2022). The period pertaining to Summer AY
2021-2022 was designated as the pre-intervention period, while Summer AY 2022-2023 was
the post-intervention period.

4. Results and Discussion
4.1 Basic Statistical and Descriptive Analysis
Figures 1 and 2 show the overall average number of logs per class generated by students
and teachers, respectively, per modality type and time period, in order to illustrate thedifference in LMS usage behavior by user type and modality. For the students data, only theclasses using the Fully Onsite modality experienced an increase in the average number ofCanvas logs per class (95.54%). This is despite the fact that average teacher activity for thesame classes decreased by -29.62%. In contrast, teacher activity in the Flex and Online+
classes saw an increase of 4.93% and 47.88%, respectively.

Figure 1. Overall Average Number of Logs per Class Generated by Students.

Figure 2. Overall Average Number of Logs per Class Generated by Teachers.

4.2 Fully Online Classes (Control Group)
Figure 3 shows that the average number of logs per class for Fully Online classes
significantly decreased during the Summer of AY 2022-2023 with p = 0.014. An explanationof the graphs shown in Figures 3 to 7 is as follows:“Each unit on the x-axis represents one day in the time period. The topmost graphlabeled “original” shows a solid line representing the actual observed data, i.e., the numberof transactions per day. The broken line represents the prediction. The light blue band
represents the confidence interval of the prediction. The middle graph labeled “pointwise”

332

shows the difference between the predicted number of transactions and the actual numberof transactions per day….Finally, the cumulative graph at the bottom shows the accumulateddifference between the predicted number of transactions and the actual number oftransactions….The [vertical gray dashed line] is the intervention [time]. There is no
accumulated difference during the pre-intervention period. The differences are accumulated
post-intervention.” (Lagmay & Rodrigo, 2022)
The average value obtained was 98.70 as opposed to the counterfactual prediction of
115.28. The overall estimated effect was -16.58 with a 95% confidence interval of [-31.83, 1.78]. When the data points during the intervention period are summed, the response
variable had an overall value of 3,950, as opposed to the counterfactual prediction of 4,610
with a 95% interval of [4,020, 5,220]. Average Canvas logs per class decreased by -14%
with a 95% confidence interval of [-24%, -2%].

Figure 3. CausalImpact Graph for Fully Online Classes.

4.3 Hybrid Classes (Experimental Group 1)
Figure 4 shows that the average number of logs per class for Hybrid classes decreased
during the Summer of AY 2022-2023, but not significantly. The average value obtained was
102.83 as opposed to the counterfactual prediction of 117.57. The overall estimated effectwas -14.74 with a 95% confidence interval of [-35.91, 5.68]. When the data points during theintervention period are summed, the response variable had an overall value of 4,110, as
opposed to the counterfactual prediction of 4,700 with a 95% interval of [3,890, 5,550].
Average Canvas logs per class decreased by -12% with a 95% confidence interval of [-26%,+6%]. The difference in the signs meant that the decrease with p = 0.07 was not significant.

Figure 4. CausalImpact Graph for Hybrid Classes.

333

4.4 Flex Classes (Experimental Group 2)
Figure 5 shows that the average number of logs per class for Flex classes significantly
decreased during the Summer of AY 2022-2023 with p = 0.001. The average value obtained
was 149.68 as opposed to the counterfactual prediction of 227.76. The overall estimatedeffect was -78.09 with a 95% confidence interval of [-113.40, -43.26]. When the data points
during the intervention period are summed, the response variable had an overall value of
5,990, as opposed to the counterfactual prediction of 9,110 with a 95% interval of [7,720,
10,520]. Average Canvas logs per class decreased by -34% with a 95% confidence intervalof [-43%, -22%].

Figure 5. CausalImpact Graph for Flex Classes.

4.5 Online+ (Experimental Group 3)
Figure 6 shows that the average number of logs per class for Online+ classes significantly
decreased during the Summer of AY 2022-2023 with p = 0.001. The average value obtained
was 69.85 as opposed to the counterfactual prediction of 137.58. The overall estimatedeffect was -67.73 with a 95% confidence interval of [-86.15, -51.62]. When the data points
during the intervention period are summed, the response variable had an overall value of
2,790, as opposed to the counterfactual prediction of 5,500 with a 95% interval of [4,860,
6,240]. Average Canvas logs per class decreased by -49% with a 95% confidence interval of[-55%, -42%].

Figure 6. CausalImpact Graph for Online+ Classes.

4.6 Fully Onsite Classes (Experimental Group 4)334

Figure 7 shows that the average number of logs per class for Fully Onsite classes
significantly increased during the Summer of AY 2022-2023 with p = 0.001. The averagevalue obtained was 122.88 as opposed to the counterfactual prediction of 57.44. The overall
estimated effect was 65.43 with a 95% confidence interval of [49.51, 80.58]. When the data
points during the intervention period are summed, the response variable had an overallvalue of 4,920, as opposed to the counterfactual prediction of 2,300 with a 95% interval of[1,690, 2,930]. Average Canvas logs per class increased by +117% with a 95% confidence
interval of [+67%, +191%].

Figure 7. CausalImpact Graph for Fully Onsite Classes.

5. Conclusion
We used CausalImpact analysis to examine differences in student LMS participation when
all classes were Fully Online versus when they were offered in Fully Onsite, Fully Online,
Hybrid, Flex, and Online+ modes. We hypothesized that fully online student online
participation would remain unchanged and that participation in all other modes would
decline. As hypothesized, online participation in Flex and Online+ classes declined while
online participation in Hybrid classes did not differ significantly from activity in pandemic fully
online classes. What was surprising though was that participation in Fully Online classes
declined while online participation in Fully Onsite classes increased, and these are thefindings we would like to unpack further.
From prior literature, we draw several possible explanations for these outcomes. As
mentioned in the review of literature, students generally were unsatisfied with the online
learning format. They found it isolating and lonely (Singh et al., 2021). They much preferredface-to-face classes or modes that mixed online and onsite formats (Sharma & Alvi, 2021).
They suffered “Zoom fatigue” and lacked interactions with peers and mentors (Corral &Fronza, 2022). Working online made them feel less productive (Singh et al., 2021).
Furthermore, classes that were purely asynchronous made students feel less capable ofunderstanding course content (Hews et al., 2022). To continue online meant to continue toexperience these feelings, all of which may have diminished interest and engagement.
When given the opportunity to work onsite, students felt more productive (Singh etal., 2021). They said that in-person classes with teacher and peers contributed to greater
focus and engagement (Hews et al., 2022). They could ask questions as these arose.
The main limitation of this study is that it relies purely on LMS data. It does not haveany survey data or qualitative feedback that might help explain the results, and we did notconsider the differences among courses nor the learning design utilized by the courses.
While references to prior work are useful, the populations across all these studies aredifferent and therefore differences in student experiences may exist. Future work shouldattempt to collect and analyze more qualitative feedback to weave a fuller narrative
explaining pandemic versus post-pandemic student online activity. Also, a more fine-grained
analysis such that the logs are aggregated according to event type (e.g. assignments,335

quizzes, etc.) should be considered in order to determine which specific parts of the LMS
actually increased or decreased depending on the modality.
The other limitation is that the study does not capture the case where students wereenrolled in classes with different modalities. Many students had a mix of onsite, online, andmixed-mode classes, and the impact of this intermingling on their overall participation andengagement was not considered in this analysis. In addition, we did not consider the
possibility that some students may have failed a course and had to retake it, which may also
affect the results. However, such data is not readily available from the Canvas log files
alone. Lastly, this study did not take into consideration previous studies on fully online andhybrid courses prior to the pandemic, and as well studies related to the development ofdigital skills which may help further explain the results.
Despite these limitations, this study contributes to the discourse in at least three
ways. It offers an analysis of empirical data from an underrepresented country. It illustrates
quantitatively how student participation changed from the pandemic to the post-pandemic
period. These changes are a cue to data scientists to think about how to build more robustmodels that generalize in the fact of major events such as COVID-19. Overall, the findings
give educators insight as to what to expect when the next lockdown occurs, and how wemight better prepare and recover from it.

Acknowledgements
The authors would like to thank the Ateneo Laboratory for the Learning Sciences and Ateneode Manila University Research Council for the funding and support needed for this research.
The authors would also like to thank Deni Jaramillo and Miguel Saavedra for their assistancein setting up the necessary servers for the collection of Canvas data, and as well Deni
Jaramillo and Joannie Ereño for providing the other datasets required for this study. Lastly,
the authors would like to thank Jessica Sugay for her explanation on the different learningmodalities.

References
Algorithmic Bias in Education [Wiki]. (n.d.). Retrieved December 22, 2022, fromhttps://www.pcla.wiki/index.php/Algorithmic_Bias_in_Education
Ateneo de Manila University Loyola Schools. (2022, May 31). Academic calendar school year 20222023 [PDF file]. Ateneo de Manila University. https://www.ateneo.edu/sites/default/files/202206/LS%20Academic%20Calendar%20SY2022-2023%20as%20of%2031%20May%202022.pdf
Baker, R. S. (2022, November 30). When might a detector generalize? [Keynote Session]. 30th
International Conference for Computers in Education, Kuala Lumpur, Malaysia.
https://www.youtube.com/live/mFJCmryXQZA?feature=share&t=6408
Brodersen, K. (2014, September 10). CausalImpact: a new open-source package for estimating
causal effects in time series. Google Open Source Blog.
https://opensource.googleblog.com/2014/09/causalimpact-new-open-source-package.html
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring causal impact
using bayesian structural time-series models. Annals of Applied Statistics, 9(1), 247-274.
https://doi.org/10.1214/14-AOAS788
Clary, G., Dick, G., Akbulut, A. Y., & Van Slyke, C. (2022). The after times: college students’desire to continue with distance learning post pandemic. Communications of the
Association for Information Systems, 50(1), 122-142. http://dx.doi.org/10.17705/1CAIS.05003
Corral, L., & Fronza, I. (2022). It’s great to be back: an experience report comparing course
satisfaction surveys before, during and after pandemic. In Proceedings of the 23rd
Annual Conference on Information Technology Education (pp. 66-72).
Hews, R., McNamara, J., & Nay, Z. (2022). Prioritising lifeload over learning load: understanding
post-pandemic student engagement. Journal of University Teaching & Learning Practice, 19(2),128-146. https://doi.org/10.53761/1.19.2.9
Kuromiya, H., Majumdar, R., Kondo, T., Nakanishi, T., Takii, K., & Ogata, H. (2020). Impact of school
closure during COVID-19 emergency: a time series analysis of learning logs. In 28th International

336

Conference on Computers in Education Conference Proceedings (Vol. 1, pp. 272–277). AsiaPacific Society for Computers in Education (APSCE).
Lagmay, E. A. D., & Rodrigo, M. M. T. (2022). The impact of extreme weather on student online
learning participation. Research and Practice in Technology Enhanced Learning, 17(26), 1-21.
https://doi.org/10.1186/s41039-022-00201-2
Li, S., & Bühlmann, P. (2020). Estimating heterogeneous treatment effects in nonstationary timeseries with state-space models. In Seminar for Statistics. ETH Zürich.
LS One Student Blueboard. (2021, April 22). Academic calendar for intersession S.Y. 2020-2021[Photo]. Facebook. https://www.facebook.com/AteneoSBB/photos/pb.100057350834446.2207520000./4107831912570510/?type=3
Sharma, A., & Alvi, I. (2021). Evaluating pre and post COVID 19 learning: an empirical study oflearners’ perception in higher education. Education and Information Technologies, 26(6), 70157032. https://doi.org/10.1007/s10639-021-10521-3
Singh, J., Steele, K., & Singh, L. (2021). Combining the best of online and face-to-face learning:hybrid and blended learning approach for COVID-19, post-vaccine, & post-pandemic world.
Journal of Educational Technology Systems, 50(2), 140-171.
https://doi.org/10.1177/00472395211047865
Szumski, G., & Karwowski, M. (2019). Exploring the pygmalion effect: the role of teacher
expectations, academic self-concept, and class context in students’ math achievement.
Contemporary Educational Psychology, 59, 101787.
https://doi.org/10.1016/j.cedpsych.2019.101787
Zacharis, G., & Nikolopoulou, K. (2022). Factors predicting university students’ behavioral intention touse eLearning platforms in the post-pandemic normal: an UTAUT2 approach with ‘learning
value’. Education and Information Technologies, 27, 12065–12082.
https://doi.org/10.1007/s10639-022-11116-2

337

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Using Augmented Reality to Facilitate
Music Learning for Preschool Children
Xiaodong WEIa*, Rui QIUa & Ruixue LIUa
School of Educational Technology, Northwest Normal University, China*wxd1633@163.comaAbstract: New technologies such as Augmented Reality (AR) have been widely usedin education fields. AR’s deep integration of virtual objects and real environments
meets the diverse learning needs of learners. However, relatively few studies have
explored the application of AR in preschool music education. Therefore, this study
developed a mobile AR application for preschool children in learning music andinvestigated its impact on children’s learning achievement and enjoyment. We used amixed research method, measuring academic performance through pre-test and posttest, a questionnaire for children’s enjoyment, and interviews to determine children’s
views of AR usage in music learning. Sixty participants in two kindergarten classes
participated in this study. The children in the experimental group used mobile-based
AR applications for music learning, while the control group learned the same content
through the traditional teaching method. The results showed that compared with thetraditional teaching method, mobile AR has significantly improved children’s academic
performance and enjoyment. In addition, the interview results also showed that thechildren considered AR learning activities to be very interesting and that they are willingto use AR for music learning in the future. The conclusions provide a concrete practicalcase on how to effectively integrate AR into preschool music education.
Keywords: Augmented reality, preschool children, music learning, academic
achievement, enjoyment

1. Introduction
Music is important for promoting children’s comprehensive and harmonious development(Klim-Klimaszewska, 2015). Generally, music learning is mainly conducted through visualand auditory activities, and traditional teaching methods (e.g., singing) (Francesconi et al.,2013). However, He (2019) proposed that traditional ways of teaching music are usually theteacher sings first, and then children sing alone, which is unattractive and boring. Previous
studies have proposed that Augmented Reality (AR) can be regarded as strategies toimprove music teaching levels (Lu et al., 2022). Therefore, incorporating AR in music
learning provides the potential to promote learners’ effective learning.
AR is a technology that integrates the real environment with the virtual environment
which provides learners with an interactive learning experience (Azuma, 1997). AR
superimposes designed virtual objects on the real environment through various technologies
to achieve the effect of virtual and real fusion (Karagozlu et al., 2019). Cote and Diaz (2017)
believed that AR plays an active role in music education, and the AR music teachingmaterials produced can cultivate students’ autonomy. However, few studies explored theapplication of AR in preschool music education. Therefore, this study aims to integrate AR
into musical learning and explore its impact on children’s academic performance andenjoyment. The following research questions were as follows:z
Is there a significant difference in the academic performance of children who use AR
and the traditional teaching method?
z
Is there a significant difference in the enjoyment of children who use AR and the
traditional teaching method?
zWhat are the children’s views on using AR in music learning activities?

338

2. Background
2.1 Music Learning in Early Childhood Education
Many studies have shown the importance of music learning in the process of early childhood
development. Pearce and Rohrmeier (2012) proposed a link between music and cognitive
science in their study. The results showed that music is a universal feature of all people, andplays a vital role in our daily lives, as well as in different stages of human development. Qiao
(2016) proposed that music education is a practical activity with rich emotions that canpromote the emotional development of preschool children. Moreover, music learning canstimulate preschool children’s imagination, improve creativity, activate thinking andintelligence (Klim-Klimaszewska, 2015), promote social interaction, cultivate a good sense ofsocial responsibility (Hallam, 2010), and develop positive social values (Ozturk & Can,2020).

2.2 AR in Music Learning
Some scholars have used AR music systems for music therapy. For example, Correa et al.,(2009) showed that AR music learning system can provide therapeutic intervention incognitive, motor, and psychological aspects to help children with cerebral palsy recover.
Correa et al., (2017) showed that the AR music system was beneficial to the rehabilitation ofchildren with disabilities. Moreover, Huang et al. (2011) created an AR-based markerless
piano teaching system for piano beginners, their results demonstrated that the AR systemincreased the sense of accomplishment and interest. Lu et al. (2022) found that using AR tolearn chord knowledge in music theory has a positive impact on children’s learning andcreativity. The above-mentioned studies demonstrated the advantages of AR in music
therapy and education. However, research on the effects of using AR for music learning inearly childhood education has not been explored.

3. Development of a Mobile-based AR Music Learning Application
3.1 Learning Material
In this study, we used the Unity game engine to develop an application that can be installedon the Android system. Variety of technologies, such as cube recognition, and multi-image
recognition was utilized to develop the AR system. The learning material is the “Forest
Kingdom Concert”, which aims to improve children’s singing skills in contextualized learning.
There are three main steps, and the specific learning content is shown in Fig 1.

Figure 1. The Home Page of the Learning Activity.

339

3.2 Activities for Numbered Musical Notation and Musical Scales
Before learning the numbered musical notation, we selected five different heights of cubes
as the identification objects (see Fig 2a). It presented the numbered musical notation of “Do,Re, Mi, Fa, Sol” respectively. When the children scan the yellow-black cube with a mobile
phone (see Fig 2b), the 3D virtual model will run out from behind the cube and rotate aroundthe cube, making a “Hello, I’m Fa” sound. Children can control the 3D virtual model throughoperations (zooming in, zooming out, rotating, etc.). During the musical scale learning
process, we used two bar images with different heights and colors to present musical scales:“Do, Re, Mi, Fa, Sol” and “Sol, Fa, Mi, Re, Do” (see Fig 2c). When children scan the musicalscale picture, the 3D virtual object is superimposed on the picture, accompanied by thesound of musical scales (see Fig 2d). The two learning activities each last 35 minutes.

Figure 2. Numbered Musical Notation and Musical Scale Learning Activity Screen.

3.3 Activities for Sing Learning
In the singing learning activity (see Fig 3), the animal’s feet for the 3D virtual object coincidewith the footprints of the image to achieve the effect of virtual and real fusion. Children canfreely observe and manipulate the 3D virtual object. For example, they can zoom in or out ofa 3D virtual object with two fingers and rotate them with one finger (see Fig 3b). After that,the children can click the animal buttons, and click on the animal’s footprint (virtual button) tointeract with a 3D virtual object (see Fig 3a). At this time, the lamb’s singing video shown on
the picture for children to learn (see Fig 3c). This learning activity lasts 35 minutes.

Figure 3. Singing Learning Activity Screen.

4. Methods
4.1 Research Design and Participants
In this study, we used a mixed-study method. Before the experiment, both groups of children
took a pre-test determining that they had equal prior knowledge levels. Following that, theexperimental group used AR to learn music, and the control group learned throughtraditional methods. After the experiment, a post-test and enjoyment questionnaire wasadministered to the two groups of children, and a semi-structured interview was conducted.
Participants came from two large classes in a kindergarten in northwestern China, each with
30 children, an average age of 6. The children in both groups were taught by the sameteacher and were familiar with mobile device.

340

4.2 Measurement Tools
4.2.1 Learning Achievement Tests
The content of the pre and post-test were the same and was developed by the researcher.
The pre-test aimed at assessing whether the children had the same knowledge level before
the experiment. The post-test explores whether children who use AR apps for music learningshow better academic performance compared to the traditional method. Both pre and posttest questions consist of 10 items, consisting of 5 choice questions, and 5 yes or no items,with a maximum score of 100. For example, the choice question is: “Please select the
numbered musical notation of Do”. The reliability of both tests was assessed using
Cronbach’s alpha value, which is greater than 0.80, indicating good internal consistency.
4.2.2 Enjoyment Scale
The enjoyment scale was used in this study based on Dalim et al. (2020). It consistedof 7 items (e.g., “I enjoyed doing this activity very much”), assessed using a 5-point Likert
scale ranging from 1 (strongly disagree) to 5 (strongly agree). To measure happiness, weuse a smiley image instead of the Likert scale’s answer categories. The scale is used bypreschool children to find emojis that fit their answers with their fingers, and all are filled out
by the teacher. The Cronbach’s alpha values for the enjoyment scale were above 0.83,
indicating good reliability.
4.2.3 Semi-structured Interviews
The semi-structured interview focuses on understanding preschool children’s views of the
AR technology learning experience. In this semi-structured interview, questions include
children’s feelings and willingness to use AR applications.

4.3 Procedure
The experiment lasted for about three weeks. In the first week, we pre-tested two groups ofchildren. Moreover, the children in the experimental group were required to be familiar withthe operation of AR. In the second week, children in the experimental group learned music
through AR, while the control group learned the same material through the traditional
method. In the third week, post-test and enjoyment questionnaires were administered to bothgroups of preschool children. Preschool children in the experimental group also conductedsemi-structured interviews.

5. Result
5.1 Learning Achievement
As shown in Table 1, the M and SD of the pre-test scores were 29.93 and 10.26 for theexperimental group, and 29.50 and 11.24 for the control group, respectively. The t-test
results (t = 0.16, p >0.05) showed that there was no significant difference between the twogroups. Therefore, both groups had the same prior knowledge before learning the activity.
After the experiment, the M and SD of the post-test were 87.03 and 6.84 for theexperimental group, and 83.50 and 5.38 for the control group, respectively. The t-test results(t = 2.22, p < 0.05) showed significant differences between the two groups, indicating thatchildren who use AR applications in music learning showed better academic performancecompared to children with the traditional teaching method.

341

Table 1. T-test Result of the Learning Achievement for Both Groups
Test

Group
Experimental group
Control group
Experimental group
Control group

Pre-test
Post-test

N
30
30
30
30

M
29.93
29.50
87.03
83.50

SD
10.26
11.24
6.84
5.38

F

t

0.42

0.16

1.02

2.22*

*p < 0.05.

5.2 Enjoyment
Table 2 shows the t-test results of the enjoyment, the M and SD were 4.22 and 0.57 for theexperimental group and 3.85 and 0.69 for the control group, respectively. The t-test results (t= 2.28, p < 0.05) showed that AR applications for music education had a significant impact
on the enjoyment of preschool children.
Table 2. T-test Results of the Enjoyment for Both Groups
Item
Enjoyment

Group
Experimental group
Control group

N
30
30

M
4.22
3.85

SD
0.57
0.69

F

t

0.91

2.28*

*p < 0.05.

5.3 Interview
From the interview results, most children believed that AR has a positive effect on theirmusic learning. Among them, 80.33% of children considered that AR music lessons are very
interesting, and learning content makes them feel very happy. 90.33% of children like to use
AR for music learning. 82% of children are willing to continue using AR for learning in the
future. Here are a few examples taken from interviews with children: “I hope the teacher will
let us play AR games next time.” “I like the little animals in AR.”

6. Discussion and Conclusion
In this study, we developed a mobile-based AR application and investigated its effect onpreschool children’s music learning. Results found that the children who use AR in music
learning achieved better academic performance and enjoyment compared to the traditional
teaching method. Moreover, Through the interview, it can be found that the children gave
positive feedback on the use of AR in music learning. Regarding achievement, the findings
are consistent with previous research findings (Lu et al., 2022). AR increases the fun andprovides preschool children with an interactive experience. In addition, in terms ofenjoyment, it can also be concluded that the experimental children have higher enjoyment inmusical activities than the control group. This is similar to previous research findings (Aladin
et al., 2020). In addition, interview results showed that many children believe that AR
technology has a positive impact on their music learning, and they enjoy using AR for music
learning and look forward to using AR to learn music in the future. AR provides learners withan interactive teaching process and rich learning forms and provides a reference foreducators who implement activities in similar fields.
However, this study also has some limitations. First, the children’s influence on thenovelty of AR technology may contribute to children’s high satisfaction. Second, some
practical activities may be carried out, allowing teachers to lead children to interact with eachother. Third, the teaching time of this experiment is relatively short. Future research should
enlarge more related knowledge into the music curriculum to form a unified teaching.

342

Acknowledgments
The research was supported by the National Natural Science Foundation of China [grant
number 62067009], and the 2023 Gansu Province University Youth Doctoral Support Project(No. 2023QB-110).

References
Aladin, M. Y. F., Ismail, A. W., Salam, M. S. H., Kumoi, R., & Ali, A. F. (2020). AR-TO-KID: A speechenabled augmented reality to engage preschool children in pronunciation learning. In M. R. B. A.
Kadir, A. W. Ismail (Eds), Proceedings of IOP conference series: Materials science andengineering (pp. 012011). IOP Press.
Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual environments,
6(4), 355-385.
Correa, A. G. D., Ficheman, I. K., do Nascimento, M., & de Deus Lopes, R. (2009). Computerassisted music therapy: A case study of an augmented reality musical system for preschool
children with cerebral palsy rehabilitation. In I. Aedo, NS. Chen, D. Sampson, L. Zaitseva, X.
Kinshunk (Eds), 2009 Ninth IEEE International Conference on Advanced Learning Technologies(pp. 218-220). IEEE Press.
Correa, A. G. D., de Assis, G. A., do Nascimento, M., & de Deus Lopes, R. (2017). Perceptions ofclinical utility of an Augmented Reality musical software among health care professionals.
Disability and Rehabilitation: Assistive Technology, 12(3), 205-216.
Cote, L. P. A., & Diaz, J. S. S. (2017). Evaluation of the use of augmented reality in music education.
Music notebooks, visual arts and performing arts, 12(1), 143–157.
Dalim, C. S. C., Sunar, M. S., Dey, A., & Billinghurst, M. (2020). Using augmented reality with speech
input for non-native preschool children’s language learning. International Journal of HumanComputer Studies, 134, 44-64.
Francesconi, J. I., Larrea, M. L., & Manresa-Yee, C. (2013). Tangible music composer for preschool
children. Journal of Computer Science and Technology, 13(02), 84-90.
Hallam, S. (2010). The power of music: Its impact on the intellectual, social, and personal
development of preschool children and young people. International journal of music education,
28(3), 269-289.
He Y. Y. (2019). The Function of Music Education in Preschool children. In J. Gao (Eds), Proceedingsof 2019 International Conference on Arts, Management, Education, and Innovation (ICAMEI2019) (pp.1218-1221). Clausius Scientific Press.
Huang, F., Zhou, Y., Yu, Y., Wang, Z., & Du, S. (2011). Piano ar: A markerless augmented reality
based piano teaching system. In Hangzhou (Eds), Proceedings of 2011 Third international
conference on intelligent human-machine systems and cybernetics (pp. 47-52). IEEE Press.
Karagozlu, D., Kosarenko, N., Efimova, O., & Zubov, V. (2019). Identifying students’ attitudes
regarding augmented reality applications in science classes. International Journal of Emerging
Technologies in Learning (IJET), 14(22), 45-55.
Klim-Klimaszewska, A. (2015). Music influence on preschool children’ imagination development at thekindergarten age. In L.G. Chova, A.L. Martinez, I.C. Torres (Eds), Proceedings of ICERI 2015
(pp. 3267-3272). IATED Press.
Lu, Y., Wang, X., Gong, J., & Liang, Y. (2022). ChordAR: An Educational AR Game Design for
Preschool children’s Music Theory Learning. Wireless Communications and Mobile
Computing, 2022.
Ozturk, E., & Can, A. A. (2020). The Effect of Music Education on the Social Values of Preschool
children. Cypriot Journal of Educational Sciences, 15(5), 1053-1064.
Pearce, M., & Rohrmeier, M. (2012). Music cognition and the cognitive sciences. Topics in cognitive
science, 4(4), 468-484.
Qiao, Q. (2016). Value of Music Education to the Development of Preschool children. In H. Li, L.
Zhang (Eds), Proceedings of 2016 5th International Conference on Social Science, Educationand Humanities Research (SSEHR 2016) (pp. 171-174). Atlantis Press.

343

&RPSDULQJ 3HUFHLYHG &RJQLWLYH /RDG ZKLOH/HDUQLQJ 2QOLQH ZLWK $, &KDWERWV3UHUHFRUGHG 9LGHRV DQG /LYH /HFWXUHV+DL[L 6KHQJD ;LQUDQ =KRXE <XH =KDRF *XRTLQJ =KDRG
DEFG)DFXOW\ RI (GXFDWLRQ %HLMLQJ 1RUPDO 8QLYHUVLW\ &KLQD
JXRTLQJ]K#EQXHGXFQ$EVWUDFW 7KH &RJQLWLYH /RDG 7KHRU\ SURYLGHV DQ LPSRUWDQW SHUVSHFWLYH IRU REVHUYLQJ
OHDUQLQJ ZLWK GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV 6LQFH PRUH DQG PRUH SURIHVVLRQDOGHYHORSPHQW SURJUDPV DUH FDUULHG RXW RQOLQH IRU LQVHUYLFH WHDFKHUV LW LV FUXFLDO WR
H[SORUH WKHLU SHUFHLYHG FRJQLWLYH ORDG EURXJKW E\ GLIIHUHQW RQOLQH WHFKQRORJLHV GXULQJ
RQOLQH OHDUQLQJ 7KLV VWXG\ DLPHG WR H[DPLQH WKH ([WUDQHRXV &RJQLWLYH /RDG (&/ DQG*HUPDQH &RJQLWLYH /RDG *&/ RI WHDFKHUV ZKHQ OHDUQLQJ ZLWK WKUHH RQOLQH OHDUQLQJ
WHFKQRORJLHV $, FKDWERW SUHUHFRUGHG YLGHRV DQG OLYH OHFWXUH  $OWRJHWKHU WHDFKHUV IURP GLIIHUHQW DUHDV DQG VFKRROV SDUWLFLSDWHG LQ WKH VWXG\ E\ HQUROOLQJ RQ WKH
GD\ RQOLQH OHDUQLQJ WUDLQLQJ DERXW YLVXDO WKLQNLQJ WRROV 7KH SDUWLFLSDQWV UHSRUWHG
WKH\ SHUFHLYHG (&/ DQG *&/ XVLQJ D VHOIUHSRUWHG VFDOH 7KH UHVXOWV VKRZHG WKDW WKH\
SHUFHLYHG VLJQLILFDQWO\ ORZHU (&/ GXULQJ OHDUQLQJ ZLWK SUHUHFRUGHG YLGHRV WKDQ ZLWK $,FKDWERWV DQG OLYH OHFWXUHV 0HDQZKLOH QR VLJQLILFDQW GLIIHUHQFHV DSSHDUHG LQ WKH *&/DPRQJ WKHVH WKUHH RQOLQH OHDUQLQJ WHFKQRORJLHV.H\ZRUGV 2QOLQH /HDUQLQJ 3URIHVVLRQDO 'HYHORSPHQW (&/ *&/ $, &KDWERW

 ,QWURGXFWLRQ
7HFKQRORJ\ KDV FKDQJHG KRZ ZH OHDUQ DQG WHDFK ,QLWLDOO\ LW ZDV WKH SUHUHFRUGHG YLGHROHFWXUH UHSUHVHQWHG E\ 022& GRPDLQ WKH KLJKODQG RI RQOLQH OHDUQLQJ 'XULQJ WKH &29,'
SDQGHPLF OLYH OHFWXUHV UHSUHVHQWHG E\ 7HQFHQW 0HHWLQJ DQG =220 KDYH EHFRPH WKHPDLQVWUHDP IRUP RI WHDFKLQJ 5HFHQWO\ WKHUH LV D QHZ ZD\ RI WHDFKLQJ XVLQJ $, FKDWERWVZKLFK DUH OLNH WDONLQJ URERWV +RZHYHU GLIIHUHQW RQOLQH OHDUQLQJ PHWKRGV KDYH GLIIHUHQW HIIHFWVRQ OHDUQHUV 7KLV VWXG\ ORRNV DW KRZ WKHVH PHWKRGV DIIHFW WHDFKHUV¶ WKLQNLQJ DQG OHDUQLQJ 7R
XQGHUVWDQG WKLV HIIHFW ZH HPSOR\ WKH &RJQLWLYH /RDG 7KHRU\ &/7 
7KH &/7 WKHRU\ VD\V WKDW ZKHQ ZH OHDUQ VRPHWKLQJ RXU EUDLQ KDV WR GHDO ZLWK WKUHHNLQGV RI ORDG ,QWULQVLF ([WUDQHRXV DQG *HUPDQH ) 3DDV HW DO   ,QWULQVLF ORDG LV KRZKDUG WKH FRQWHQW LV ([WUDQHRXV ORDG LV KRZ KDUG WKH PHWKRG LV *HUPDQH ORDG LV KRZ ZHOO ZH
FDQ XVH ZKDW ZH OHDUQ 7KH WKHRU\ VD\V WKDW ZH VKRXOG WU\ WR ORZHU WKH H[WUDQHRXV ORDG DQG
LQFUHDVH WKH JHUPDQH ORDG WR OHDUQ EHWWHU 6ZHOOHU HW DO   0DQ\ UHVHDUFKHUV KDYH XVHGWKLV WKHRU\ WR VWXG\ RQOLQH OHDUQLQJ SUREOHPV $QPDUNUXG HW DO  
2QOLQH OHDUQLQJ PHWKRGV JLYH WHDFKHUV PRUH FKRLFHV EXW WKH\ DOVR QHHG PRUH VNLOOV7HDFKHUV KDYH WR NQRZ KRZ WR XVH GLIIHUHQW PHWKRGV IRU GLIIHUHQW VLWXDWLRQVGHYHORS HQJDJLQJ
DQG HIIHFWLYH RQOLQH DFWLYLWLHV DQG PDQDJH WKHLU RZQ WKLQNLQJ DQG OHDUQLQJ ZLWK WKH KHOS RI
WHFKQRORJLFDO WRROV 7KLV VWXG\ PHDVXUHV KRZ OLYH OHFWXUHV SUHUHFRUGHG YLGHRV DQG $,FKDWERWV DIIHFW WHDFKHUV¶ H[WUDQHRXV DQG JHUPDQH ORDG

344 /LWHUDWXUH 5HYLHZ
 2QOLQH /HDUQLQJ DQG 7HFKQRORJLHV'LVWDQFH HGXFDWLRQ KDV EHFRPH SRSXODU DQG LWV HIIHFWLYHQHVV KDV FRQWLQXRXVO\ LPSURYHG ZLWKWHFKQRORJLFDO GHYHORSPHQW $PRQJ WKH H[LVWLQJ RQOLQH OHDUQLQJ WHFKQRORJLHV SUHUHFRUGHG
YLGHRV DQG OLYH OHFWXUHV DUH FXUUHQWO\ WKH PRVW FRPPRQ RQOLQH WHDFKLQJ WHFKQRORJLHV 7KH
HIIHFWLYHQHVV RI SUHUHFRUGHG YLGHRV DQG OLYH OHFWXUHV KDV EHHQ SURYHQ LQ SUHYLRXV UHVHDUFKVWXGLHV %HUWVFK HW DO  ,VODP HW DO  6KTDLGHI HW DO   1HYHUWKHOHVV VWXGHQWVFRXOG QRW JHW DQ RSSRUWXQLW\ WR DVN TXHVWLRQV DQG JHW IHHGEDFN SURPSWO\ ZKLOH OHDUQLQJWKURXJK SUHUHFRUGHG YLGHRV :LWK WKH DGYDQFHPHQW RI WHFKQRORJ\ OLYH OHFWXUHV HPHUJHG DQG
DGGUHVVHG WKLV ZHDNQHVV /LYH OHFWXUH PLPLFV IDFHWRIDFH FODVVHV 7KH LQVWUXFWRU SUHVHQWVKLPVHOI RQ RQH VLGH RI WKH FRPSXWHU $QG JHVWXUHV RQ WKH RWKHU VLGH DVN TXHVWLRQV DQG
LQWHUDFW ZLWK HDFK RWKHU WKURXJK WKH FKDW IXQFWLRQ +RZHYHU WKH OLYH OHFWXUH LV OLPLWHG E\
WHFKQRORJ\ ,QWHUQHW VSHHG DQG OLYH EURDGFDVW HTXLSPHQW &KLWDQDQD HW DO   $V WKHODWHVW HPHUJLQJ WHFKQRORJ\ RI RQOLQH OHDUQLQJ DQG LQVWUXFWLRQ $, FKDWERW FDQ DVVLVW WKHLQVWUXFWRU FRQYHUWV WKH LQVWUXFWLRQDO GHVLJQ LQWR D GLDORJXH LQ DGYDQFH DQG WKH OHDUQLQJ RFFXUVLQ D TXHVWLRQDQGDQVZHU LQWHUDFWLYH IRUPDW $, FKDWERW FDQ QRW RQO\ HPEHG LPDJHV YLGHRVWHVWV DQG RWKHU FRQWHQW EXW DOVR FROOHFW OHDUQLQJ GDWD 0HDQZKLOH $, FKDWERWV FDQ HQKDQFH
LQWHUDFWLYH OHDUQLQJ ,Q D VXUYH\ RQ WKH DSSOLFDWLRQ RI $, FKDWERWV LQ ,QGLD PRVW VWXGHQWV VWDWHG
WKDW $, FKDWERWV FDQ KHOS WKHP VROYH WKHLU SUREOHPV PRVW TXLFNO\ 6DQGX *LGH  $OWKRXJK WKH $, FKDWERW DOORZV IRU WLPHO\ LQWHUDFWLRQ LW ODFNV HPRWLRQDO WUDQVIHU 7KHUH LV D ODFNRI UHVHDUFK WKDW FRPSDUHV $, FKDWERWV SUHUHFRUGHG YLGHRV DQG OLYH OHFWXUHV )XUWKHUUHVHDUFK LV QHHGHG WR H[SORUH ZKHWKHU WKHUH DUH GLIIHUHQFHV LQ RQOLQH OHDUQLQJ &RJQLWLYH /RDG 7KHRU\ DQG LWV 8VH LQ 2QOLQH /HDUQLQJ&RJQLWLYH ORDG UHIHUV WR WKH WRWDO PHQWDO DFWLYLW\ LPSRVHG RQ ZRUNLQJ PHPRU\ DW DQ\ JLYHQ
PRPHQW ZKHQ SHUIRUPLQJ D VSHFLILF WDVN ) 3DDV   6ZHOOHU  ILUVW SURSRVHG WKH&RJQLWLYH /RDG 7KHRU\ &/7 LQ WKH SURFHVV RI SUREOHPVROYLQJ0HDQZKLOH 6ZHOOHU GLYLGHG FRJQLWLYH ORDG LQWR WKUHH FDWHJRULHV ,QWULQVLF &RJQLWLYH /RDG ,&/  ([WUDQHRXV&RJQLWLYH /RDG (&/ DQG *HUPDQH &RJQLWLYH /RDG *&/  (IIHFWLYH LQVWUXFWLRQDO GHVLJQVKRXOG PLQLPL]H (&/ DV PXFK DV SRVVLEOH ZKLOH XVLQJ WKH VDPH ,&/ WR VHW PRUH FRJQLWLYHUHVRXUFHV IRU *&/ 6ZHOOHU HW DO  
7KH WKUHH W\SHV RI FRJQLWLYH ORDG KDV DOVR EHHQ ZLGHO\ GLVFXVVHG 7KH GHILQLWLRQ RI WKH
WKUHH W\SHV RI FRJQLWLYH ORDG VKRZV WKDW WKH FRPSOH[LW\ RI FRQWHQW GHWHUPLQHV LQWULQVLF
FRJQLWLYH /RDG ,&/  $W WKH VDPH WLPH RQOLQH OHDUQLQJ WHFKQRORJ\ LV D YHKLFOH RI NQRZOHGJH
IRU LWV FRQFHUQV ZLWK WKH SUHVHQWDWLRQ RI OHDUQLQJ PDWHULDOV DQG WKH SURFHVVLQJ RI NQRZOHGJH E\
OHDUQHUV 2QOLQH OHDUQLQJ WHFKQRORJ\ FRXOG QRW LQIOXHQFH ,&/ %DVHG RQ WKH SUHYLRXV FRJQLWLYH
ORDG PRGHO 6ZHOOHU   6NXOPRZVNL DQG ;X 6NXOPRZVNL ;X  SURSRVHG WKH&RVWEHQHILW PRGHO LQ GLJLWDO OHDUQLQJ ZKLFK FDQ EH VHHQ LQ )LJXUH  ,Q WKH &RVWEHQHILW PRGHO(&/ LV VHHQ DV D FRVW ZKLOH *&/ LV VHHQ DV D EHQHILW (&/ LV FRQVLGHUHG WKH HIIRUW UHTXLUHG IRUOHDUQHUV WR XQGHUVWDQG WKH SUHVHQWDWLRQ RI OHDUQLQJ PDWHULDOV ZKLOH *&/ LV WKH H[WHQW WR ZKLFKOHDUQHUV XWLOL]H WKHLU ZRUNLQJ PHPRU\ UHVRXUFHV 7KH &RVWEHQHILW PRGHO VXSSRUWV (&/ DQG*&/ DV HVVHQWLDO FRQVLGHUDWLRQV LQ RQOLQH OHDUQLQJ)LJ  7KH HYROXWLRQ RI WKH PRGHOV RI FRJQLWLYH /RDG 6NXOPRZVNL

345;X 

 7KH 0HDVXUHPHQW RI &RJQLWLYH /RDG
2EMHFWLYH DQG VXEMHFWLYH PHDVXUHPHQWV DUH FXUUHQWO\ WKH PDLQ ZD\V RI PHDVXULQJ FRJQLWLYH/RDG ) 3DDV HW DO   6XEMHFWLYH PHDVXUHPHQW LV SDUWLFLSDQWV ZKR UHSRUW WKHLU FRJQLWLYH/RDG E\ 6HOIUHSRUWHG 2EMHFWLYH PHDVXUHPHQW FDQ EH SV\FKRORJLFDO SK\VLRORJLFDO
SURFHGXUHV RI WDVNV WLPH XVHG LQ WDVNV RU UHDFWLRQ WLPH 2EMHFWLYH PHDVXUHPHQWV PD\ FDXVH
GLIIHUHQW UHVXOWV LQ D VXUYH\ IRU LWV VHQVLWLYLW\ DQG SK\VLRORJLFDO TXDOLW\ $GGLWLRQDOO\ REMHFWLYH
PHDVXUHPHQW UHOLHV RQ H[SHQVLYH HTXLSPHQW DQG WDVN SHUIRUPDQFH LV KLJKO\ UHODWHG WR WKH
WDVN W\SH 7KHUHIRUH VXEMHFWLYH PHDVXUHPHQW PD\ EH EHWWHU WKDQ REMHFWLYH PHDVXUHPHQW LQWKLV UHVHDUFK VWXG\
7KHUH DUH PDQ\ VHOIUHSRUWHG VFDOHV IRU WKH FRJQLWLYH ORDG ,Q WKH HDUO\ V )UHG 3DDV GHYHORSHG WKH ILUVW FRJQLWLYH ORDG VFDOH 7KLV 8QLGLPHQVLRQDO 3RLQW 6\PPHWULFDO&DWHJRU\ 5DWLQJ 6FDOH KDV H[FHOOHQW FRQVLVWHQF\ DQG LV ZLGHO\ XVHG LQ UHVHDUFK ) 3DDV HW DO  1HYHUWKHOHVV WKHVH H[LVWLQJ VFDOHV GR QRW QRWLFH FDWHJRULHV RI FRJQLWLYH ORDG /HSSLQN
HW DO  GHYHORSHG D VFDOH WR PHDVXUH GLIIHUHQW W\SHV RI FRJQLWLYH ORDG ZKLFK FRQWDLQV WKH
H[WUDQHRXV DQG LQWULQVLF FRJQLWLYH ORDG ,Q WKLV VWXG\ ZH DGDSW WKH &RJQLWLYH /RDG 6FDOH/HSSLQN HW DO  IRU GDWD FROOHFWLRQ 7KH 3UHVHQW 6WXG\,Q VXP GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV PD\ EULQJ GLIIHUHQW (&/V WR OHDUQHUV DQG DOORZ
GLIIHUHQW *&/V ZKLFK HIIHFWLYHO\ LQGLFDWHV OHDUQLQJ LQYROYHPHQW 7KLV VWXG\ DLPV WR LQYHVWLJDWH(&/ DQG *&/ SHUFHLYHG E\ LQVHUYLFH WHDFKHUV ZKLOH OHDUQLQJ RQOLQH ZLWK WKUHH RQOLQH OHDUQLQJ
WHFKQRORJLHV $, FKDWERW SUHUHFRUGHG YLGHRV DQG OLYH OHFWXUH  7KH UHVHDUFK TXHVWLRQV DUH
DV IROORZVz:KDW DUH WKH LQVHUYLFH WHDFKHUV¶ SHUFHLYHG (&/ DQG *&/ ZKLOH OHDUQLQJ RQOLQH ZLWK WKUHH
RQOLQH OHDUQLQJ WHFKQRORJLHV"+RZ GR WKH LQVHUYLFH WHDFKHUV¶ SHUFHLYHG (&/ DQG *&/ GLIIHU DPRQJ WKHVH WKUHHzOHDUQLQJ WHFKQRORJLHV"

 0(7+2'2/2*<
 3DUWLFLSDQWV
7KLV RQOLQH WUDLQLQJ ODVWHG  GD\V VWDUWLQJ RQ -XO\   DQG HQGLQJ RQ -XO\   $WRWDO RI  WHDFKHUV SDUWLFLSDWHG LQ WKLV FRXUVH $IWHU WKH WUDLQLQJ WKH WHDFKHUV ZKR DWWHQGHG
WKH RQOLQH WUDLQLQJ FRPSOHWHG TXHVWLRQQDLUHV RQ WKH ZLGHO\ XVHG RQOLQH TXHVWLRQQDLUH SODWIRUPKWWSVZZZZM[FQ  :H FROOHFWHG  YDOLG TXHVWLRQQDLUHV ZLWK D YDOLG UHWXUQ UDWH RI $OO SDUWLFLSDQWV YROXQWDULO\ FRPSOHWHG WKH DQRQ\PRXV TXHVWLRQQDLUH DQG WKHLU SULYDF\ ZDVHQVXUHG QRW WR EH GLVFORVHG  Q  RI WKHP DUH PDOHV DQG  Q  DUH
IHPDOHV 7HDFKHUV DUH LQYROYHG LQ DOO VXEMHFWV  Q  RI WKH WHDFKHUV FDPH IURP
SULPDU\ VFKRROV  Q  IURP MXQLRU KLJK VFKRROV DQG  Q  IURP VHQLRU KLJKVFKRROV 0DWHULDOV
7KH PDLQ DLP RI WKLV VWXG\ ZDV WR H[SORUH RQOLQH OHDUQLQJ WHFKQRORJLHV ZKLFK ZHUH PRUH
HIIHFWLYH EHWZHHQ $, FKDWERW SUHUHFRUGHG YLGHRV DQG OLYH OHFWXUHV 7R FRPSDUH WKH
GLIIHUHQFH LQ *&/ SHUFHLYHG E\ WHDFKHUV ZKHQ XVLQJ GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV ZH
DUUDQJHG D GD\ RQOLQH WUDLQLQJ DQG DSSOLHG DQ RQOLQH LQVWUXFWLRQDO SODWIRUP WR SURYLGH WKHSDUWLFLSDWRUV ZLWK WKHVH WKUHH WHFKQRORJLHV
7R LQYHVWLJDWH WKH LPSDFW RI GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV RQ (&/ DQG *&/GXULQJ OHDUQLQJ ZH DGDSWHG D TXHVWLRQQDLUH EDVHG RQ WKH &RJQLWLYH /RDG 6FDOH /HSSLQN HW DO  7KH DGDSWHG &RJQLWLYH /RDG 6FDOH HIIHFWLYHO\ HYDOXDWHV WKH (&/ DQG *&/ SHUFHLYHG

346

E\ WHDFKHUV LQFOXGLQJ  LWHPV DERXW EDVLF LQIRUPDWLRQ (&/ DQG *&/ 0RUHRYHU QR
SHUVRQDO LQIRUPDWLRQ WKDW FRXOG UHYHDO WKH LGHQWLW\ RI WKH SDUWLFLSDQWV ZDV JDWKHUHG

 3URFHGXUH$OO  WHDFKHUV ZKR FRPSOHWHG D YDOLGDWHG TXHVWLRQQDLUH SDUWLFLSDWHG LQ D GD\ RQOLQH
WUDLQLQJ FRXUVH HQFRPSDVVLQJ XQGHUVWDQGLQJ PDQLSXODWLQJ DSSO\LQJ DQG HYDOXDWLQJ YLVXDOWKLQNLQJ WRROV 7KH OHDUQLQJ ZDV GLYLGHG LQWR WHQ PRGXOHV XVLQJ DQ $, FKDWERW SUHUHFRUGHG
YLGHRV DQG OLYH OHFWXUHV 7KH OHDUQLQJ SURFHVV ZDV H[SODLQHG DV IROORZV ,QLWLDOO\ WHDFKHUVOHDUQHG WKH FRQFHSWV DQG FDVHV WKURXJK $, FKDWERW DQG SUHUHFRUGHG YLGHRV DQG WKHQSDUWLFLSDWHG LQ OLYH OHFWXUHV WR UHYLHZ DQG FRQVROLGDWHG ZKDW WKH\ KDYH OHDUQHG LQ WKH PRGXOHV7KH OLYH OHFWXUH IDFLOLWDWHG IDFHWRIDFH DQG RQOLQH FRPPXQLFDWLRQ EHWZHHQ WKH OHDUQHU DQG WKHOHFWXUHU ZLWK WKH OHFWXUHU SUHSDULQJ WKH FRQWHQW EHIRUHKDQG DQG DOORZLQJ WKH OHDUQHU WR DVN
TXHVWLRQV DW DQ\ WLPH $IWHU WKH FRXUVH D TXHVWLRQQDLUH ZDV GLVWULEXWHG WR FROOHFW WKH GDWD
QHHGHG IRU WKH H[SHULPHQW 'DWD $QDO\VLV:H XVHG 6366  WR DQDO\]H WKH TXDQWLWDWLYH GDWD FROOHFWHG E\ WKH TXHVWLRQQDLUH )LUVWO\ ZH
H[DPLQHG WKH UHOLDELOLW\ DQG YDOLGLW\ RI WKH PHDVXUHPHQW TXHVWLRQQDLUH WKURXJK H[SORUDWRU\IDFWRU DQDO\VLV 6XEVHTXHQWO\ RQHZD\ UHSHDWHGPHDVXUHV $129$ ZDV XVHG WR FRPSDUH WKHVL]H RI (&/ DQG *&/ JHQHUDWHG E\ WHDFKHUV ZKLOH OHDUQLQJ NQRZOHGJH XVLQJ GLIIHUHQW RQOLQHOHDUQLQJ WHFKQRORJLHV &RPSDUHG WR WUDGLWLRQDO $129$ UHSHDWHG PHDVXUHV $129$ LV PRUH
VXLWDEOH IRU DQDO\]LQJ WKH HIIHFWV RI LPSOHPHQWLQJ DQ LQWHUYHQWLRQ RQ PXOWLSOH YDULDEOHV ZLWKLQWKH VDPH JURXS RI VXEMHFWV 5(68/76
 $QDO\VLV RI 5HOLDELOLW\ DQG 9DOLGLW\
7KH UHOLDELOLW\ DQG YDOLGLW\ RI WKH TXHVWLRQQDLUH ZHUH H[DPLQHG IURP WKH YDOLG GDWD $V VKRZQ LQ
7DEOH  WKH (&/ DQG *&/ VFDOHV GHPRQVWUDWHG KLJK LQWHUQDO FRQVLVWHQF\ ZLWK &URQEDFK¶V Į
FRHIILFLHQWV RI  DQG  UHVSHFWLYHO\ 7KLV TXHVWLRQQDLUH ZDV DGDSWHG IURP D
ZHOOHVWDEOLVKHG VFDOH LQ SUHYLRXV VWXGLHV DQG WKH VFDOH¶V YDOLGLW\ ZDV IXUWKHU FRQILUPHG E\
H[SORUDWRU\ IDFWRU DQDO\VLV ZLWK .02  ! 7DEOH  ()$ DQG &URQEDFK¶Vʐ 9DOXHV IRU WKH (&/ DQG *&/ Q)DFWRU DQG ,WHPV)DFWRU /RDGLQJ

/LYH OHFWXUH
3UHUHFRUGHG
YLGHRV$, FKDWERW



0HDQ











/LYH OHFWXUH
3UHUHFRUGHG
YLGHRV

















(&/

*&/

6'



&URQEDFK¶V ʐ




 $QDO\VLV RI WKH (IIHFW RI 'LIIHUHQW 2QOLQH /HDUQLQJ 7HFKQRORJLHV RQ ,QVHUYLFH
7HDFKHUV¶ (&/ DQG *&/7KH UHVXOWV RI WKH UHSHDWHG PHDVXUHV $129$ VKRZHG WKDW WKHUH LV D VLJQLILFDQW ERUGHUOLQH
HIIHFW DPRQJ (&/ RI GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV )   S 347

  SDUWLDO Ș   ZKLFK VXJJHVWV WKDW WHDFKHUV SURGXFH GLIIHUHQW (&/ ZKHQ XVLQJYDULRXV WHFKQRORJLFDO WRROV WR OHDUQ NQRZOHGJH (&/ SHUFHLYHG E\ WHDFKHUV ZKHQ OHDUQLQJ ZLWKSUHUHFRUGHG YLGHRV LV VLJQLILFDQWO\ ORZHU WKDQ WKH (&/ RI $, FKDWERW S    DQG (&/RI OLYH OHFWXUH S     (&/ RI WKH $, FKDWERW DQG OLYH OHFWXUH KDYH QR VLJQLILFDQW
GLIIHUHQFH S  !   5HJDUGLQJ WKH DQDO\VLV RI *&/ ZH XVHG *UHHQKRXVH*HLVVHU
HVWLPDWHV RI VSKHULFLW\ İ  WR FRUUHFW GHJUHHV RI IUHHGRP IRU VSKHULFLW\ YLRODWLRQV0DXFKO\¶V :  Ȥ   S   5HSHDWHGPHDVXUHV $129$ VKRZHG QR
VLJQLILFDQW GLIIHUHQFH LQ WKH *&/ ZKHQ WHDFKHUV XVHG GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHVZLWK S  ! 
7DEOH  5HSHDWHGPHDVXUHV $129$ DQG SRVW KRF WHVWV
9DULDEOH)

6LJ

SDUWLDO ʖ

3RVW KRF FRPSDULVRQV
SUHUHFRUGHG YLGHRV  $,FKDWERWSUHUHFRUGHG YLGHRV  OLYHOHFWXUH(&/ RI GLIIHUHQW
2QOLQH /HDUQLQJ







*&/ RI GLIIHUHQW
2QOLQH /HDUQLQJ







 ',6&866,21
7KLV VWXG\ WULHV WR FRPSDUH KRZ WHDFKHUV IHHO ZKHQ WKH\ OHDUQ RQOLQH ZLWK FKDWERWV YLGHRVDQG OLYH OHFWXUHV 7KH UHVXOW VKRZV WKDW SUHUHFRUGHG YLGHRV DUH PRUH HIIHFWLYH WKDQ $, FKDWERW
DQG OLYH OHFWXUHV LQ UHGXFLQJ (&/ ,Q FRQWUDVW $, FKDWERW DQG OLYH OHFWXUHV KDYH QR
GLIIHUHQFH7KLV PD\ EH EHFDXVH RI VRPH UHDVRQV )LUVW QHZ WHFKQRORJLHV FDQ PDNH OHDUQHUVVWUHVVHG DQG FRQIXVHG 5LHGO   /HDUQHUV KDYH D EHWWHU XQGHUVWDQGLQJ RI YLGHRVFRPSDUHG WR OLYH OHFWXUHV DQG FKDWERWV $GGLWLRQDOO\ QHZ WHFKQRORJLHV FDQ HQFRXQWHU HUURUVRU WHDFKLQJUHODWHG LVVXHV )RU H[DPSOH FKDWERWV PD\ QRW SURYLGH DFFXUDWH RU FOHDU DQVZHUVZKLOH OLYH OHFWXUHV FDQ EH DIIHFWHG E\ QHWZRUN SUREOHPV $V D UHVXOW WKHVH IDFWRUV FDQ FDXVH
OHDUQHUV WR ORVH WKHLU IRFXV RU LQWHUHVW+RZHYHU LW LV ZRUWK QRWLQJ WKDW FKDWERWV GLG QRW FUHDWH PRUH FRQIXVLRQ IRU OHDUQHUV FRPSDUHG
WR OLYH OHFWXUHV ZKLFK LQGLFDWHV WKDW WHDFKHUV IRXQG WKHP EHQHILFLDO
1RQH RI WKH WKUHH RQOLQH OHDUQLQJ WHFKQRORJLHV JUHDWO\ LPSDFWHG *&/ LQGLFDWLQJ WKDW
H[WULQVLF WHFKQRORJ\ KDG D OLPLWHG HIIHFW RQ IRVWHULQJ LQWULQVLF FRJQLWLYH VWUXFWXUHV DQG
PRWLYDWLRQ /HDUQHUV QHHG WR WDON DQG LQWHUDFW PRUH LQ WKH FODVV WR XQGHUVWDQG EHWWHU 2WKHUVWXGLHV DOVR VDLG WKDW YLGHRV ZRUN EHWWHU ZLWK GLIIHUHQW NLQGV RI NQRZOHGJH DQG WHDFKHU KHOS+RQJ HW DO  
7KLV VWXG\ KDV VRPH OLPLWV ,W RQO\ DVNHG OHDUQHUV KRZ WKH\ IHOW ZKLFK PD\ QRW VKRZ KRZ
WKH\ UHDOO\ WKRXJKW ,W DOVR GLG QRW ORRN DW RWKHU WKLQJV WKDW FDQ DIIHFW WKLQNLQJ DQG OHDUQLQJ OLNHZKDW OHDUQHUV NQRZ EHIRUH KRZ WKH\ IHHO ZKDW WKH\ OLNH RU ZKDW WKH\ ZDQW &21&/86,216 $1' ,03/,&$7,216
7KHVH ILQGLQJV VXJJHVW WKDW WHDFKHUV ZKR OHDUQHG WKURXJK SUHUHFRUGHG YLGHRV KDG D ORZHUSHUFHSWLRQ RI WKHLU (&/ FRPSDUHG WR WKRVH ZKR XVHG $, FKDWERWV RU DWWHQGHG OLYH OHFWXUHV ,Q
UHJDUGV WR *&/ WKH WKUHH GLIIHUHQW RQOLQH OHDUQLQJ WHFKQRORJLHV GR QRW VKRZ D VLJQLILFDQW
GLIIHUHQFH LQ *&/ 7KLV VWXG\ JLYHV VRPH LGHDV IRU IXWXUH UHVHDUFK VWXGLHV )LUVW IXWXUHUHVHDUFK VWXGLHV FRXOG XWLOL]H PRUH DGYDQFHG $, WHFKQRORJLHV WR LPSURYH WKH GHVLJQ DQG
LPSOHPHQWDWLRQ RI $, FKDWERW VXFK DV QDWXUDO ODQJXDJH SURFHVVLQJ PRGHOV VSHHFKUHFRJQLWLRQ V\VWHPV DQG JHQHUDWLYH $, 6HFRQG IXWXUH UHVHDUFK VWXGLHV FRXOG H[SDQG IURPWHDFKHUV WR VWXGHQWV :H FDQ DQDO\]H KRZ GLIIHUHQW RQOLQH OHDUQLQJ PHWKRGV DIIHFW GLIIHUHQWJURXSV RI OHDUQHUV VXFK DV DJH JHQGHU NQRZOHGJH DQG SUHIHUHQFHV 7KLUG VXEMHFWLYH DQG
REMHFWLYH PHDVXUHPHQWV FDQ EH FRPELQHG WR HQKDQFH WKH UHVXOWV IURP PXOWLSOH SHUVSHFWLYHVZKHQ PHDVXULQJ FRJQLWLYH ORDG RU FRQVLGHULQJ RWKHU UHOHYDQW YDULDEOHV LQ RQOLQH OHDUQLQJ ZLOO

348

EH D YDOXDEOH WRSLF )RXUWK IXWXUH UHVHDUFK FRXOG IRFXV PRUH RQ WHFKQRORJ\DVVLVWHG
LQVWUXFWLRQ WKDQ WHFKQRORJ\UHSODFHG LQVWUXFWLRQ7HFKQRORJ\ FDQ PDNH WHDFKLQJ HDVLHU EXW LW
FDQQRW GR ZKDW WHDFKHUV GR 7HDFKHUV GR QRW RQO\ JLYH NQRZOHGJH EXW DOVR VKRZ WKLQNLQJVNLOOV DQG YDOXHV WR OHDUQHUV

5HIHUHQFHV$QPDUNUXG  $QGUHVHQ $ %UnWHQ ,   &RJQLWLYH /RDG DQG :RUNLQJ 0HPRU\ LQ 0XOWLPHGLD/HDUQLQJ &RQFHSWXDO DQG 0HDVXUHPHQW ,VVXHV (GXFDWLRQDO 3V\FKRORJLVW  ±
KWWSVGRLRUJ%HUWVFK 7 ) &DOODV 3 : 5XELQ $ &DSXWR 0 3 5LFFL 0 $   (IIHFWLYHQHVV RI OHFWXUHVDWWHQGHG YLD LQWHUDFWLYH YLGHR FRQIHUHQFLQJ YHUVXV LQSHUVRQ LQ SUHSDULQJ WKLUG\HDU LQWHUQDO
PHGLFLQH FOHUNVKLS VWXGHQWV IRU &OLQLFDO 3UDFWLFH ([DPLQDWLRQV &3;  7HDFKLQJ DQG /HDUQLQJ LQ
0HGLFLQH    ± KWWSVGRLRUJ&KLWDQDQD / 0DND]D ' 0DG]LPD .   7KH FXUUHQW VWDWH RI HOHDUQLQJ DW XQLYHUVLWLHV LQ=LPEDEZH 2SSRUWXQLWLHV DQG FKDOOHQJHV ,QWHUQDWLRQDO -RXUQDO RI (GXFDWLRQ DQG 'HYHORSPHQW
8VLQJ ,&7
KWWSVZZZVHPDQWLFVFKRODURUJSDSHU7KHFXUUHQWVWDWHRIHOHDUQLQJDWXQLYHUVLWLHVLQ&KLWDQD
QD0DND]DDGHGIIGGGGFEIDFG
+RQJ - 3L = <DQJ -   /HDUQLQJ GHFODUDWLYH DQG SURFHGXUDO NQRZOHGJH YLD YLGHR OHFWXUHV&RJQLWLYH ORDG DQG OHDUQLQJ HIIHFWLYHQHVV ,QQRYDWLRQV LQ (GXFDWLRQ DQG 7HDFKLQJ ,QWHUQDWLRQDO   ± KWWSVGRLRUJ
,VODP 0 .LP '$ .ZRQ 0   $ &RPSDULVRQ RI 7ZR )RUPV RI ,QVWUXFWLRQ 3UH5HFRUGHG
9LGHR /HFWXUHV YV /LYH =220 /HFWXUHV IRU (GXFDWLRQ LQ WKH %XVLQHVV 0DQDJHPHQW )LHOG
6XVWDLQDELOLW\    $UWLFOH  KWWSVGRLRUJVX
.RUEDFK $ %UQNHQ 5 3DUN %   0HDVXUHPHQW RI FRJQLWLYH ORDG LQ PXOWLPHGLD OHDUQLQJ $FRPSDULVRQ RI GLIIHUHQW REMHFWLYH PHDVXUHV ,QVWUXFWLRQDO 6FLHQFH    ±KWWSVGRLRUJV/HSSLQN - 3DDV ) 9DQ GHU 9OHXWHQ & *RJ 7 9DQ 0HUULHQERHU - - *   'HYHORSPHQW RIDQ LQVWUXPHQW IRU PHDVXULQJ GLIIHUHQW W\SHV RI FRJQLWLYH ORDG %HKDYLRU 5HVHDUFK 0HWKRGV 
KWWSVGRLRUJV
0XWOX%D\UDNWDU ' &RVJXQ 9 $OWDQ 7   &RJQLWLYH ORDG LQ PXOWLPHGLD OHDUQLQJHQYLURQPHQWV $ V\VWHPDWLF UHYLHZ &RPSXWHUV (GXFDWLRQ  
KWWSVGRLRUJMFRPSHGX
3DDV ) * : &   7UDLQLQJ VWUDWHJLHV IRU DWWDLQLQJ WUDQVIHU RI SUREOHPVROYLQJ VNLOO LQ VWDWLVWLFV $FRJQLWLYHORDG DSSURDFK -RXUQDO RI (GXFDWLRQDO 3V\FKRORJ\    ±KWWSVGRLRUJ
3DDV ) 5HQNO $ 6ZHOOHU -   &RJQLWLYH /RDG 7KHRU\ DQG ,QVWUXFWLRQDO 'HVLJQ 5HFHQW'HYHORSPHQWV (GXFDWLRQDO 3V\FKRORJLVW    ±
KWWSVGRLRUJ6(3B
5LHGO 5   2Q WKH ELRORJ\ RI WHFKQRVWUHVV /LWHUDWXUH UHYLHZ DQG UHVHDUFK DJHQGD $&0 6,*0,6'DWDEDVH 7KH '$7$%$6( IRU $GYDQFHV LQ ,QIRUPDWLRQ 6\VWHPV    ±
KWWSVGRLRUJ
6DQGX 1 *LGH (   $GRSWLRQ RI $,&KDWERWV WR (QKDQFH 6WXGHQW /HDUQLQJ ([SHULHQFH LQ+LJKHU (GXFDWLRQ LQ ,QGLD  WK ,QWHUQDWLRQDO &RQIHUHQFH RQ ,QIRUPDWLRQ 7HFKQRORJ\ %DVHG+LJKHU (GXFDWLRQ DQG 7UDLQLQJ ,7+(7  ± KWWSVGRLRUJ,7+(7
6KTDLGHI $ - $EX%DNHU ' $O%LWDU = % %DGUDQ 6 +DPGDQ $ 0   $FDGHPLF
SHUIRUPDQFH RI GHQWDO VWXGHQWV $ UDQGRPLVHG WULDO FRPSDULQJ OLYH DXGLR UHFRUGHG DQG YLGHRUHFRUGHG OHFWXUHV (XURSHDQ -RXUQDO RI 'HQWDO (GXFDWLRQ    ±KWWSVGRLRUJHMH
6NXOPRZVNL $ ;X .   8QGHUVWDQGLQJ &RJQLWLYH /RDG LQ 'LJLWDO DQG 2QOLQH /HDUQLQJ $ 1HZ
3HUVSHFWLYH RQ ([WUDQHRXV &RJQLWLYH /RDG (GXFDWLRQDO 3V\FKRORJ\ 5HYLHZ  ±
KWWSVGRLRUJV
6ZHOOHU -   &RJQLWLYH /RDG 'XULQJ 3UREOHP 6ROYLQJ (IIHFWV RQ /HDUQLQJ &RJQLWLYH 6FLHQFH   ± KWWSVGRLRUJVFRJB
6ZHOOHU - YDQ 0HUULsQERHU - - * 3DDV )   &RJQLWLYH $UFKLWHFWXUH DQG ,QVWUXFWLRQDO'HVLJQ  <HDUV /DWHU (GXFDWLRQDO 3V\FKRORJ\ 5HYLHZ    ±
KWWSVGRLRUJV

349

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Do the Same Rules Apply? Transferring
MOOC Success Behaviors to University
Online Learning
Clarence James Monterozoa* & Maria Mercedes RODRIGOa
Ateneo Laboratory for the Learning Sciences, Ateneo de Manila University, Philippines*clarence.monterozo@obf.ateneo.edu

aAbstract: In this paper, we replicated the methods and findings of a study on
successful completion of massive open online courses (MOOCs) in the context of
online learning in a university in Metro Manila, Philippines. Specifically, we examine
whether discussion forum behaviors associated with completion in MOOCs also
resulted in better outcomes in university online learning courses. To this end, wecollected logs from the university’s learning management system. The Mann-Whitney
U test was used to compare the average scores of those who conformed to thesebehaviors and those who did not. Pearson’s r correlation was then used to determine
if a significant relationship exists between each of the measured behaviors and theaverage scores of the students for each course. The same analyses were thenperformed for each of the five schools in the same university. The study failed toreplicate the results of the MOOC study in all cases. The difference in means was not
far apart for all rules. This could be attributed to the difference in pacing anddemographic between universities and MOOCs. Despite these, the study still
demonstrates that techniques used to study MOOCs can be used to study the
behaviors and success indicators of students.
Keywords: MOOC, university online learning, academic success1. Introduction
Massively Open Online Courses (MOOCs) are Internet-based classes offered at minimal tono cost. They are available to any participant with Internet access, anytime, anywhere(Cagiltay, Cagiltay, & Celik, 2020). Because barriers to entry are low, millions of people
enroll in these courses. The top 5 MOOC providers—Coursera, edX, XuetangX, Udacity,and FutureLearn—account for a collective population of 90 million learners.
Completion rates for these MOOCs, though, tend to be low, typically between 5 and
15% (Ahern, 2018). Researchers have therefore tried to study the relationship between
student in-course behaviors and their course completion. Typically, these studies examined
students within a single course, raising questions about replicability. Which of these findings
generalize across MOOCs and which do not? Andres and colleagues (2018) addressed theissue of replicability by testing prior findings against data from multiple MOOCs. They foundthat students who were likely to complete a course tended to spend more time in forums andon assignments, make longer posts, post, and respond more frequently, and so on.
The question that we pose in this paper is whether these findings transfer beyond
MOOCs, to online learning during the COVID-19 pandemic. Transferring these findings
could provide an alternative method of studying success indicators beyond the usual surveys
done for online learning. This study investigates the following behaviors associated withearning a certificate in MOOCs studied by Andres et al. (2017):
z
Rule 1. Number of posts in discussion forums is greater than average.
z
Rule 2. Length of posts in discussion forums is longer than average.
z
Rule 3. Number of responses to others in discussion forums is greater than average.
z
Rule 4. Number of respondents to threads in discussion forums is greater than average.

350More specifically, the study aims to answer the following research questions:z
RQ1: Is there a significant difference between the average scores of students that fallunder the behaviors listed and those that do not?
z
RQ2: Is there a correlation between the average word count per discussion post, theaverage number of posts, the average number of replies received, and the average
number of replies to others, and the average score of each student?
z
RQ3: Are these correlations present within the courses offered by the different schoolsof the university?

2. Background Literature
2.1 MOOCs and Universities
MOOCs have often been compared to on-campus university courses, especially given thatsome universities also offer MOOCs. On-campus university courses consist of students whoare closer to one another while MOOCs have a wider reach across different countries.
University courses also follow a directed learning curriculum with a pre-defined course
outline. MOOCs mostly follow self-directed learning with a more flexible timeline. (Begiu &Strobl, 2015). Even educators have a difference in their approach and roles between the
two. Educators in MOOCs those courses cited managing the course, creating resources,and guiding students as part of their roles. On the other hand, on-campus university
educators cited that guiding and accompanying students formed the bulk of their work whileproviding feedback carried a lesser weight (Cabrera & Fernández-Ferrer, 2017).

2.2 Student Success in MOOCs and University Online Learning
Research into MOOCs has focused on behaviors associated with completing the course as
most MOOCs have a low completion rate (Crossley et al., 2016). The present study focuses
on data collected from discussion forums which have similarly been the subject of several
studies into MOOC completion. Andres et al. (2017) investigated 21 findings or rules fromprevious studies to identify if these held true for their dataset. Out of the 21 rules, 14
involved behaviors in discussion forums, but the present study focuses on the four rules
mentioned earlier. Out of the four, the study failed to replicate the second rule since theanalysis was not statistically significant.
On the other hand, studies into the performance of university students in online learning
primarily focus on survey-based questions. A survey of undergraduate nursing students inSouth Korea by Kim et al. (2022) found that self-directed learning behavior and satisfaction
with their course were found to be predictors of academic achievement. Other studies(Basith et al., 2020; Gopal et al., 2021) similarly made use of surveys and explored therelationship between online learning satisfaction and academic performance. It is evident
that previous studies have focused on data collection through surveys. As such, the present
study aims to investigate the use of techniques in MOOC researchers to understand the
behaviors associated with academic success in university online learning.

3. Methods
3.1 Description and Preparation of Dataset
The study made use of students’ log data from the Canvas LMS of a university in Metro
Manila, Philippines, collected during one semester from August 25, 2021, to December 18,
2021. During this time, all classes at the university were held online. There were a total of
3,429 classes offered and 6,439 students enrolled in the university. The university was
divided into 5 schools: education, humanities, management, science and engineering, andsocial sciences. Many courses within each school can be taken by students from any field ofspecialization, i.e., a student specializing in management can take a social science course,351

and a student specializing in science can take a humanities course. The discussion forum
logs from each course were used to compute the average word count, average number of
replies to others, and average number of replies received for each of the students.
In the study of Andres and colleagues (2017), success was defined as the earning ofa certificate. In our dataset, though, students earned grades that ranged from 0 to 100, so
success had more variation. Hence, we used the grade logs for all graded assignments andcomputed the average score percentage of each student for each course.

3.2 Comparison of Means for each Rule
Since the success indicator (average score across all graded assignments) is no longer
binary (completed or did not complete the course), a comparison of means was used insteadof the chi-square test used by Andres et al. (2017). For each course, a course average foreach of the four rules was obtained to split the students into two groups for each rule: those
who are above average and those who are below average. A test was then used to compare
the means of each group for all four rules. In choosing the test, the normality of the data was
first assessed using Q-Q plots (Das & Imon, 2016). Q-Q analysis for all four rules showedthat all four data are not normal. Thus, the Mann-Whitney U test (McKnight & Najab, 2010)
was used to test if there is a significant difference in the average score percentage of those
who are above average and those who are below average for each rule. The same analysis
was done for the data after being divided by school.

3.3 Analysis using Correlation for each Rule
Given that both the rules and the success indicator are continuous variables, it is possible toidentify if there exists a correlation between each rule and the average score percentages.
For this, Person’s r correlation (Akoglu, 2018) was used. This analysis provides moreinsights for courses in the university where academic success is dependent on numerical
grades and not a binary variable (pass or fail).

4. Results and Discussion
4.1 Analysis Across the Entire Population
Table 1 presents the median and interquartile range for the two groups under each rule. The
results of the Mann-Whitney U test show that there is a significant difference (p≤0.05)
between the mean score percentage of the above and below-average groups for Rules 2–4.
This seems to follow the findings of Andres et al. (2017), but the median and interquartile
range of the average score percentage per group doesn’t support the replication. The two
median average score percentages of the two groups under Rules 2–4 only differ by at most
1%. Similarly, the interquartile ranges of both groups are mostly overlapping.
Table 1. Median and IQR for the average score percentage of the groups in each rule.
#
1

Rule
Average number of
posts

Above average
Below average
2
Average word
Above averagecount*
Below average
3 Average number ofAbove average
replies given*
Below average
4 Average number ofAbove average
replies received*
Below average*statistically significant (‫ ݌‬൑ ͲǤͲͷ)352

Median
91.00
90.67
91.60
90.00
91.18
90.65
91.34
90.63

IQR
83.69 – 96.66
83.78 – 96.07
82.94 – 95.83
84.80 – 96.67
84.27 – 96.67
83.67 – 96.09
84.76 – 96.43
83.54 – 96.18

The results of the Pearson’s r Correlation in Table 2 provide more insights into thedata. Only the correlation coefficients for Rules 1 and 2 were found to be statisticallysignificant (‫ ݌‬൑ ͲǤͲͷ). For Rule 1, the result shows that there exists a negligible correlation
between the students’ average number of posts and their average score percentage.
Similarly, for Rule 2, the result shows that there exists a negligible correlation between thestudents’ average word count per post and their average score percentage. These results
show that students with a higher average number of posts and word counts did notnecessarily have higher average scores.
Table 2. Results of Pearson’s r Correlation across all four rules.
#
1
Average number of posts
2
Average word count3
Average number of replies given
4
Average number of replies received*statistically significant (‫ ݌‬൑ ͲǤͲͷ)Pearson’s ‫ ݎ‬coefficient-0.041
0.060
0.01
0.00

‫݌‬-value
0.00*
0.00*
0.25
0.99*

Overall, the results suggest that the study failed to replicate the results of Andres etal. (2017). Although the Mann-Whitney U test showed a statistically significant differencewith the means for Rules 2–4, the difference turned out to be at most 1%. For Rule 2, thispoint was confirmed by the Person’s r Correlation which showed negligible correlation.
Moreover, the Person’s r Correlation also showed that Rule 1 may not hold true as well
given the negligible correlation. These results could be explained by how university
instructors often give more rigid guidelines—such as requiring only one post—compared tothose in MOOCs. Similarly, results show that the length of these responses don’t have asignificant relationship with the students’ performance. It might be more insightful, then, toanalyze the content of responses. Aside from instructor guidelines following the directed
learning approach (Belgiu & Strobl, 2015), the failure to replicate could also be attributed tothe difference in demographic between universities and MOOCs.

4.2 Analysis per School
Tables 3 and 4 show the results of the same analyses conducted on the dataset divided by
the school that offers each course. The results of the Mann-Whitney U test for all schools
show that, among those with a statistically significant difference in means, the difference inthe medians is at most 1%, and the interquartile ranges are mostly overlapping. The only
exceptions are Rules 3 and 4 for the science and engineering courses where a difference of
3% was observed. However, the interquartile ranges for the two groups under both rules arestill mostly overlapping. Moreover, the Pearson’s r correlation for Rules 3 and 4 of thescience and engineering courses show negligible correlations for both rules. Thus, we areunable to conclude that these courses adhere to Rules 3 and 4. The remaining statisticallysignificant results for the Pearson’s r correlation show negligible. Therefore, even when
divided by school, the study fails to replicate the results of Andres et al. (2017), and theresults are similar to the results of the analysis done across the entire population. This could
indicate that instructors across schools provide the same rigid guidelines described earlier.
Table 3. Median and IQR for the average score percentage of the groups in each rule
divided by school.
Schooleducation

Rule
1

Below Average
MedianIQR
96.25
93.45 – 99.57

353

Above Average
Median
IQR
96.19
92.50 – 98.88

2
96.38
3
96.19
4
96.15
management
1*
89.00
2*
88.00
3*
89.00
4
88.67
social1*
91.0
sciences
2*
90.64
3*
91.11
4
91.32
science and1
90.63
engineering
2
90.15
3*
90.28
4*
90.00
*statistically significant (‫ ݌‬൑ ͲǤͲͷ)93.57 – 99.33
93.45 – 99.58
93.33 – 99.56
82.92 – 94.00
81.15 – 92.76
82.45 – 93.82
82.00 – 93.80
84.92 – 96.06
84.21 – 96.00
85.00 – 96.00
85.00 – 96.19
82.24 – 96.00
81.41 – 96.24
81.24 – 95.94
81.02 – 96.00

96.08
96.19
97.62
88.27
89.17
87.64
89.22
92.5
92.00
92.79
91.43
91.38
91.24
93.35
93.62

92.92 – 99.89
90.62 – 98.64
93.41 – 98.90
78.34 – 92.75
83.33 – 94.38
78.53 – 91.98
80.83 – 92.94
85.71 – 97.14
86.21 – 97.00
85.54 – 97.96
84.45 – 98.00
79.49 – 96.67
82.18 – 96.03
82.61 – 97.01
87.55 – 96.73

Table 4. Results of Pearson’s r Correlation across all four rules per school.
Rule
1
2
3
4

education
Pearson’s ‫ ݎ‬coefficient-0.09878
0.15506-0.06646
0.01456‫݌‬-value
0.34352
0.135630.52451
0.88923

Rulesocial sciences
Pearson’s ‫ ݎ‬coefficient‫݌‬-value
1
0.024300.173462
0.033520.0604230.055770.00177*4-0.04601
0.00993**statistically significant (‫ ݌‬൑ ͲǤͲͷ)management
Pearson’s ‫ ݎ‬coefficient‫݌‬-value-0.05855
0.04901*
0.095720.00127*-0.06733
0.02354*0.018960.52403science and engineering
Pearson’s ‫ ݎ‬coefficient‫݌‬-value-0.03003
0.192860.00179
0.938210.089930.00009*
0.05730
0.01290*

5. Conclusion and Further Studies
Overall, the study failed to replicate the results found by Andres et al. (2017). There areseveral possible reasons why the MOOC results did not transfer to our population. First, thestudents enrolled in MOOCs and those enrolled in traditional college courses differ infundamental ways. MOOC students tend to be between 25 and 65 years old (Cagiltay,
Cagiltay, & Celik, 2020). Most already have a higher education degree and are male. They
enroll in a MOOC because (1) they want to retool or explore new professional areas, (2) they
have a personal interest in the course content, or (3) the course is important for careeradvancement (Baturay, 2015; Williams, Stafford, Corliss, & Reilly, 2018). Guided by theirown motivations, most learners tend to navigate the course in a non-linear fashion, studying
what they need and skipping the rest (Cagiltay, Cagiltay, & Celik, 2020). MOOCs enable thistype of behavior because students are minimally supervised—it is rare for a student toreceive direct and timely feedback from course instructors.
On the other hand, the students in our population were enrolled in traditional coursesthat were forced online by COVID-19. Hence, students received direct supervision from354

teachers, including psycho-social support (see Moreno et al, 2021). Furthermore, culturally,
Filipinos place a high value on education (Galang & Reyes, 2009). They see education as asolution to poverty and ignorance, as a means towards better life opportunities. Finally,
Filipinos are motivated to persevere in their studies by social cues from parents, teachers,classmates, and friends. These factors may have driven students to succeed in theircourses, despite the difficulties brought on by the pandemic.
Although the study was unable to replicate the results, it was still able to demonstrate
how techniques used to study MOOCs can be used to study the behaviors and successindicators of students. This deviates from the usual surveys to study student behavior andmakes use of data collected from a learning management system. Moreover, future studies
can still consider replicating the other rules that the current study was not able to consider.

Acknowledgements
The authors would like to thank the Ateneo Laboratory for the Learning Sciences (ALLS),
Ateneo Research Institute for Science and Engineering (ARISE), and Accenture for thefunding and support for this research. The authors would also like to thank Deni Jaramillo
and Miguel Saavedra for their assistance in setting up the necessary servers for thecollection of Canvas data, Ezekiel Lagmay for his assistance with extracting and cleaning
the data, and Francine Tarrazona for proof-reading the paper.

References
Andres, J., Baker, R., Gaševic, D., Siemens, G., & Spann, C. (2017). Replicating 21 Findings on
Student Success in Online Learning. Technology, Instruction, Cognition and Learning, 10, 4.
https://eric.ed.gov/?id=EJ1257952
Andres, J. M. L., Baker, R. S., Gašević, D., Siemens, G., Crossley, S. A., & Joksimović, S. (2018,
March). Studying MOOC completion at scale using the MOOC replication framework. In
Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 7178).
Baturay, M. H. (2015). An overview of the world of MOOCs. Procedia-Social and Behavioral Sciences,
174, 427-433.
Belgiu, M., Strobl, J., & Wallentin, G. (2015). Open Geospatial Education. ISPRS International Journalof Geo-information, 4(2), 697–710. https://doi.org/10.3390/ijgi4020697
Cabrera, N., & Fernández-Ferrer, M. (2017). Examining MOOCs: A Comparative Study among
Educational Technology Experts in Traditional and Open Universities. The International Review
of Research in Open and Distributed Learning, 18(2). https://doi.org/10.19173/irrodl.v18i2.2789
Cagiltay, N. E., Cagiltay, K., & Celik, B. (2020). An analysis of course characteristics, learnercharacteristics, and certification rates in MITx MOOCs. International Review of Research in Openand Distributed Learning, 21(3), 121-139.
Crossley, S. A., Paquette, L., Dascalu, M., McNamara, D. S., & Baker, R. S. (2016). Combining clickstream data with NLP tools to better understand MOOC completion. Learning Analytics andKnowledge. https://doi.org/10.1145/2883851.2883931
Das, K. R., & Imon, A. H. M. R. (2016). A Brief Review of Tests for Normality. American Journal of
Theoretical and Applied Statistics, 5(1), 5. https://doi.org/10.11648/j.ajtas.20160501.12
Galang, A. J. R., & Reyes, M. L. (2009). Motivational and social aspects of the Filipino college
experience. Philippine Journal of Psychology, 42(2), 213-235.
Kim, S., Jeong, S., Kim, H. J., & Jeong, Y. U. (2022). Academic Success of Online Learning in
Undergraduate Nursing Education Programs in the COVID-19 Pandemic Era. Journal ofProfessional Nursing, 38, 6–16. https://doi.org/10.1016/j.profnurs.2021.10.005
McKnight, P. E., & Najab, J. (2010). Mann-Whitney U Test. Corsini Encyclopedia of Psychology, 1.
https://doi.org/10.1002/9780470479216.corpsy0524
Moreno, M. M. L., Rodrigo, M. M. T., Torres, J. M. R., Gaspar, T. J., Agapito, J. L. (2021). Cura
personalis: Institutionalizing compassion during emergency remote teaching. In M. M. T. Rodrigo,
S. Iyer, & A. Mitrovic (Eds.), Proceedings of the 29th International Conference on Computers in
Education (pp. 608-613). AsiaPacific Society for Computers in Education.
Williams, K. M., Stafford, R. E., Corliss, S. B., & Reilly, E. D. (2018). Examining student
characteristics, goals, and engagement in Massive Open Online Courses. Computers &Education, 126, 433-442.

355

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Visual Attention Patterns in Processing
Compiler Error Messages
Christine Lourrine TABLATINa,b* & Maria Mercedes RODRIGOaaAteneo de Manila University, Philippinesb
Pangasinan State University, Philippines*tablatinchristine@gmail.com
Abstract: The difference in the visual attention of subjects while performing a
debugging task can be measured using fixation count and fixation duration metrics.
Thus, this paper investigated the visual attention patterns of high and low performing
students engaged in a defect-finding task on multiple programs using these metrics.
We performed statistical tests on the proportional fixation durations and fixation counts
on the error lines and the compiler error messages to determine the difference in thevisual attention patterns between the groups. The results of the statistical analysis
revealed a significant difference between the high and low performing students acrossall programs. This implies that high performing students were associated withsignificantly higher visual attention on the error lines of the programs than the low
performing students. However, the analysis of the proportional fixation duration andfixation count on the compiler error messages revealed no significant difference
between the groups. The results suggest that both groups showed similar visual
attention to the compiler error messages. The findings of this study provide insights
into the visual attention patterns of student programmers in processing compiler error
messages. High and low performing students could be distinguished based on theirvisual attention patterns on the error lines but not on the compiler error messages.
Further, high performing students prefer a more analytical processing approach andpay attention to relevant code elements in debugging to correctly identify the errors
while low performing students choose a more holistic approach.
Keywords: visual attention, compiler error messages, debugging, eye-tracking,fixation metrics

1. Introduction
Debugging is a critical skill that remains challenging to acquire for student programmers.
Students rely on compiler error messages to aid them in fixing the errors in their programs
while debugging them. Compiler error messages are pedagogically significant to student
programmers since they give feedback on what went wrong in their programs (Becker et al.,2019). However, according to Du Boulay & Matthew (1984), computer programming
students are unable to relate compiler error messages to actual code errors. One of thepossible reasons for this is that compilers are well-known for generating cryptic anduninformative error messages (Barik et al., 2017). This problem still exists and contributes toa high attrition rate of students in computer programming (Becker et al., 2018). Therefore,
researchers invested efforts in developing systems that offer students more insightful error
messages (Nienaltowski et al., 2008; Dy & Rodrigo, 2010; Denny et al., 2014; Prather et al.,2017). The work of Nienaltowski and colleagues (2008) found that providing additional
information regarding an error did not necessarily result in greater debugging accuracy.
Denny et al. (2014) corroborated their finding. In contrast, the study of Prather, et al. (2017)
found that enhanced compiler error messages are more helpful than standard compiler error
messages through an eye tracking study.
By observing people's eye movements, researchers can determine what attracts
them and possibly understand how information is interpreted (Bol et al., 2017). As a result,researchers have used eye tracking to measure the difficulty student programmers

356

encounter in understanding the error messages they read. Barik et al. (2017) found thatinterpreting error messages is as challenging as reading source code. Some of the reasonscomes from the lack of familiarity with error messages (Marceau et al., 2011) and the needfor programmers to move between the error message and source code to understand the fullcontext of the problem. Other researchers have examined the reading patterns that students
employ to parse through code to find an error and how these patterns differed depending on
student ability. The source codes used in some studies were written in C (Chandrika &Amudha, 2017; Nivala et al., 2016; Sharif, Falcone & Maletic, 2012), C++ and Python(Turner et al., 2014), and Java programming languages (Villamor & Rodrigo, 2022; Tablatin& Rodrigo, 2022). Although research on understanding how programmers read andcomprehend source code has been conducted, research on processing compiler error
messages is still limited, especially using C++ as the source code stimuli. Through an eyetracking study, we hope to add to what is known about how student programmers read andprocess compiler error messages. In this paper, we investigated the visual attention patternsof high and low performing students in processing compiler error messages.

2. Methodology
2.1 Participants
The participants of this study were Information Technology students who have at least taken
a college-level introductory programming course using C++ as the programming language. A
total of 32 students aged 18 years old and above were recruited from a state university in thePhilippines. The study used two participant groups: high performing and low performing. The
scores of the participants in the debugging tasks were used to assign them to a particulargroup. High performing group consisted of students who scored above and equal to themean score while the low performing group consisted of students who scored lower than themean score.
2.2 Experimental Setup and Procedure
All participants underwent a screening process to determine their eligibility to take part in the
study. Students who passed the initial screening were given an informed consent form to fillout and sign. The students who signed the informed consent forms were given two types ofpre-tests: a personality test and a self-efficacy test. After the pre-tests, the participants wererequired to undergo a calibration test before starting with the eye-tracking experiment thatwas designed to last for about 30 minutes.
The experiment was conducted in a laboratory setting. The experimental setup
consisted of a laptop, a 17-inch monitor, a mouse, and a Gazepoint GP3 table-mounted eye
tracker with a sampling rate of 60 Hz and an accuracy range of 0.5–1 degree. The hardware
was set up by extending the laptop's display to the monitor connected to it. The eye tracker
was placed in front of the monitor to allow the participant to view the program codes while
the eye tracker records the participant’s eye movements. The participant sat in front of themonitor, eye tracker, and mouse. After successful calibration of the eye tracker, a custombuilt slide viewer loaded with C++ programs was activated. The slide viewer enabled theparticipant to navigate through the programs using Previous and Next buttons. It alsoenabled the participant to mark error locations with a red ellipse using the mouse. A Resetbutton can be used to clear the marked errors on a given slide and a Finish button saves theparticipant’s answers and ends the session.
2.3 Comprehension Task and Measures
The main task of the participants was to find the errors of the programs based on thecompiler error message. Each program had been injected with one syntax error and there isno need to correct them. Participant performance was measured based on the correctness357

of the answers. Eye tracking metrics were also collected to measure the visual attention
patterns of the students while performing the debugging task.
2.4 Stimuli Characteristics
The code pool size was limited to 5 unique programs with different syntax errors. The
programs shown to the participants were brief (15 to 35 lines) and represent C++ constructs
discussed during the first 6 to 8 weeks of a first collegiate programming course. These
include data types, variables, operations, conditionals, and loops. Further, the level ofdifficulty of the code was limited to the types of exercises given to students who are being
introduced to the constructs mentioned herein. We have limited the error types to syntax
errors that are usually encountered by students who are trying to complete these
programming exercises. Only 1 syntax error was injected in each program and no semantic
or logical errors were included. We have chosen 5 syntax errors from the syntax errors
identified in the study of Denny, et al. (2014) to be injected in the programs. Note that,
despite this limitation, some of the errors may be non-literal. That is, they may notaccurately reflect the error or its location. The programs used in the experiment are novicefriendly, self-contained, and did not require any special domain knowledge. The language
used was familiar to the students, brief enough to fit on a single screen, and did not use
unfamiliar APIs. The stimuli used in the experiment were static. Participants only have topoint to the location of the error in each program. They do not need to correct it.
2.5 Data Pre-processing
The gaze movements of the participants were stored in a CSV file format. The time of therecording (timestamp) when fixations occur, the location of the fixations (values of x and ycoordinates), and the fixation duration of each fixation were extracted from the CSV file forthe visual attention patterns analysis. Areas of Interests (AOIs) of the 5 programs weredrawn using the OGAMA Areas of Interest module (Vosskühler, 2009) to get the AOI
coordinates. The AOIs marked in this study are the line where the error is located and thecompiler error message of each program. Figure 3 shows the AOIs of one of the programs
used in this study. The AOI coordinates extracted from OGAMA returns coordinates withrespect to the setting of the screen resolution specified when the AOIs were defined. To map
the location of fixations to the program codes, the x and y coordinates from the eye-tracking
data were converted by multiplying the x coordinates with 1366 and the y coordinates with
768. This was done to match the coordinates of the program codes during the experimentsince the screen resolution used was 1366 x 768. In addition, the fixation durations wererecorded in terms of seconds by the eye tracker and were converted into milliseconds bymultiplying the duration with 1000. These processes were done for the eye-tracking data of
the 32 students to determine their visual attention patterns.
2.6 Data Analysis
To determine the difference in the visual attention patterns of high performing and low
performing students, fixation count and fixation duration were used. The proportional fixation
count and proportional fixation duration on the error lines and the proportional fixation countand proportional fixation duration on the compiler error messages were computed tomeasure the visual effort exerted by high performing and low performing students.
Statistical analysis was conducted to compare the visual efforts between the highand low performing students using the visual effort metrics stated above. Independent
samples t-test was used to determine whether there is statistical evidence that the visualefforts are significantly different between high and low performing students.

358

3. Results and Discussion
The eye tracking data of 32 students from a state university in the Philippines were used in
the analysis. Eighteen (18) students were identified as high performing while fourteen (14)students were considered low performing based on their debugging scores. The proportional
fixation duration and count on the identified AOIs were computed as a ratio of fixation
duration and fixation count on an AOI to the overall fixation durations and counts on eachprogram, respectively. These data were used to determine the difference in the visual
attention patterns of students while finding errors in the program.
3.1 Visual Attention Patterns on the Error Lines
To determine the difference in the visual attention patterns of the groups in the error
lines, independent samples t-tests were used. An independent samples t-test was used todetermine if there is a significant difference in the average proportional fixation duration of
the high and low performing students. The result of the analysis revealed that there is asignificant difference in the average proportional fixation duration of high performing students(M = 0.014, SD = 0.007) and low performing students (M = 0.007, SD = 0.002), t(17.823) = 3.738, p = .002. The result suggests that the average proportional fixation durations of thehigh performing students are significantly higher than the low performing students across all
programs. Further, an independent samples t-test was also performed to determine if there
is a significant difference in the visual attention of high and low performing students in termsof the average proportional fixation count on the error lines. The result of the analysis
revealed that there is a significant difference in the average proportional fixation count of thehigh performing students (M = 0.013, SD = 0.006) and low performing students (M = 0.007,
SD = 0.002), t(16.411) = -4.165, p = .001. The result suggests that the average proportional
fixation count of the high performing students is significantly higher than the low performing
students across all programs.
The results of the analysis of the visual attention patterns suggest that high performing
students were associated with significantly higher proportional fixation durations and fixation
counts on the error lines of the programs than the low performing students. Fixation
durations are influenced by the complexity and difficulty of the visual content, task being
performed (Bylinski, et al., 2015; Nivala, et al., 2016), and AOIs that are engaging the
cognitive resources of the observer (Bylinski, et al., 2015). High performing students haveexerted more visual attention in terms of the proportional fixation duration on the error line toconfirm the compiler error message that error does exist in that line. This finding
corroborates the findings of Chandrika & Amudha (2017) and Sharif et al. (2012) that experts
tend to concentrate more on areas where the errors are located while novices read thecodes more broadly. In contrast with the findings of the latter, Nivala, et al. (2016) andTurner, et al. (2014) observed that novices spent more time on the buggy lines of code thanexperts. The difference in the findings of this study with previous studies may be related tothe characteristics of the stimuli used and the tasks employed by the researchers during theeye tracking experiment. Furthermore, the number of fixations on an AOI can be linked to its
importance (Bylinski, et al., 2015). Thus, more fixation counts can be observed from the highperforming students on the error lines. This result is in line with the findings of (Chandrika &Amudha, 2017; Sharif, et al., 2012; Turner, et al., 2014), that experts or advanced
programmers have more fixation counts on the buggy lines of code.
3.2 Visual Attention Patterns on the Compiler Error Messages
An independent samples t-test was performed to determine the difference of thevisual attention of high and low performing students on the compiler error messages in termsof the average proportional fixation duration. The result of the statistical analysis revealed
that there was no significant difference in the visual attention of the high (M = 0.023, SD =0.016) and low performing students (M = 0.018, SD = 0.024), t(16.066) = -0.696, p = .497.
The result suggests that the average proportional fixation duration on the compiler error

359

messages of the high and low performing students is similar for both groups across all
programs. The same statistical test was used to determine the difference of the visual
attention of the groups on the compiler error messages in terms of the average proportional
fixation count. The result revealed that there was no significant difference in the visual
attention of the high (M = 0.025, SD = 0.015) and low performing students (M = 0.018, SD =0.022), t(16.256) = -0.926, p = .368. The result suggests that the average proportional
fixation counts is similar for both high and low performing students across all programs.
The results of the analysis of the visual attention patterns imply that both high and low
performing students have similar average proportional fixation durations and fixation counts
on the compiler error messages. Although no significant difference was observed, highperforming students have slightly higher average proportional fixation duration and fixation
count on the compiler error messages. The visual behavior of high performing students may
be related to the field-independent (FI) cognitive style theory of human cognition. The FI
individuals tend to choose a more analytical processing approach and they pay attention to
relevant details. Conversely, the visual behavior of low performing students may be related
to field-dependent (FD) cognitive style wherein they choose a more holistic way ofprocessing visual information and experience difficulties in identifying relevant details in thecomplex visual stimulus (Raptis et al., 2017).

4. Conclusion and Future Works
This study contributes to the evidence of the effectiveness of eye tracking as a method toenrich computing education research. The analysis conducted using the proportional fixation
duration and fixation count provided considerable insights about the visual attention patternsof high and low performing students in processing compiler error messages. High performing
students exert more visual attention on the lines indicated by the compiler error messages.
They prefer a more analytical processing approach and pay attention to details while low
performing students choose a more holistic approach. We may conclude that more proficient
students read error messages as much as the less proficient students but limit their reading
to the line number and not to the whole description of the error message. By exploring thevisual strategies employed by the high performing students using eye tracking data, we
could develop learning materials and activities that could help low performing students
improve their code reading and debugging skills. Further, debugging should be taught as aprogram comprehension task rather than a search task. Students should focus their attention
on the lines mentioned in the compiler error message to identify source code defects
because it provides feedback on the location and description of the error. Teaching students
to consciously employ debugging strategies would enhance their debugging ability and will
help in increasing the retention rate of students taking programming courses.
Future analysis on the collected eye tracking data will be conducted to determine if
students with different programming ability, self-efficacy levels, and personality
characteristics vary in the way they read and process compiler error messages.

Acknowledgements
We thank Ateneo de Manila University and Pangasinan State University administration forallowing us to conduct the experiment for this study. We also thank the Dagstuhl Seminar
22402 on Foundations for a New Perspective of Understanding Programming that enabled
us to identify the shortcomings in our previous study design and helped us improved thestudy design of this paper.

References
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017,360May). Do developers read compiler error messages?. In 2017 IEEE/ACM 39th International
Conference on Software Engineering (ICSE) (pp. 575-585). IEEE.
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare,A., McDonald, C., Osera, P-M., Pearce, J. L & Prather, J. (2019). Compiler error messages
considered unhelpful: The landscape of text-based programming error message research.
Proceedings of the working group reports on innovation and technology in computer scienceeducation, 177-210.
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018, February). Fix thefirst, ignore the rest: Dealing with multiple compiler error messages. In Proceedings of the 49th
ACM technical symposium on computer science education (pp. 634-639).
Bol, N., Boerman, S.C., Romano Bergstrom, J.C., & Kruikemeier, S. (2016). An Overview of How Eye
Tracking Is Used in Communication Research. In: Antona, M., Stephanidis, C. (eds) Universal
Access in Human-Computer Interaction. Methods, Techniques, and Best Practices. UAHCI 2016.
Lecture Notes in Computer Science(), vol 9737. Springer, Cham. https://doi.org/10.1007/978-3319-40250-5_40
Bylinskii, Z., Borkin, M. A., Kim, N. W., Pfisher, H., and Oliva, A. (2015). Eye fixation metrics forlarge-scale evaluation and comparison of information visualizations. In Eye Tracking and
Visualization, eds M. Burch, L Chuang, B. Fisher, A. Schmidt, and D. Weiskopf (Cham:
Springer), 235-255. Doi: 10.1007/978-3-319-47024-5_14
Chandrika. K. R., and Amudha, J. (2017). An Eye Tracking Study to Understand the Visual
Perception Behavior while Source Code Comprehension. International Journal of Control Theory
and Applications. International Science Press vol. 10(19), 169-175.
Denny, P., Luxton-Reilly, A., & Carpenter, D. (2014). Enhancing syntax error messages appears
ineffectual. In Proceedings of the ITiCSE '14: Proceedings of the 2014 conference on Innovation& technology in computer science education (pp. 273-278).
Du Boulay, B., & Matthew, I. (1984). Fatal error in pass zero: How not to confuse novices.
Behaviour & Information Technology, 3(2), 109-118.
Dy, T., & Rodrigo, M. M. (2010, October). A detector for non-literal Java errors. In Proceedings of the10th Koli Calling International Conference on Computing Education Research (pp. 118-122).
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011, October). Mind your language: on novices'interactions with error messages. In Proceedings of the 10th SIGPLAN symposium on New
ideas, new paradigms, and reflections on programming and software (pp. 3-18).
Nienaltowski, M. H., Pedroni, M., & Meyer, B. (2008, March). Compiler error messages: What canhelp novices?. In Proceedings of the 39th SIGCSE technical symposium on Computer scienceeducation (pp. 168-172).
Nivala, M., Hauser, F., Mottok, J., and Gruber, H. (2016). Developing visual expertise in software
engineering: An eye tracking study. 2016 IEEE Global Engineering Education Conference(EDUCON), 613-620.
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017, August).
On novices' interaction with compiler error messages: A human factors approach. In Proceedingsof the 2017 ACM Conference on International Computing Education Research (pp. 74-82).
Raptis, G. E., Katsini, K., Belk, M., Fidas, C., Samaras, G., & Avouris, N. (2017). Using Eye Gaze
Data and Visual Activities to Infer Human Cognitive Styles: Method and Feasibility Studies. In
Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization (UMAP'17). (Bratislava Slovakia, 2017), ACM, New York, NY, USA, 164-173.
Sharif, B., Falcone, M. and Maletic, J. I. (2012). An eye-tracking study on the role of scan timein finding source code defects. In Proceedings of the Symposium on Eye Tracking Research andApplications (ETRA’12), ACM, 381–384.
Tablatin, C. L. S., & Rodrigo, M. M. T. (2022). Identifying Code Reading Strategies in
Debugging using STA with a Tolerance Algorithm. APSIPA Transactions on Signal andInformation Processing, 11(1).
Turner, R., Falcone, M., Sharif, B., and Lazar, A. (2014). An eye- tracking study assessing thecomprehension of C++ and Python source code. In Proceedings of the Symposium on Eye
Tracking Research and Applications (ETRA '14). (Florida, 2014) ACM, New York, NY, USA, 231234.
Villamor, M. M., & Rodrigo, M. M. T. (2022). Predicting Pair Success in a Pair Programming Eye
Tracking Experiment Using Cross-Recurrence Quantification Analysis. APSIPA
Transactions on Signal and Information Processing, 11(1).
Vosskühler, A. (2009). OGAMA description (for Version 2.5). Berlin, Germany: Freie Universität
Berlin, Fachbereich Physik. Retrieved April 23, 2023 fromhttp://www.ogama.net/sites/default/files/pdf/OGAMA-DescriptionV25.pdf

361

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Process Evaluation for Concept Map
Building and Its Experimental Evaluation
Ridwan RISMANTOa,c*, Aryo PINANDITOb, Banni Satria ANDOKOc, Yusuke HAYASHIa &Tsukasa HIRASHIMAaa

Graduate School of Advanced Science and Technology, Hiroshima University, Japanb
Department of Information System, Brawijaya University, Indonesiac
Department of Information Technology, State Polytechnic of Malang, Indonesia*rismanto@polinema.ac.id
Abstract: Concept map as an assessment tool is widely accepted in the field of
education. In this research, assuming the process evaluation of concept map building
leads to better evaluation than the product (that is, built concept map) evaluation, we
proposed a process evaluation method and conducted an experimental evaluation of
the method. The process evaluation method is intended for re-composition type ofconcept maps, not for general ones. In the re-composition concept map, a learner isprovided a set of components of a concept map, that is, links and nodes, and required
to recompose a concept map by using the components. The provided components are
generated by decomposing the concept map built by an expert as a representation ofunderstanding of the learning target. In this research, we set several submaps with aset of semantics in one expert map and define a good process by recomposing thesubmap as a series of activities. The process evaluation results were compared withthe product evaluation results for the data of a practical use of re-composition ofconcept map in undergraduate class. As the results of the comparison, we have alreadyconfirmed that the result of process evaluation was a better predictor of learner’s
understanding than the results of the product evaluation. In this paper as an additional
analysis, we reported the analysis of relation between process evaluation and productevaluation. The difference between the evaluation methods could result in a casewhere there is zero score for process evaluation and full score for product evaluation.
However, analysis results confirmed that there is a strong correlation between the two
evaluation methods with r value = 0.8. These results suggest that support for theprocess of submap re-composition is a promising way for learners to recompose a goodconcept map. We further explore the usability of the process evaluation to the teachers,
and it shows a high rate of agreement of the usefulness and appropriateness of themethod. Based on the results, we are developing a formative assessment and support
environment for concept map re-composition.
Keywords: Concept map, concept map assessment, process evaluation, Kit-build

1. Introduction
Concept map, developed by Novak in 1972 (Novak & Musonda, 1991) is widely
accepted as a promising activity for learning and evaluation of learner’s understanding andfacilitate independent learning and thinking (Hu & Wu, 2012). Studies of concept map in theassessment across multiple disciplines have conducted (Gregoriades et al., 2009; Kinchin etal., 2000; Yin et al., 2005). Many studies describe the assessment criteria such as structuraland relational scoring (McClure & Bell, 1990; Novak & Gowin, 1984). These assessment
methods are accepted widely but purposed to assess the product of concept map, not theprocess of concept map building.
Research in concept map assessment mentioned that to realize sophistication andadaptation of the learning and evaluation, assessment of the process of concept map building362

is an important issue. A study by Jablokow (2015) suggests the importance of collecting data
about learner’s actual mapping process that could reveals an insight about cognitive aspect
implicit in a concept map building that related to learner’s understandings. Concept map
building process also shown that produced map quality has a relationship with the order ofcreated propositions (Srivastava et al., 2021) and sequential mapping (Chiu & Lin, 2012).
However, to realize the observation of the concept map building process is still a big issue,
because most proposed method to evaluate the concept map quality and the process ofconcept map building relies upon human-dependent, qualitative approaches to conduct theobservation (McAleese, 1998; Rautama et al., 1997; Wong & Lian, 1998) which generally lacks
consistency and a demanding time and efforts.
In this research, we proposed an automatic process evaluation method and conducted
an experimental evaluation of the method to evaluate its relationship with the quality of the
produced map. The proposed method is intended for a re-composition type of concept maps,
not for a general one where learner builds a concept map from scratch. We applied the
proposed method in Kit-Build (KB), a concept mapping re-composition framework that allows
knowledge sharing between teacher and learner and allows automatic assessment of learner’s
concept map (Hirashima et al., 2011, 2015). It is known that KB automatic assessment could
be used to assess learner’s map quality and the validity has been checked to be similar withother scoring methods (Wunnasri et al., 2018). As an assessment of learner’s understanding,
KB map score has proven to have a high correlation with learner’s actual understanding(Yoshida et al., 2013).
Since the components of building a concept map are provided in the KB, it is possible
to record the created propositions that are matched in comparison with the propositions fromteacher’s build map. To realize the process evaluation, we use the concept of sub-map as a
substructure in a concept map. We define several sub-maps with a set of semantics in theteacher’s map and define a good process by re-composing the sub-map as a series ofactivities. Then we analyze the data from learner’s KB concept map activity in respective tothe defined sub-maps. We already confirmed that the process evaluation has a relation withtest scores and therefore related to learner’s understanding (Rismanto et al., 2023).
In this paper, we report the results of the analysis of the relation between the processevaluation and learner’s map quality, that is, the product evaluation. If there is high correlation,
it means that the quality of the map can be improved by supporting the process. To realize theanalysis, the process evaluation results were compared with the product evaluation resultsfrom the data of a practical use of KB in an undergraduate class. We further explore theusability of the process score to the teacher, in the context of evaluating the process evaluationfrom the teacher's perspective. To realize it, the process evaluation results were comparedwith the product evaluation results from the data of a practical use of KB in an undergraduate
class. We further explore the usability of the process score to the teacher, in the context ofevaluating the process evaluation from teacher’s perspective.
Therefore, the research questions for this study are as follows: Does the concept map
building process have a relation to the product score (RQ1)? And does the process score isappropriate and useful from the perspective of teachers (RQ2)?

2. Background
2.1 Concept Map Assessment
Concept map is a widely recognized method to visualize knowledge in a graphicalformat (Novak & Gowin, 1984). It is a simple but powerful representation of knowledge tosupport the learning activity between teacher and learner. To represent a concept map, agraph is constructed by using only two symbols: Nodes that represent concepts, and links that
represents relationship between concepts (Cañas et al., 2016; Dwi Prasetya et al., 2020). In
an educational purpose, concept maps have been used such as for sharing meanings
between people, planning the problem-solving process, representing, and assessing thelearner’s knowledge structure.

363

Many methods and have been studied and proposed in utilizing concept map as an
assessment tool (Plotnick, 1997; Yin et al., 2005; Zheng et al., 2019). A person who can wellunderstand the meaning of words in the concept map, called “rater”, manually works to assess
the concept map. Novak and Gowin (1984) proposed a structural scoring. This method gives
a high score for the correctness of hierarchy level and each valid crosslink. It is because
ordering the concepts into hierarchy and connecting the crosslinks can reflect learner’s
creative thinking. Hence, this method is called structural scoring.
Other assessment methods that consider the meaning of a proposition instead of thestructure have been studied. McClure and Bell (McClure & Bell, 1990) proposed the relationalscoring. This method works by checking the relationship possibility between each proposition,
concept label appropriateness and compatibility between labels. The reliability and validity ofthe method has also been studied. McClure et al. (Mcclure et al., 1999) investigated theconcept map assessment by requesting students to construct a concept map using provided
concepts and creating the link words to connect between concepts. Raters then scored eachindividual map separately. Result shows that the relational scoring method has a close relation
with structural scoring method, using criteria map (teacher-build map).

2.2 Kit-build Framework and Automatic Assessment
Kit-build (KB) is a framework in concept map knowledge sharing and automatic
assessment (Hirashima et al., 2011, 2015). In this framework, the learner creates a concept
map by recomposing existing components called “kit” which consists of nodes and links. The
nodes and links are components that decomposed from an “expert map” (teacher-build map).
The teacher constructs the expert map by extracting from the learning material. This methodof concept mapping is a re-composition concept map, which means that instead of creating
their own concept map from scratch, learners recompose it from the kit. The map constructedby learner (learner map) then superimposed with the expert map to determine the matching
propositions.
There are two main tasks in the KB framework (Yamasaki et al., 2010). The first is thesegmentation task, in which the teacher is requested to prepare the teacher-build map, called
the expert map. The expert map is extracted from the learning material and is the teacher’s
expression of a comprehension of the learning topic. The expert map is then submitted to thesystem and decomposed into “kit” which contains a list of concepts and links from the expert
map. The kit then provided to the learners to be recomposed into a learner map. Hence, whenutilizing the kit, students are just required to identify the provided components and link them ,no component creation is required. Figure 1 (left) shows an example of an expert map and akit.
The second task is restructuring task. Learners are asked to restructure the concept
map from the provided kit, creating the learner-build map, or learner map. Figure 2 (left) shows
an example of learner map. After completing the restructuring, the learner map was thenuploaded to the system for the analysis. The system then evaluates the learner map by
superimposing it with the expert map. The automatic assessment works by matching eachlearner’s proposition with the expert map propositions.

Figure 1. Teacher-build map, or expert map (left) and Kit (right).

364

Propositions from the learner map that match with propositions from the expert map
will be given a score. The total score is the evaluation result of the concept map product, called“map score”. Map score is the percentage of the matching propositions number from thelearner map compared with the total amount of propositions from the expert map. Figure 2(right) shows the learner map superimposed with the expert map. The solid lines are thecorrect propositions, and the dashed lines are the incorrect and lacked propositions incomparison with the expert map. Map score has been known with a high relationship with thelearner’s understanding (Yoshida et al., 2013) which implies that learners with a high map
score tend to have better knowledge compared with the ones with low map score. As an
assessment of concept map, map score has been validated with other concept map productscoring method and comparable to the manual assessment methods (Wunnasri et al., 2018).

Figure 2. Learner map (left) and learner map superimposed with expert map (right).

2.3 Process Score
The process score proposed in this study aims to assess learner’s process in the KB
concept map recomposing activity. The current scoring system of KB, the map score, is basedon the produced concept map built by learners using the provided components (Jablokow etal., 2015). Our proposed method purposed to evaluate learner’s actual concept mapping
process by analyzing the sequence of connected propositions in KB concept map
recomposing activity. As mentioned by Chiu and Lin (2012) and Srivastava (2021), thesequence in concept map building process is related to the map quality built by learner where
good process produces a good quality concept map.
In realizing the automatic process evaluation of concept map building, we analyzed it
as a set of sequences, grouped by sub-maps. Sub-map is subset composed of closely related
propositions in a concept map. Sub-map defined the concept map’s substructure (Gerstner &Bogner, 2009), that demonstrate the teacher’s deep understanding as the subject specialist(Kinchin & Alias, 2005; Roberts, 1999). Therefore, achieving it is an important target forlearners. The re-composition process suggests that a learner has a solid comprehension of
the substructure indicated by the sub-map when they continually recompose a collection of
propositions in a sub-map.
The sub-maps are defined by the teacher after creating the expert map. Generally, a
sub-map contains propositions that have a close meaning or extracted from the same
subdomain of knowledge. As mentioned by Schneider et al. (2021), segmenting the concept
map into a group of sub-map can be based on a particular knowledge domain of the concept
map. This is in line with spatial continuity principle stated in Moreno and Mayer (1999) andSchroeder and Cenkci (2018), in which the information in a concept map is spatially orderednearby within a learning material. Figure 3 (left) shows the example of sub-maps in a conceptmap.
After the sub-map is defined, learners are then given the task to recompose the kit
back into the learner-build map, or learner map. In this stage, learners are not able to see thesub-map. The process scoring is transparent from the learners, as they will be performing therecomposing activity as usual. After completing the activity, learners will submit the map to thesystem and be given the map score and process score.

365

To calculate the process score, we analyze the sequence of connected propositions.
We scanned the learner's propositions that are sequentially connected as a set andcorresponds to the propositions in each sub-map. The percentage of matched propositions
sequence set for each sub-map is then calculated, which is, the number of connected
propositions as a set divided by the number of propositions in a sub-map. As long as the
propositions that are connected sequentially belong in a sub-map, the order of the proposition
sequence in that sub-map is not considered.
For example, there are two sub-maps: Sub-map 1 (SM1) contains proposition 1 (P1),P2, P3. And SM2 contains P4, P5, P6. If a learner’s connected proposition sequence is P2,
P3, P5, P4, P6, P1 then the calculation will be as follows:
P2, P3 connected sequentially as a set and belong to SM1. Therefore, the SM1 scoreis 2/3 * 100 = 66.67. Moving forward, P5, P4, P6 connected sequentially as a set and belong
to SM2. Therefore, the SM2 score is 3/3 * 100 = 100. The process score is the average of thetotal score of each sub-map (SM). So, this learner will get a process score of (66.67 + 100) /2 = 83.33. Figure 3 (right) shows the illustration of the process score calculation.
Sequenceorder (O)

P1

SM1
P2

P3

P4

O2

P3

P5

O3
P4

O4
O5

P

P6

P2

O1

O6

SM2
P5

P6
P1
: Proposition number: Set of propositions connected sequentially

Figure 3. Sub-map in a concept map (left) and example of process score calculation (right).

3. Method
3.1 Experiment details
The study subjects were 28 second year undergraduate students majoring inInformation Technology at State Polytechnic of Malang, Indonesia. Subjects have no priorexperience or knowledge in building a concept map and in using the KB system. At the firstsession of the experiment, subjects were given a tutorial about creating a concept map andthe usage of KB system. At the next session, the teacher explained about the learningmaterials in the form of lecture using power point slides. Further, subjects were tasked toperform KB concept map recomposing activity. After completing the recomposing, subjects
upload the learner map into the system to be scored for a map score and process score. The
total time for the experiment sessions is 60 minutes.
The learning subject is Polymorphism in Object Oriented Programming (OOP). The
material was taken from the OOP course module released by the team teaching of the course
at the Information Technology department at the State Polytechnic of Malang, Indonesia. The
expert map was extracted from the module by the author of this paper and presented to thethree teacher that has teaches OOP subjects for undergraduate students. The expert map
was then discussed and modified according to the team teaching’s input, and the sub-map
was also defined. The expert map consists of 10 propositions, 10 links, 11 concepts and fivesub-maps.

3.2 Analysis366

To assess the relation between KB concept map building process reflected by processscore and learner’s map quality reflected by map score, a correlation test between processscore and map score was performed. This assessment is based on the research by Chiu and
Lin (2012) and Srivastava et al. (2021) that mentions the strong relationship between concept
map building process and the quality of concept map product. Furthermore, a multiple linear
regression analysis between map score and process score was conducted to check whether
the process score has a strong influence and can be a predictor to the map score.
Detailed analysis of the sub-maps was carried out to confirm that the process analysis
could provide useful information for teachers in recognizing parts of the concept map in
relation to learner’s map quality. Based on the sub-structure of concept map that is defined bysub-maps (Gerstner & Bogner, 2009; Kinchin & Alias, 2005), we conduct this analysis toevaluate how each sub-map correlates to the quality of concept map product. The information
can be used for teachers to provide adequate feedback for learners to improve their quality of
produced concept map. To analyze the parts of the concept map that have a strong influenceon learner’s map quality, correlation test was performed between each sub-map score andmap score. Multiple linear regression analysis was performed to confirm that the highest
correlated sub-map score with map score, can also be a predictor to the learner’s map quality.
To evaluate the teacher’s perspective on the reasonability and perspective of theprocess scoring method, a questionnaire for teachers that have taught OOP subjects forundergraduate students and have familiarities with KB concept mapping has been conducted.
The questionnaire was asked after observing the experiment and the method of processscoring and evaluating the results. We explained to the teachers how the process evaluation
method works and its result, and the relation between process score and product score.
Therefore, the teachers answered the questionnaire based on the relation between processscoring result and product scoring result. The first part of the questionnaire is about thereasonableness of the scoring method (Wunnasri et al., 2018). And the second part is theirperspective of usability of the sub-map evaluation.

4. Result and Discussion
To check the relationship between process score and learner’s map quality whichreflected by the map score, a Pearson correlation test was performed, and the result can beseen in Figure 4. According to the result of the correlation test, the R value of 0.8 means thatprocess score has a high correlation with the map score. With p-value of 3.4e-07 which is lessthan significance level alpha = 0.05, it is concluded that process score and map score are
significantly correlated.
This result means that learners with a high process score tend to have a better map
quality compared to the ones that have a low process score. In a cognitive point of view, theprocess of recomposing the kit into concept map, reflected by the proposition sequence, could
be an indication to assess the understanding level of learners, as mentioned by Yoshida et al.
(2013) that the map score has a high correlation with learner’s understanding.

Figure 4. Correlation between map score and process score

367

Multiple linear regression analysis was carried out between map score and processscore, to check that the concept map building activity process can be a predictor of learner’s
map quality. Result shown in Table 3 concluded that with a p-value of 3.45e-07 which is lessthan significance level alpha = 0.05, process score has a strong influence on the map score.
Therefore, the process score can be used to predict the map score. This result combined withthe correlation test result concluded that the process score could represent the learner’s map
quality reflected by the map score.
Table 2. Multiple linear regression analysis between map score and process score
Estimate
Std. Errort value(Intercept)
16.3931
10.3655
1.582
sms
0.985
0.1454
6.774
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Pr(>|t|)0.126
3.45E-07***

Detailed analysis on the sub-map has been carried out to check the significant submap. Pearson correlation was used to check the correlation between map score and sub-map
1 score (SM1), SM2, SM3, SM4 and SM5. The results shown in Figure 5. It is shown that submap 2 (SM2) has the highest correlation with map score compared with the other sub-maps
with R value = 0.63. Multiple linear regression analysis between map score and each sub-map
has been carried out to check the significance of the SM2 on the influence with the map score.
Result shown in Table 3 concluded that the SM2 has high influence with the map scorecompared to the other sub-maps and can be a predictor of the map score.
The correlation analysis result combined with the multiple linear regression analysis
result shows that the SM2 is the significant sub-map, that is part of concept map that has astrong influence on learner’s map quality. Learners that can connect propositions in the SM2
as a set of sequence tend to have a good map score, therefore, have a good understandingof the learning subjects. In the context of formative assessment, teachers can use thisinformation to provide adequate feedback emphasized on the knowledge domain represented
in the SM2 to further improve learner’s map quality. Improved map quality leads to a better
understanding of the learning subjects, since it represents the learner’s knowledge (Novak &Gowin, 1984).

Figure 5. Correlation map between map score and each sub-map (sm).

368

Table 3. Multiple linear regression analysis between map score and each sub-map
Estimate
Std. Error(Intercept)
16.74635
12.23106sm1
0.229160.12185sm2
0.264480.11127sm3
0.05553
0.14394sm4
0.214710.11667sm5
0.17877
0.09257
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

t value
1.369
1.881
2.377
0.386
1.84
1.931

Pr(>|t|)0.18480.07330.02660.70340.07930.0664.
*.
.

Furthermore, a questionnaire to the three teachers that have taught OOP subjects forundergraduate students and have familiarities in KB concept mapping was given tounderstand their perspectives on the KB process scoring. This questionnaire was asked after
observing the experiment, the method and process evaluation result and sub-map analysis
result. Thus, the teacher answered the questionnaire based on the relationship between
process scoring and product scoring results. The appropriateness of process score evaluation
method was asked in the first part, and the usability of sub-maps evaluation was asked in thesecond part.
The result of the teacher’s perspective questionnaire shown in Table 4. In the firstquestion about the appropriateness of the process scoring method, it is noted that the processscore is an appropriate method to evaluate learner’s process in KB concept map re-composing
activity. In the second question, as an alternative to the map score, one teacher disagrees
with the process score as an alternative assessment for KB. When asked for the reason, theteacher suggested that the process score is to be used as a complementary score to the map
score. This is so that the results of the process score become an additional consideration fora teacher in assessing learner’s understanding. In the third question about the detailed
analysis of the sub-maps, the teachers noted that the evaluation of the sub-maps in relation
to product score is an appropriate method to investigate parts of sub-maps that highly
correlated with product score. Finally in the last question, it is noted the provided information
can be used for the teacher to provide adequate feedback in emphasizing the important
domain knowledge represented by the highest correlated sub-map.
Based on the results of the questionnaire, we concluded that the teachers agreed with
the necessity of process evaluation. The teachers agreed that the submap based processevaluation method is a reasonable one. Based on these agreements, they judged the processevaluation scores would be useful for learning support. Therefore, based on these results, weconcluded that the process evaluation is a promising method, and it is worth continuous
investigation.
Table 4. Questionnaire on the teacher’s perspective
Strongly
disagree

Questions
Appropriateness of process score evaluation method
Process evaluation on kit-build is an appropriate method
to evaluate learner's process in kit-build concept mapping
re-composing activity.
Process scores can be used as an alternative
assessment for kit-builds other than map scores.
Usability of sub-maps evaluation
Analysis of sub-maps is an appropriate method to findout which parts of a concept map affect the map-score.
By understanding the important sub-maps and subdomain of knowledge, teacher can provide appropriate
feedback to improve student understanding.

369

Disagree33.30%

NeutralAgree

Strongly
agree

66.70%

33.30%

66.70%
66.70%

33.30%

33.30%

66.70%

5. Conclusion
Assessing the learner’s process in the KB concept map re-composition activity reveals
that process has a relationship with learner’s map quality. In this study, we proposed a
concrete technique to assess the process of concept map re-composition in KB concept
mapping. Based on the KB map score that can represent learner’s map quality, correlation
analysis between process score and map score shows that process score has a highcorrelation with map score. Therefore, it answered the first research question (RQ1) thatprocess in KB concept map building activity has a relation with learner’s map quality.
Detailed analysis on the sub-map shows that there is a significant sub-map that has ahigh correlation with the map score. This analysis could provide information for teachers torecognize part of the concept map that represents a knowledge domain that has theimportance of the learner’s map quality. Teachers can use this information to provide adequate
feedback to improve learner’s map quality and therefore, improve their understanding. The
questionnaire result shows that the process scoring method is adequate and provides useful
information in the perspective or teachers. Thus, it answered the second research question(RQ2).
Based on the questionnaire result, we concluded that the process evaluation is apromising method, and it is worth continuous investigation. The limitation of the proposed
method is the order of sub-map is not considered. Therefore, further study to conduct more
analysis and experiment could expand another aspect of KB concept map re-composing
process. As a future works, the bigger size of concept map and number of participants canstrongly confirm the conclusion of this study. Furthermore, building an environment to support
concept map re-composition based on adequate process is one potential study to improveleaner’s map quality.

Acknowledgements
We would like to thank all the people who prepared and revised previous versions of thisdocument and Indonesia Endowment Funds for Education (LPDP) for supporting this study.

References
Cañas, A. J., Reiska, P., & Novak, J. D. (2016). Is My Concept Map Large Enough? Communications
in Computer and Information Science, 635, 128–143. https://doi.org/10.1007/978-3-319-455013_10
Chiu, C.-H., & Lin, C.-L. (2012). Sequential Pattern Analysis: Method And Application In Exploring How
Students Develop Concept Maps. TOJET: The Turkish Online Journal of Educational Technology,
11(1).
Dwi Prasetya, D., Hirashima, T., & Hayashi, Y. (2020). Study on Extended Scratch-Build Concept Map
to Enhance Students’ Understanding and Promote Quality of Knowledge Structure. International
Journal of Advanced Computer Science and Applications, 11(4). www.ijacsa.thesai.org
Gerstner, S., & Bogner, F. X. (2009). Concept Map Structure, Gender and Teaching Methods: An
Investigation of Students’ Science Learning. Educational Research, 51(4), 425–438.
https://doi.org/10.1080/00131880903354758
Gregoriades, A., Pampaka, M., & Michail, H. (2009). Assessing students’ learning in MIS using concept
mapping. Journal of Information Systems Education, 20(4), 419.
Hirashima, T., Yamasaki, K., Fukuda, H., & Funaoi, H. (2011). Kit-Build Concept Map for Automatic
Diagnosis. 466–468. https://doi.org/10.1007/978-3-642-21869-9_71
Hirashima, T., Yamasaki, K., Fukuda, H., & Funaoi, H. (2015). Framework of kit-build concept map forautomatic diagnosis and its preliminary use. Research and Practice in Technology Enhanced
Learning, 10(1), 17. https://doi.org/10.1186/s41039-015-0018-9
Hu, M.-L. M., & Wu, M.-H. (2012). The effect of concept mapping on students’ cognitive load. World
Transactions on Engineering and Technology Education, 10(2), 134–137.
Jablokow, K. W., Defranco, J. F., Richmond, S. S., Piovoso, M. J., & Bilén, S. G. (2015). Cognitive Styleand Concept Mapping Performance. Journal of Engineering Education, 104(3), 303–325.
https://doi.org/10.1002/jee.20076

370

Kinchin, I. M., & Alias, M. (2005). Exploiting Variations in Concept Map Morphology as a Lessonplanning Tool for Trainee Teachers in Higher Education. Journal of In-Service Education, 31(3),569–592.
Kinchin, I. M., Hay, D. B., & Adams, A. (2000). How a qualitative approach to concept map analysis canbe used to aid learning by illustrating patterns of conceptual development. Educational Research,
42(1), 43–57.
McAleese, R. (1998). The Knowledge Arena as an Extension to the Concept Map: Reflection in Action.
Interactive Learning Environments, 6(3), 251–272. https://doi.org/10.1076/ilee.6.3.251.3602
McClure, J. R., & Bell, P. E. (1990). Effects of an Environmental Education-Related STS Approach
Instruction on Cognitive Structures of Preservice Science Teachers.
Mcclure, J. R., Sonak, B., & Suen, H. K. (1999). Concept Map Assessment of Classroom Learning:
Reliability, Validity, and Logistical Practicality. In J Res Sci Teach (Vol. 36).
Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality andcontiguity. Journal of Educational Psychology, 91(2), 358–368. https://doi.org/10.1037/00220663.91.2.358
Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. cambridge University press.
Novak, J. D., & Musonda, D. (1991). A Twelve-Year Longitudinal Study of Science Concept Learning.
AmericanEducational
Research
Journal,28(1),
117–153.
https://doi.org/10.3102/00028312028001117
Plotnick, E. (1997). Concept mapping: A Graphical System For Understanding the Relationship
Between Concepts. Syracuse, NY: ERIC Clearinghouse on Information and Technology.
Rautama, E., Sutinen, E., & Tarhio, J. (1997). Supporting learning process with concept map scripts.
Journal of Interactive Learning Research, 8(3), 407.
Rismanto, R., Pinandito, A., Andoko, B. S., Hayashi, Y., & Hirashima, T. (2023). Evaluating the Kit-Build
Concept Mapping Process Using Sub-Map Scoring. Manuscript Submitted for Publication.
Roberts, L. (1999). Using Concept Maps to Measure Statistical Understanding. International Journal of
Mathematical
Educationin
ScienceandTechnology,
30(5),
707–717.
https://doi.org/10.1080/002073999287707
Schneider, S., Krieglstein, F., Beege, M., & Rey, G. D. (2021). How organization highlighting throughsignaling, spatial contiguity and segmenting can influence learning with concept maps. Computers
and Education Open, 2, 100040.
Schroeder, N. L., & Cenkci, A. T. (2018). Spatial Contiguity and Spatial Split-Attention Effects in
Multimedia Learning Environments: a Meta-Analysis. Educational Psychology Review, 30(3),679–701. https://doi.org/10.1007/s10648-018-9435-9
Srivastava, A., Srivastava, N., & Chandrasekharan, S. (2021). Quality of concept maps is affected bymap building strategies. Biochemistry and Molecular Biology Education, 49(1), 129–139.
https://doi.org/https://doi.org/10.1002/bmb.21412
Wong, M., & Lian, S. (1998). An investigation into high-achiever and low-achiever knowledgeorganisation and knowledge processing in concept mapping: A case study. Research in Science
Education, 28(3), 337–352. https://doi.org/10.1007/BF02461567
Wunnasri, W., Pailai, J., Hayashi, Y., & Hirashima, T. (2018). Validity of kit-build method for assessmentof learner-build map by comparing with manual methods. IEICE Transactions on Information and
Systems, E101D(4), 1141–1150. https://doi.org/10.1587/transinf.2017EDP7177
Yamasaki, K., Fukuda, H., Hirashima, T., & Funaoi, H. (2010). Kit-Build Concept Map and Its Preliminary
Evaluation. 18th International Conference on Computers in Education, 290–294.
Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2005). Comparison of two
concept-mapping techniques: Implications for scoring, interpretation, and use. Journal of
Research in Science Teaching: The Official Journal of the National Association for Research in
Science Teaching, 42(2), 166–184.
Yoshida, K., Sugihara, K., Nino, Y., Shida, M., & Hirashima, T. (2013). Practical use of kit-build concept
map system for formative assessment of learners’ comprehension in a lecture. Proc. of ICCE2013,
892–901.
Zheng, L., Li, X., Zhang, X., & Sun, W. (2019). The Effects of Group Metacognitive Scaffolding on Group
Metacognitive Behaviors, Group Performance, and Cognitive Load in Computer-Supported
Collaborative
Learning.
InternetandHigher
Education,42,
13–24.
https://doi.org/10.1016/j.iheduc.2019.03.002

371

" !   !" " "   #" !
#" !"$  #" ! #"                    
 ! !" #   <  ;JC 68 ?E6I:G6>;JC 68 ?E  "9/08,9/70.?@=0>,=0-0.:849249.=0,>4927D;:;@7,=/@0?:") 
0,9B3470?309@8-0=>:1>?@/09?>A40B0=>B3:.,99:?.:9.09?=,?0:9:9/08,9/
70.?@=0>,=0,7>:49.=0,>4927D"90=0,>:91:=?34>4>-:=0/:8@8:=,9/<@0>?4:9>
,=0,//0/?:?3070.?@=0>.09,=4:49:=/0=?:=0>4>?-:=0/:8/@=492A40B492:9/08,9/
70.?@=0A4/0:>'3@>,80?3:/B,>;=:;:>0/?:2090=,?03@8:=B4?3F:605:60>G
F'>@66:84.:==0.?4:9>G,9/<@0>?4:9>1:=48;:=?,9?B:=/>4970.?@=0>.09,=4:>,9/
.=0,?0 70.?@=0 A4/0:> 1:= /08:9>?=,?4:9 -D ?B: ,A,?,=> @=?30=8:=0 ?: A0=41D ?30A,74/4?D:1?30;=:;:>0/80?3:/?B:0C;0=4809?>B0=0.:9/@.?0/B4?3
,9/ ;,=?4.4;,9?>'30=0>@7?>:1?300C;0=4809?>>@220>??3,?49.7@/4923@8:=,9/<@0>?4:9>
49 :9/08,9/ 70.?@=0> /08:9>?=,?0/ -D ?B: ,A,?,=> 3,/ ?30 0110.? :1 4934-4?492
-:=0/:8 @8:=$@0>?4:9>0.?@=0>.09,=4:A,?,=

  
$CG:8:CIN:6GH I=:C::9;DGDC 9:B6C9A:8IJG:H=6H>C8G:6H:99J:ID*1$  6C9DC
9:B6C9 K>9:DH =6K: 6AHD 7::C JI>A>O:9 >C ;A>E HINA: A:8IJG:H 6;I:G *1$   #DL:K:G  I=:
C::9ID9:K>H:6L6NIDEG:E6G:;DGA:8IJG:H=6H6G>H:C7:86JH:HDB:HIJ9:CIH;>C9>I9>;;>8JAI
ID 8DC8:CIG6I: DC A:8IJG:H 6II:C9:9 ;GDB =DB:  88DG9>C< ID *HJ<>     DC 9:B6C9
A:8IJG:H 86C <J6G6CI:: @CDLA:9<: <6>C =DL:K:G  >I >H 9>;;>8JAI ID >BEA:B:CI >C9:E:C9:CI >CI:G68I>K: 6C9>C 9:EI=A:6GC>C< 6C9>IG:B6>CH9>;;>8JAIID8=6C<:I=:L6N6E:GHDCE:G8:>K:H
I=>C<H 6C9 I=>C@H  !JGI=:GBDG:  (6IHJH=>B6    ED>CI:9 DJI I=6I DC 9:B6C9 A:8IJG:H
G:FJ>G:H><C>;>86CII>B:;DGI:68=:GHIDEG:E6G: L=>8=>C9>86I:HI=6II=:7JG9:CDCI=:;68JAIN
B6N>C8G:6H:>CEG:E6G>C<;DGDC 9:B6C9A:8IJG:H $C6EG:K>DJHHIJ9NDCHJ7HI>IJI:A:8IJG:H7NGD7DIH#>NDG>:I6A   6HNHI:B>C
L=>8= 6 GD7DI <6K: A:8IJG:H 6I 6 JC>K:GH>IN DC 7:=6A; D; A:8IJG:GH L6H EGDEDH:9  #DL:K:G 
I=:G:6G:HI>AAHDB:EGD7A:BH;68:97NA:8IJG:H<>K:C7NGD7DIH HJ8=6HB:8=6C>86AC6IJG:
6C97DG:9DB (:6CL=>A: 688DG9>C<ID26I6C67::I6A    >I=6H7::CHJ<<:HI:9I=6I6C
6K6I6GD;I=:A:6GC:GSHDLCA>@>C<B6N7:67A:ID<:C:G6I:>CI:G:HI>CI=:8DCI:CI6C9BDI>K6I:
I=:A:6GC:GIDI6@:I=:C:MI8A6HH $C699>I>DC 6EG:K>DJHG:H:6G8=36B6H=>GD:I6A   DC>C8DGEDG6I>C<=JBDG>CID
:9J86I>DC6AH:II>C<H=6HH=DLCI=6I699>C<ID>A:I=JBDGID&6C?>9G>AAH86C8=6C<:HIJ9:CIHS
<G69:H6C98DC8:CIG6I>DC 88DG9>C<ID36B6H=>GD:I6A HIJ9:CIHL=DJH:9FJ:HI>DC7DD@H
I=6I >C8DGEDG6I:9 =JBDG H=DL:9 6C >C8G:6H:9 E:G8:CI6<: D; 8DGG:8I 6CHL:GH 8DBE6G:9 ID
HIJ9:CIHL=DJH:9IG69>I>DC6AFJ:HI>DC7DD@H $C699>I>DC L=:C8DC8:CIG6I>DCL6HB:6HJG:9
7N;>M:9 ED>CI86B:G6>B6<>C< >IL6H;DJC9I=6IHIJ9:CIHL=DJH:9=JBDG >C;JH:9B6I:G>6AH
8DC8:CIG6I:9DCI=:E6E:G;GDB6C:6GA>:GHI6<: !JGI=:GBDG: .JOJ@>:I6A   EGDEDH:9
68DB>8H8G>EI8DBEDH>I>DCB:I=D9;DGEG:H:CI6I>DCI6A@H $CI=>HEG:K>DJHHIJ9N I=:6JI=DGH
;DJC9 I=6I >C8DGEDG6I>C< 6 8DB>8 9>6AD<J: A>@: H8G>EI HIGJ8IJG: >CID 6 EG:H:CI6I>DC >BEGDK:9
6J9>:C8:6L6G:C:HH :BE6I=N 6C9;6B>A>6G>IN !JGI=:GBDG: I=::;;:8ID;FJ:HI>DCHDCK>:L:GH
>CA:8IJG:K>9:DH=6H7::C6;D8JHD;G:H:6G8= $CDC:EG:K>DJHHIJ9N36H=>D G6@>   
I=:>CH:GI>DCD;FJ:HI>DCH;DGI=:K>:L:G>C6A:8IJG:H8:C6G>DL6HEGDEDH:9ID<:C:G6I:A:8IJG:
K>9:DHI=6I9DCDI7DG:I=:K>:L:G +G:K>DJH HIJ9>:H HJ<<:HI =JBDG 6C9 >CI:G68I>DC 9JG>C< K>:L>C< DC 9:B6C9 A:8IJG:H
B6N7::;;:8I>K:IDG:H>HI7DG:9DB #DL:K:G 699>C<=JBDGDGFJ:HI>DCH6H>CI:G68I>DCIDI=:

372

G:<JA6GA:8IJG:H8G>EIH>C8G:6H:I=:7JG9:CDC;68JAIN>CEG:E6G>C<;DGDC 9:B6C9A:8IJG:H /=>H
E6E:G EGDEDH:H 6 B:I=D9 ID <:C:G6I: =JBDG 6C9 FJ:HI>DCH ;DG >BEDGI6CI LDG9H >C A:8IJG:
H8:C6G>DH6C98G:6I:A:8IJG:K>9:DH;DG9:BDCHIG6I>DC7NILD6K6I6GHID>BEGDK:I=:EGD7A:B
D;P7DG:9DBL>I=DC 9:B6C9A:8IJG:H QA:8IJG:H8:C6G>DL>I==JBDG>H<:C:G6I:97N699>C<PD@: ?D@:HQ 6C9 P/HJ@@DB> 8DGG:8I>DCHQ 67DJI >BEDGI6CI LDG9H ID I=: A:8IJG: H8:C6G>D /=:A:8IJG:H8:C6G>DH;DAADL68DCK:GH6I>DC6A;DGB6I>CL=>8=ILD6K6I6GHEGD8::9L>I==JBDGIDHD;I:CI=:HIG>8I6IBDHE=:G: $C699>I>DC FJ:HI>DCHG:<6G9>C<>BEDGI6CILDG9H6G:699:9ID
I=:H8:C6G>DH;DGBDG:68I>K:A:6GC>C< /=:6K6I6G6H@HHIJ9:CIHFJ:HI>DCH <>K>C<I=:BI>B:
IDI=>C@ I=JH8G:6I>C<6C68I>K:A:6GC>C<:CK>GDCB:CII=6IB6@:HI=:A:8IJG:A:HH7DG>C<  

2: EGDEDH: 6 =JBDG B:I=D9 8DCH>HI>C< D; PD@: Q P/HJ@@DB> Q 6C9 FJ:HI>DCH G:<6G9>C<>BEDGI6CILDG9H>CI=:A:8IJG:H8:C6G>DHI=6I6G:>CEJI7N6A:8IJG:G6H6JH:G /LD6K6I6GH
G:EG:H:CI>C< I:68=:G 6C9 HIJ9:CI 8=6G68I:GH PI:68=:G 6K6I6GQ 6C9 PHIJ9:CI 6K6I6GQ
9:BDCHIG6I:I=:H:H8:C6G>DHHDI=6IHIJ9:CIH9DCDI<:I7DG:9L>I=I=:A:8IJG:K>9:DH /=:
I:68=:G6K6I6GG:69HI=:A:8IJG:H8:C6G>D6ADJ9 /=:HIJ9:CI6K6I6GB6@:H6?D@:6H6PD@: Q
6C9 >BB:9>6I:AN 6;I:G I=: PD@: Q I=: I:68=:G 6K6I6G B6@:H 6 8DGG:8I>DC D; I=: ?D@: 6H 6
P/HJ@@DB> Q $C 699>I>DC  I=: HIJ9:CI 6K6I6G 6H@H I=: HIJ9:CI K>:L:G FJ:HI>DCH G:<6G9>C<>BEDGI6CI LDG9H >C I=: A:8IJG: H8:C6G>D  !JGI=:GBDG:  I=: HIJ9:CI 6K6I6G B6@:H P>OJ8=>
G:68I>DCQ6ADC<L>I=I=:I:68=:G6K6I6GHI6A@IDB6@:I=:>G9>6AD<J:BDG:C6IJG6A /=:A:8IJG:6K6I6GH6G:EA68:96I7DI=:C9HD;I=:A:8IJG:K>9:DH8G::C 6HH=DLC>C!><JG: 1GD>9.IJ9>DL6HJH:9ID<:C:G6I:I=:ILD6K6I6GH L>I=I=:I:68=:GDCI=:G><=IH>9:
6C9HIJ9:CIDCI=:A:;IH>9:H::!><JG:  $C699>I>DC I=:A:8IJG:H8:C6G>D>HG:696ADJ9JH>C<D:!DCI7N6K6I6GHL>I=6C6K6I6GA>E HNC8I=:6K6I6GHBDJI=BDK>C<>CI>B:L>I=I=:KD>8:
JH>C<I:C: $C699>I>DC *E:CGD6986HI:G.D;IL6G:*.L6HJH:9IDDK:GA6N6A:8IJG:HA>9:
6C96K6I6GHID8G:6I:6H>C<A:A:8IJG:K>9:D 6C9HJ7I>IA:HL:G:699:9JH>C<9D7:+G:B>:G:
+GD!><JG: 7:86JH:I=:I:MIG:69>C<JH>C<HNCI=:I>8KD>8:D:!DCI B6N7:9>;;>8JAIID
=:6GI=:I:MI !><JG: ':8IJG:K>9:D7NILD6K6I6GH

   

/:BEA6I:HP#D =D QP.D )6C96 QP)6GJ=D9D Q6C9P#::Q%6E6C:H:P>OJ8=>Q:MEG:HH>C<6C
6<G::B:CIA>@:P$6<G::QL:G:8G:6I:9 6C9DC:D;I=:BL6HG6C9DBANH:A:8I:9IDB6@:
:68=P>OJ8=> Q88DG9>C<IDI=:EG:K>DJHG:H:6G8=&6B>N6:I6A  6;I:G6JM>A>6GN
K:G7HP:HJ QP(6HJ Q:I8 I=:G:>H6 EGD767>A>INI=6IP>OJ8=>Q>H<>K:C>C69>6AD<J: /=JH 6HHJB>C<I=6II=:E6GID;HE::8=6II=::C9D;6H:CI:C8:>C6A:8IJG:H8:C6G>D>H
6ABDHI6AL6NH6C6JM>A>6GNK:G7 6CP>OJ8=>QL6H>CH:GI:96;I:GI=::C9D;I=:H:CI:C8:>C
I=:A:8IJG:6K6I6GHI6A@6I EGD767>A>IN    


373

/=:EGDEDH:9B:I=D96>BHIDB6@:I=:A:8IJG:H8:C6G>D=JBDGDJH76H:9DCI=:%6E6C:H:
8DB:9NHINA:P(6CO6> Q>CL=>8=ILD8DB:9>6CH=6K:6=JBDGDJH8DCK:GH6I>DC $CP(6CO6> Q
DC:HE:6@:GH6NH6PD@:?D@:Q6C9I=:DI=:GHE:6@:G8DGG:8IHI=:?D@:7NP/HJ@@DB> Q
/=JH 6PD@:?D@:Q6C9P/HJ@@DB>QL:G:<:C:G6I:9;DGI=:>BEDGI6CILDG9H>CI=:A:8IJG:
H8:C6G>DH   $%$C6EG:K>DJHHIJ9NDCI=:6JIDB6I>8<:C:G6I>DCD;8DB>89>6AD<J: (6CO6>3DH=>96 #6<>L6G6  EGDEDH:9I=G::INE:HD;PD@:?D@:QP&DID76 HD7> D@:LDG9<6B: Q
P&6?DD@::M8:HH>K:?D@: Q6C9P(J?JCD@:8DC;A>8I>C<?D@: Q$CI=>HHIJ9N L:69DEI:9
P&DID76 HD7> D@:LDG9<6B: QL=>8=86C:6H>AN7:6EEA>:9ID>BEDGI6CILDG9H>C6
A:8IJG: $CP&DID76 HD7> D@:LDG9<6B: QI=:%6E6C:H:I6G<:ILDG9>HGDB6C>O:96C9
DC:8DCHDC6CI>HG:EA68:9L>I=6CDI=:G 6HH=DLC>C!><JG: /=:<:C:G6I:9PD@:QBJHI
7:6G:6ALDG9I=6I>H8DBBDCAN=:6G9>C96>ANA>;: DG:AH:I=:B:6C>C<D;I=:PD@:QL>AA7:
JC@CDLC /=:G:;DG: 6I:MI7DD@KD867JA6GN8DGEJH&ND@6H=DDGEJH"D>=NDL6HJH:9 /=:I:MI7DD@8DGEJH8DCI6>CHLDG9HJH:9>C:A:B:CI6GN ?JC>DG=><= 6C9=><=H8=DDA
I:MI7DD@H 6C9:68=LDG9>HLG>II:C>C%6E6C:H:>C&6C?>6C9-DB6?>6ADC<H>9:>IH;G:FJ:C8N
D;JH: /=:8DCHDC6CIHD;I=:I6G<:ILDG96G:G6C9DBAN8DCK:GI:96C98DBE6G:9L>I=I=:
LDG9H>CI=:I:MI7DD@8DGEJH 6C9I=DH:I=6IB6I8=I=:G:69>C<D;I=:LDG9>CI=:I:MI7DD@
8DGEJHL>I=I=:=><=:HI;G:FJ:C8ND;JH:6G:DJIEJI6HPD@: Q2DG9HHJ8=6HP9:6I=QI=6I
9DCDI:6H>ANA:69IDA6J<=I:GL:G:G:BDK:9;GDBI=:PD@:Q86C9>96I:H 2DG9HI=6I8DCH>HID;BJAI>EA:BDGE=:B:H: < P.=JHD@JI:@>H=>@D=I=>C@>C<D;8DCK:G<:C8:Q6G:HEA>I>CID
I=:>G>C9>K>9J6ABDGE=:B:H7N(:67&J9D:I6A   DC:BDGE=:B:: < P.=JHD@J8DCK:G<:C8:QDI=:GI=6C6HJ;;>M: < PI:@>D;Q7:8DB:HI=:I6G<:I 6C9I=:8DCHDC6CI
D;I=:I6G<:IBDGE=:B:>H8=6C<:9G6C9DBANID8G:6I:I=:PD@:: < P.=JID@JI:@>H=>@D=
I=>C@>C<D;A:6GC>C< Q
!><JG: ":C:G6I>DCB:I=D9D;PD@:Q  $!%H6P/HJ@@DB>Q6<6>CHII=:<:C:G6I:9PD@: QL:69DEI:9I=:P.:IHJB:>/HJ@@DB>
:MEA6C6I>DC8DGG:8I>DCQJH:97N3DH=>966C9#6<>L6G6  /=:P.:IHJB:>/HJ@@DB>Q
EGDK>9:HC:<6I>DCH6C9:MEA6C6I>DCH;DGI=:PD@: Q/D<:C:G6I::MEA6C6I>DCH;DGPD@: QI=:
B:6C>C<D;I=:8DC8:EILDG9JH:96HI=:PD@:Q>HH:6G8=:9>CI=:%6E6C:H:2DG9):I %6E6C:H:2DG9):I>H6%6E6C:H:9>8I>DC6GN>CL=>8=I=:G:A6I>DCH=>EH7:IL::C8DC8:EIH
6C99:;>C>I>DCHD;LDG9H8DGG:HEDC9>C<ID8DC8:EIH6G:G:<>HI:G:9 !DGLDG9HI=6I8DCH>HID;
BJAI>EA:BDGE=:B:H DCANLDG9HI=6I6G:<:C:G6I:96H6PD@:Q;DGI6G<:IBDGE=:B:H6G:
H:6G8=:9 !DG:M6BEA: L=:CP.=JID@JA:6GC>C<Q>H<:C:G6I:96H6PD@:Q;DGP.=JHD@J8DCK:G<:C8:Q>CP.=JHD@JI:@>H=>@D=I=>C@>C<D;8DCK:G<:C8: QP.=JID@JA:6GC>C<Q>H
H:6G8=:9;DG>CI=:%6E6C:H:2DG9):IID<:C:G6I:6C:MEA6C6I>DC 
P.:IHJB:>/HJ@@DB>Q8DCH>HI>C<D;C:<6I>DCH6C9:MEA6C6I>DCH>H<:C:G6I:97N
G6C9DBAN8DB7>C>C<I=G::INE:HD;C:<6I>DCHP2=N6G:NDJ7D@:Q P)D7D@: Q6C9PCDI
H6N>C<6CNI=>C<Q6C9I=G::INE:HD;:MEA6C6I>DCHP7D@:>H:MEA6C6I>DC QP/=6I>H:MEA6C6I>DC Q6C9P/=6I>H:MEA6C6I>DC >HCI>IQ: < P)D.=JID@JA:6GC>C< .=JID@JA:6GC>C<>H68FJ>H>I>DCD;@CDLA:9<:DG67>A>IN Q /=:H:6G:76H:9DCI=:86CC:9H:CI:C8:H
>C3DH=>96:I6A H456C9L:G:BD9>;>:96HP/HJ@@DB>Q6EEGDEG>6I:;DGI=:I:68=:G8=6G68I:G
7:86JH:I=:6K6I6GEA6N>C<I=:GDA:D;6I:68=:GE:G;DGBHI=:P/HJ@@DB> Q374   
/=:FJ:HI>DCH6G:<:C:G6I:976H:9DC6EG:K>DJHHIJ9N36H=>D G6@>   3DH=>D6C9
G6@>:MIG68I:9FJ:HI>DCH=6K:CDJCHI=6IH>B>A6GIDI=:>BEDGI6CILDG9H>CI=:A:8IJG:;GDB
I=:DGEJHD;%6E6C:H:8A6HHGDDB':8IJG:HE::8=DCI:CIH%'/HJ8=>N6:I6A  
6C9<:C:G6I:9FJ:HI>DCH7NG:EA68>C<I=:CDJCH>CI=:FJ:HI>DCHL>I=I=:>BEDGI6CILDG9H>C
I=:A:8IJG:H!><JG: $CHI:69D;%' I=>HHIJ9NJH:9I=:)6<DN60C>K:GH>INDCK:GH6I>DCDGEJH!J?>BJG6:I6A  6C9DGEJHD;%6E6C:H:/:68=:G.E::8=)>=DC<D&NDH=>
#6IHJL6DGEJH L=>8=>H68DGEJHD;IG6CH8G>7:98DCK:GH6I>DCH B6>CAN7N<G69J6I:
HIJ9:CIH I=6I8DJA97:JH:9ID<:C:G6I:BDG:8DBEG:=:CH>7A:FJ:HI>DCH;DGHIJ9:CIH /=:
DGEJHD;%6E6C:H:/:68=:G.E::8=8DCI6>CHFJ:HI>DCH68IJ6AAN6H@:9>CA:8IJG:H 6C9
I=JH >I>HBDG:G:6A>HI>86HFJ:HI>DCHI=6I6EE:6G>CA:8IJG:H ,J:HI>DCHI=6I=6K:H>B>A6GCDJCHIDI=:>BEDGI6CILDG9H>CI=:A:8IJG:L:G:H:A:8I:9>CI=:H:ILD8DGEDG66C9G:EA68:9
I=:CDJCHI=6I6EE:6G>CI=:FJ:HI>DCHL>I=I=:>BEDGI6CILDG9HI=6I6EE:6G>CI=:A:8IJG: !DG:M6BEA: ;DGI=:>BEDGI6CILDG9P.=JID@JI:@>H=>@D=Q 6FJ:HI>DCQ.=JID@JI:@>H=>@D=
HJGJ@DID6G>B6HJ@6DNDJ:K:G9DI=>C@>C<D;8DCK:G<:C8:PL6H<:C:G6I:9 !><JG: ":C:G6I>DCB:I=D9D;FJ:HI>DC  

/LD:ME:G>B:CIHL:G:8DC9J8I:9IDK6A>96I:I=:EGDEDH:9B:I=D9  
/=:EJGEDH:D;I=::K6AJ6I>DC:ME:G>B:CIL6HIDK:G>;NI=::;;:8ID;=JBDG8DCH>HI>C<D;
PD@:Q6C9P/HJ@@DB>Q;DGP8DBEG:=:CH>7>A>INQ6C9P7DG:9DBG:H>HI6C8: Q  IDI6AD; E:DEA: L=DL:G:E6>98ADJ9LDG@:GHG:8GJ>I:97N'6C8:GH E6GI>8>E6I:9>CI=:
:ME:G>B:CI 6H 6 L:7 76H:9 HJGK:N (6A:H   !:B6A:H )D CHL:G  K:G6<:
6<:  .D;6<:  /=G::INE:HD;A:8IJG:K>9:DHL:G:JH:9 6A:8IJG:K>9:D>C
L=>8= 6 I:68=:G 6K6I6G G:69H 6 A:8IJG: H8:C6G>D L>I=DJI 6 HIJ9:CI 6K6I6G /:68=:G   6
A:8IJG:K>9:D>CL=>8=6I:68=:G6K6I6GG:69H6A:8IJG:H8:C6G>D6C96HIJ9:CI6K6I6GDCAN
<>K:H6CP>OJ8=>Q/:68=:G.IJ9:CI 6C96A:8IJG:K>9:D>CL=>8=6I:68=:G6K6I6GG:69H
6 A:8IJG: H8:C6G>D 6C9 P/HJ@@DB> Q 6C9 6 HIJ9:CI 6K6I6G <>K:H 6C P>OJ8=>Q 6C9 PD@:Q
/:68=:G.IJ9:CIL>I=#JBDG $CI=>HHIJ9N I=:HJ7?:8IHL:G:CDIH=DLCI=:DG><>C6AA:8IJG:H
7:86JH:>I>H9>;;>8JAIID<:C:G6AAN8DBE6G:7:IL::CA:8IJG:H<>K:C7N6K6I6GH6C97NI=:68IJ6A
I:68=:GH /=G::INE:HD;:M>HI>C<H8:C6G>DH6C9HA>9:H':8IJG:H  6C9I=6IL:G:68IJ6AAN
JH:9>C68D<C>I>K:H8>:C8:A:8IJG:G:A6I:9ID8G:6I>K>IN6I!JIJG:0C>K:GH>IN#6@D96I:L:G:
JH:9  I I=: :C9 D; I=: FJ:HI>DCC6>G:  6H@:9 ID G6I: I=:>G A:6GC>C< :ME:G>:C8: D; 8D<C>I>K:
H8>:C8:DC6H:K:C ED>CIH86A: /=:B:6CD;I=:A:6GC>C<:ME:G>:C8:D;8D<C>I>K:H8>:C8:L6H
 6C9I=:HI6C96G99:K>6I>DCL6H  /=:A:8IJG:G L=D<:C:G6I:9I=:H8:C6G>DH6C9

375

HA>9:H6C9L6HDC:D;I=:6JI=DGH H:A:8I:9ILD>BEDGI6CILDG9H ;DG:68=H8:C6G>D /D6KD>9
6CDG9:G:;;:8I I=:E6GI>8>E6CIHL:G:9>K>9:9>CIDI=G::<GDJEH"GDJE  "GDJE  
6C9"GDJE  6C9I=:NL6I8=:9I=:K>9:DH>CI=:DG9:GD;':8IJG:H  6C9JC9:G
9>;;:G:CI8DC9>I>DCH9:E:C9>C<DCI=:<GDJE/67A:  /=:A:C<I=D;:68=A:8IJG:K>9:D>H ID  6C9K>:L:GH=69I>B:ID6CHL:GFJ:HI>DCH6;I:G:68=K>9:D /67A:':8IJG:/NE:':8IJG:

1>9:DL6I8=>C<8DC9>I>DCH ME:G>B:CI "GDJE
"GDJE
/:68=:G/:68=:G.IJ9:CI':8IJG:

/:68=:G.IJ9:CI':8IJG:

/:68=:G.IJ9:CI
L>I=#JBDG/:68=:G.IJ9:CI
L>I=#JBDG/:68=:G"GDJE
/:68=:G.IJ9:CI
L>I=#JBDG/:68=:G/:68=:G.IJ9:CI$BB:9>6I:AN6;I:GL6I8=>C<:68=K>9:D I=:NL:G:6H@:9IDG6I:=DL:6HNI=:N;DJC9
I=: K>9:DH ID JC9:GHI6C9 8DBEG:=:CH>7>A>IN 6C9 =DL G:H>HI6CI ID 7DG:9DB I=:N L:G:
7DG:9DB G:H>HI6C8: DC 6  ED>CI H86A: JH>C< "DD<A:!DGB  ;I:G L6I8=>C< 6AA K>9:DH  I=:
E6GI>8>E6CIHL:G:6H@:9IDG:HEDC9ID;DJG 8=D>8:8DC;>GB6I>DC FJ:HI>DCHG:<6G9>C<I=:ILD
>BEDGI6CILDG9H>C:68=K>9:D *C:D;I=:>BEDGI6CILDG9HL6H6I6G<:ILDG9;DGPD@:Q JC9:G
I=:/:68=:G.IJ9:CIL>I=#JBDG8DC9>I>DC ! 
0H>C<- 6 ILD L6N)*1<GDJET8DC9>I>DCL6H8DC9J8I:9ID6HH:HH8DBEG:=:CH>7>A>IN
C>CI:G68I>DCL6HD7H:GK:96II=: A:K:A!    E  6C9BJAI>EA:
8DBE6G>HDCHJH>C<#DABHB:I=D9G:K:6A:9H><C>;>86CI9>;;:G:C8:H!><JG:6 /=:G:HJAIH
H=DLH><C>;>86CIANADL:G 6K:G6<:G6I>C<H;DGI=:K>9:DHA6HIK>:L:9 G:<6G9A:HHD;8DC9>I>DC .>B>A6GAN ILD L6N)*1L6H8DC9J8I:9;DG7DG:9DBG:H>HI6C8: C>CI:G68I>DCL6H
D7H:GK:96I !    E  6C9BJAI>EA:8DBE6G>HDCHJH>C<I=:#DABB:I=D9
G:K:6A:9H><C>;>86CI9>;;:G:C8:H!><JG:7 /=:6K:G6<: G6I>C<L6H=><=:G;DGI=:;>GHI
K>9:DK>:L:9 I=6C;DGI=:A6HI>CI=:/:68=:G8DC9>I>DC6C9I=:6K:G6<:G6I>C<L6H
H><C>;>86CIANADL:G;DGI=:A6HIK>9:DK>:L:9 I=6CI=:DI=:GH>C/:68=:G.IJ9:CI8DC9>I>DC #DL:K:G I=:G:>HCDH>BEA:B6>C:;;:8IL6HD7H:GK:9>CI=:/:68=:G.IJ9:CIL>I=#JBDG8DC9>I>DC6K:G6<:G6I>C<HD;8DBEG:=:CH>7>A>IN7K:G6<:G6I>C<HD; 7DG:9DBG:H>HI6C8:!><JG:  K:G6<:G6I>C<HD;68DBEG:=:CH>7>A>IN 6C977DG:9DBG:H>HI6C8:
E
E /=:CJB7:GH7:ADLI=:76GH>C9>86I:I=:DG9:GD;K>:L>C</D8DBE6G:8DGG:8IG:HEDCH:G6I:H;DGFJ:HI>DCH67DJII=:>BEDGI6CILDG9L>I=
PD@:Q 6C967DJII=:DI=:GLDG9L>I=DJIPD@: Q 6(8):B6GI:HIL6H8DC9J8I:9JH>C<I=:
G:HJAIHD;I=:8DC;>GB6I>DCI:HIJC9:G/:68=:G.IJ9:CIL>I= #JBDG8DC9>I>DC /=:CJB7:GD;

376

E6GI>8>E6CIH;DGI=:G:HJAIHD;I=:FJ:HI>DCH6G:H=DLC>C/67A: /=:G6I:HD;8DGG:8I
G:HEDCH:HIDFJ:HI>DCHG:<6G9>C<>BEDGI6CILDG9HL>I=6C9L>I=DJIPD@:QL:G: 6C9  G:HE:8I>K:AN L=>8=L:G:H><C>;>86CIAN9>;;:G:CI6II=: A:K:A     E   /67A: -:HJAIHD;8DC;>GB6I>DCI:HIG:<6G9>C<>BEDGI6CILDG9HL>I=6C9L>I=DJIPD@:Q
JC9:GI=:/:68=:G.IJ9:CIL>I=#JBDG8DC9>I>DC
2>I=PD@:Q
DGG:8I
$C8DGG:8I
DGG:8I 
2>I=DJIPD@:Q
$C8DGG:8I
 


!/=:8DBEG:=:CH>7>A>ING:HJAIHH=DL=><=:G6K:G6<:G6I>C<H;DGI=:K>9:DHK>:L:9>C:6GA>:G
DG9:G G:<6G9A:HHD;8DC9>I>DC .>B>A6GAN I=:7DG:9DBG:H>HI6C8:G:HJAIH>C9>86I:I=::;;:8ID;K>:L>C<DG9:G /=:
6K:G6<:G6I>C<L6H=><=:G;DGI=:;>GHIK>9:DK>:L:9>C:68=<GDJEG:<6G9A:HHD;I=:
8DC9>I>DC 6C9I=:G6I>C<L6HH><C>;>86CIANADL:G;DGI=:A6HIK>9:DK>:L:9>C:68=<GDJE 
G:<6G9A:HHD;I=:8DC9>I>DC (:6CL=>A: H>BEA:B6>C:;;:8IHL:G:D7H:GK:96BDC<<GDJEH>C
I=:/:68=:G6C9/:68=:G.IJ9:CI8DC9>I>DCH 7JICDI>CI=:/:68=:G.IJ9:CIL>I=#JBDG
8DC9>I>DC /=:H:G:HJAIHHJ<<:HII=6II=:>C8AJH>DCD;=JBDG>C6A:8IJG:K>9:DB6@:H>I
9>;;>8JAIID7:>C;AJ:C8:97NI=:K>:L>C<DG9:G6C9B6NB6@:I=:E6GI>8>E6CIH8DC8:CIG6I:DC
I=:A:8IJG:JCI>AI=::C9 /=:G:HJAIHD;I=:8DC;>GB6I>DCI:HIJC9:G/:68=:G.IJ9:CIL>I=#JBDG8DC9>I>DC
H=DLI=6II=:G6I:D;8DGG:8IG:HEDCH:H;DGI=:>BEDGI6CILDG9HL>I=PD@:QL6HH><C>;>86CIAN=><=:GI=6CL>I=DJIPD@: Q/=:>C8AJH>DCD;PD@:Q;DGI=:>BEDGI6CILDG9HB6N=6K:6C
:;;:8IDCI=:G:8D<C>I>DCD;I=:LDG9H  
/=:EJGEDH:D;I=::K6AJ6I>DC:ME:G>B:CIL6HIDK:G>;NI=::;;:8ID;FJ:HI>DCH;DG
P8DBEG:=:CH>7>A>INQ6C9P7DG:9DBG:H>HI6C8: Q  IDI6AD; E:DEA: L=DL:G:E6>98ADJ9LDG@:GHG:8GJ>I:97N'6C8:GH E6GI>8>E6I:9>CI=:
:ME:G>B:CI6H6L:7 76H:9HJGK:N D;L=>8=E6GI>8>E6CIH6AHD6II:C9:9 ME:G>:CI 6C9
I=:G:B6>C>C<E6GI>8>E6CIHL:G:K6A>9(6A:  !:B6A: )DCHL:G K:G6<:
<:  <:.  /=:HJGK:NL6H8DC9J8I:9;GDB *8I   ID 
*8I   .6B:6H ME:G>B:CI I=:I6G<:IA:8IJG:L6H68D<C>I>K:H8>:C8:8DJGH:DC
8G:6I>K>IN /=:B:6CD;I=:A:6GC>C<:ME:G>:C8:D;8D<C>I>K:H8>:C8:L6H  6C9I=:
HI6C96G99:K>6I>DCL6H  !DJGINE:HD;A:8IJG:K>9:DHL:G:JH:9 6I:68=:G6K6I6G
G:69H6A:8IJG:H8:C6G>DL>I=DJI6HIJ9:CI6K6I6G/:68=:G 6I:68=:G6K6I6GG:69H6
A:8IJG:H8:C6G>D6C96HIJ9:CI6K6I6G6H@H6FJ:HI>DC67DJI6C>BEDGI6CILDG96C9<>K:H6C
P>OJ8=>Q/:68=:G.IJ9:CIL>I=, 6I:68=:G6K6I6GG:69H6A:8IJG:H8:C6G>D6C96
HIJ9:CI6K6I6G6H@HILDFJ:HI>DCH67DJI>BEDGI6CILDG9H6C9<>K:H6CP>OJ8=>Q
/:68=:G.IJ9:CIL>I=,H 6C96I:68=:G6K6I6GG:69H6A:8IJG:H8:C6G>D6C9
P/HJ@@DB> Q6C96HIJ9:CI6K6I6G<>K:H6CP>OJ8=> QPD@: Q6C9FJ:HI>DC/:68=:G.IJ9:CI
L>I=,#JBDG /=G::INE:HD;:M>HI>C<H8:C6G>DH6C9HA>9:HL:G:JH:9':8IJG:H  6C9
 6H>C ME:G>B:CI .6B:6H:ME:G>B:CI I=:HJ7?:8IHL:G:CDIH=DLCI=:DG><>C6A
A:8IJG:H /D6KD>9DG9:G:;;:8I I=:E6GI>8>E6CIHL:G:9>K>9:9>CIDH>M<GDJEH"GDJE "GDJE "GDJE "GDJE "GDJE 6C9"GDJE 6C9I=:NL6I8=:9
I=:K>9:DH>CI=:DG9:GD;':8IJG:  6C9JC9:G9>;;:G:CI8DC9>I>DCH9:E:C9>C<DCI=:
<GDJE/67A: $I>H6HHJB:9I=6IFJ:HI>DCH6G:H:A:8I:97NI=:A:8IJG:GJH:GI=JH :68=

377

FJ:HI>DCJH:9>C ME:G>B:CIL6HH:A:8I:9;GDB <:C:G6I:986C9>96I:FJ:HI>DCH
688DG9>C<IDI=:G:HJAIHD;6E>ADI:ME:G>B:CI>CL=>8=H:K:CJC9:G<G69J6I:6C9<G69J6I:
HIJ9:CIHL:G:6H@:9IDG6I:I=:FJ:HI>DCHDC6 ED>CIH86A: H>C ME:G>B:CI I=:A:C<I=
D;:68=K>9:DHL:G:  ID  6C9K>:L:GH=69I>B:ID6CHL:GI=:FJ:HI>DCH6;I:G:68=
K>9:D':8IJG:/NE:':8IJG:
':8IJG:
':8IJG:
':8IJG:/NE:':8IJG:
':8IJG:
':8IJG:

/67A:  1>9:DL6I8=>C<8DC9>I>DCH ME:G>B:CI"GDJE
"GDJE
"GDJE
/:68=:G/:68=:G/:68=:G.IJ9:CI
L>I=,/:68=:G.IJ9:CI
/:68=:G.IJ9:CI
/:68=:G.IJ9:CI
L>I=,
L>I=,H
L>I=,#JBDG/:68=:G.IJ9:CI
/:68=:G.IJ9:CI
/:68=:G
L>I=,#JBDG
L>I=,#JBDG"GDJE
/:68=:G.IJ9:CI
L>I=,H/:68=:G.IJ9:CI
L>I=,#JBDG/:68=:G"GDJE
/:68=:G.IJ9:CI
L>I=,#JBDG/:68=:G"GDJE
/:68=:G.IJ9:CI
L>I=,#JBDG/:68=:G/:68=:G.IJ9:CI
/:68=:G.IJ9:CI
L>I=,
L>I=,H
P,Q >H DC:FJ:HI>DC699:9IDI=:H8:C6G>D 6C9P,HQ >H ILDFJ:HI>DCH 699:9 ! 
0H>C<HI6I>HI>86AHD;IL6G:- 6 ILD L6N)*1<GDJET8DC9>I>DCL6H8DC9J8I:9;DG
8DBEG:=:CH>7>A>INJH>C<<GDJEHL>I=I=:/:68=:G.IJ9:CI,8DC9>I>DC"GDJEH  6C9 C>CI:G68I>DCL6HD7H:GK:96II=: A:K:A!    E  6C9BJAI>EA:
8DBE6G>HDCHJH>C<#DABSHB:I=D9G:K:6A:9H><C>;>86CI9>;;:G:C8:H!><JG:6 $C699>I>DC 
ILD L6N)*1L6HE:G;DGB:9JH>C<<GDJEHJC9:GI=:/:68=:G.IJ9:CI,H8DC9>I>DC"GDJEH  6C9 C>CI:G68I>DCL6HD7H:GK:96II=:A:K:A!  E  
6C9BJAI>EA:8DBE6G>HDCHG:K:6A:9H><C>;>86CI9>;;:G:C8:H!><JG:7 DI=D;I=:G:HJAIH
H=DLH=DLH><C>;>86CIANADL:G 6K:G6<:G6I>C<H;DGI=:K>9:DHA6HIK>:L:9 >C/:68=:G8DC9>I>DC6"GDJEHL>I=/:68=:G.IJ9:CI,7"GDJEHL>I=/:68=:G.IJ9:CI,H!><JG:  K:G6<:G6I>C<HD;8DBEG:=:CH>7>A>IND;6<GDJEHL>I=/:68=:G.IJ9:CI,
8DC9>I>DC 6C97<GDJEHL>I=/:68=:G.IJ9:CI,HE
E  /=:CJB7:GH7:ADLI=:76GH>C9>86I:I=:DG9:GD;K>:L>C</LD L6N)*1<GDJET8DC9>I>DCL6H8DC9J8I:9;DG7DG:9DBG:H>HI6C8:JH>C<<GDJEHL>I=I=:/:68=:G.IJ9:CI,8DC9>I>DC"GDJE  6C9 C>CI:G68I>DCL6H
D7H:GK:96II=: A:K:A!   E  6C9BJAI>EA:8DBE6G>HDCHJH>C<#DABSH

378

B:I=D9G:K:6A:9H><C>;>86CI9>;;:G:C8:H6II=:A:K:A!><JG:6 $C699>I>DC ILD L6N)*1L6HE:G;DGB:9JH>C<<GDJEHJC9:GI=:/:68=:G.IJ9:CI,H8DC9>I>DC"GDJE  6C9 C>CI:G68I>DCL6HD7H:GK:96II=: A:K:A!  E  6C9BJAI>EA:
8DBE6G>HDCHG:K:6A:9H><C>;>86CI9>;;:G:C8:H!><JG:7 DI=D;I=:G:HJAIHH=DLH=DL
H><C>;>86CIANADL:G 6K:G6<:G6I>C<H;DGI=:K>9:DHA6HIK>:L:9 >C/:68=:G8DC9>I>DC
6"GDJEHL>I=/:68=:G.IJ9:CI,7"GDJEHL>I=/:68=:G.IJ9:CI,H!><JG:  K:G6<:G6I>C<HD;7DG:9DBG:HDC6C8: D;6<GDJEHL>I=/:68=:G.IJ9:CI,
8DC9>I>DC 6C97<GDJEHL>I=/:68=:G.IJ9:CI,HE
E  /=:CJB7:GH7:ADLI=:76GH>C9>86I:I=:DG9:GD;K>:L>C</D8DBE6G:8DGG:8IG:HEDCH:G6I:H;DGFJ:HI>DCH67DJII=:>BEDGI6CILDG9HL>I=6C9
L>I=DJI6FJ:HI>DC (8):B6GSHI:HIL6H8DC9J8I:9JH>C<I=:G:HJAIHD;I=:8DC;>GB6I>DCI:HI
JC9:GI=:/:68=:G.IJ9:CIL>I=,8DC9>I>DC"GDJEH  6C9 /=:CJB7:GD;
E6GI>8>E6CIH;DGI=:G:HJAIHD;I=:FJ:HI>DCH>HH=DLC>C/67A: /=:G6I:HD;8DGG:8I
G:HEDCH:HIDI=:FJ:HI>DCH67DJI>BEDGI6CILDG9HL>I=6C9L>I=DJIFJ:HI>DCL:G: 6C9 G:HE:8I>K:AN L=>8=L:G:H><C>;>86CIAN9>;;:G:CI6II=: A:K:A      E  /67A:  -:HJAIHD;8DC;>GB6I>DCI:HI67DJI>BEDGI6CILDG9HL>I=6C9L>I=DJI6FJ:HI>DC
JC9:G/:68=:G.IJ9:CIL>I=,8DC9>I>DC
2>I=FJ:HI>DCDGG:8I$C8DGG:8I
DGG:8I

2>I=DJIFJ:HI>DC$C8DGG:8I

/D8DBE6G:8DGG:8IG:HEDCH:G6I:H;DGFJ:HI>DCH67DJII=:>BEDGI6CILDG9HL>I=6
FJ:HI>DC6C9L>I=6RSD@: Q 6  I:HIL6H8DC9J8I:9JH>C<G:HJAIHD;I=:8DC;>GB6I>DCI:HI
JC9:GI=:/:68=:G.IJ9:CIL>I=#JBDG8DC9>I>DC"GDJEH  6C96C9/:68=:G.IJ9:CI
L>I=,8DC9>I>DC"GDJEH  6C9 )DH><C>;>86CI9>;;:G:C8:HL:G:D7H:GK:96II=:  
A:K:A
!JGI=:GBDG: ID8DBE6G:8DGG:8IG:HEDCH:G6I:H;DGFJ:HI>DCH67DJII=:>BEDGI6CI
LDG9HL>I=6FJ:HI>DC6C9L>I=6RSD@:Q >CI=:H6B:K>9:D (8):B6GSHI:HIL6H8DC9J8I:9
JH>C<G:HJAIHD;I=:8DC;>GB6I>DCI:HIJC9:GI=:/:68=:G.IJ9:CIL>I=,#JBDG 8DC9>I>DC /=:CJB7:GD;E6GI>8>E6CIH;DGI=:G:HJAIHD;I=:FJ:HI>DCH>HH=DLC>C/67A: /=:G6I:HD;
8DGG:8IG:HEDCH:HIDI=:FJ:HI>DCH67DJI>BEDGI6CI LDG9HL>I=6FJ:HI>DC6C9L>I=6RSD@:Q
L:G: 6C9  G:HE:8I>K:AN /=:G:L6H6H><C>;>86CI9>;;:G:C8:6II=: A:K:A     E  /D8DBE6G::;;:8IHD;FJ:HI>DCH6C9RSD@:Q L=:CI=:DI=:GFJ:HI>DCL6H>C8AJ9:9>C

I=:H6B:A:8IJG:K>9:D 6 I:HIL6H8DC9J8I:9JH>C<G:HJAIHD;I=:8DC;>GB6I>DCI:HIJC9:G
I=:/:68=:G.IJ9:CIL>I=,#JBDG8DC9>I>DC6C9/:68=:G.IJ9:CI L>I=,H8DC9>I>DC"GDJEH  6C9 /=:G6I:HD;8DGG:8IG:HEDCH:HIDI=:FJ:HI>DCH67DJI>BEDGI6CILDG9H
L>I=6FJ:HI>DC6C9L>I=6RSD@:Q L:G: 6C9  G:HE:8I>K:AN/67A: 6C9I=:G:>H
H><C>;>86CIIG:C97:IL::CI=:B     E 

379

/67A: -:HJAIHD;8DC;>GB6I>DCI:HI;DG>BEDGI6CILDG9HL>I=6FJ:HI>DC6C9L>I=6RSD@:Q>C
I=:H6B:A:8IJG:K>9:DJC9:G/:68=:G.IJ9:CIL>I=,#JBDG8DC9>I>DC
2>I=PD@:Q
DGG:8I
$C8DGG:8I
DGG:8I
2>I=FJ:HI>DC$C8DGG:8I
 


/67A: -:HJAIHD;8DC;>GB6I>DCI:HI;DG>BEDGI6CILDG9HL>I=6FJ:HI>DC6C9L>I=6RSD@:Q
L=:CI=:DI=:GFJ:HI>DCL6H>C8AJ9:9>CI=:H6B:A:8IJG:K>9:DJC9:G/:68=:G.IJ9:CIL>I=
,#JBDG8DC9>I>DC6C9/:68=:G.IJ9:CIL>I=,H8DC9>I>DCCHL:GDGG:8I
$C8DGG:8I
,J:HI>DC
PD@:Q!/=:8DBEG:=:CH>7>A>ING:HJAIHH=DL=><=:G6K:G6<:G6I>C<H;DGI=:K>9:DHK>:L:9;>GHI6C9
ADL:G6K:G6<:G6I>C<H;DGI=DH:K>:L:9A6HI>CI=:K>:L>C<DG9:G G:<6G9A:HHD;I=:8DC9>I>DC 
H>BEA:B6>C:;;:8I7:IL::C<GDJEH>C:68=8DC9>I>DCL6HD7H:GK:9JC9:GI=:/:68=:G6C9
/:68=:G.IJ9:CIL>I=,#JBDG8DC9>I>DCH7JICDIJC9:GI=:/:68=:G.IJ9:CIL>I=,DG
/:68=:G.IJ9:CIL>I=,H8DC9>I>DCH .>B>A6GAN I=:G:HJAIH;DG7DG:9DBG:H>HI6C8:>C9>86I:6C>C;AJ:C8:D;K>:L>C<DG9:G #DL:K:G I=:G:HJAIHD;"GDJE/:68=:G.IJ9:CIL>I=,#JBDG /:68=:G 6C9
/:68=:G.IJ9:CIL>I=, >CI=6IDG9:G9DCDIH=DL6CDG9:G:;;:8I !JGI=:GBDG: 6H>BEA:
B6>C:;;:8I7:IL::C<GDJEH>C:68=8DC9>I>DCL6HD7H:GK:9JC9:GI=:/:68=:G8DC9>I>DC7JI
CDIJC9:GI=:DI=:G8DC9>I>DCH /=>HHJ<<:HIHI=6II=:>C8AJH>DCD;6FJ:HI>DCI:MI9>G:8I:9IDI=:E6GI>8>E6CIH
G:<6G9>C<>BEDGI6CILDG9H>CI=:A:8IJG:K>9:DH8:C6G>DB6NB6@:>I9>;;>8JAIID7:>C;AJ:C8:9
7NI=:DG9:G>CL=>8=I=:NL6I8=I=:K>9:D6C9B6N6AADLI=:E6GI>8>E6CIHID8DC8:CIG6I:DC
I=:A:8IJG:JCI>AI=::C9 /=:G:HJAIHD;I=:8DC;>GB6I>DCI:HI;DGI=:>BEDGI6CILDG9HL>I=6C9L>I=DJI6
FJ:HI>DCH=DLI=6II=:G6I:D;8DGG:8IG:HEDCH:H;DGI=:>BEDGI6CILDG9HL>I=FJ:HI>DCHL6H
H><C>;>86CIAN=><=:GI=6CL>I=DJIFJ:HI>DCH /=:>C8AJH>DCD;FJ:HI>DCH;DGI=:>BEDGI6CI
LDG9HB6N6;;:8ILDG9G:8D<C>I>DC !JGI=:GBDG: I=:G:HJAIHD;8DBE6G>HDC7:IL::C8DGG:8IG:HEDCH:G6I:H;DGFJ:HI>DCH
67DJII=:>BEDGI6CILDG9HL>I=6FJ:HI>DC6C9L>I=6D@:Q>CI=:H6B:K>9:DH=DLI=6II=:
G6I:D;8DGG:8IG:HEDCH:H;DGI=:>BEDGI6CILDG9HL>I=FJ:HI>DCL6HH><C>;>86CIAN=><=:GI=6C
L>I=D@: Q/=>HHJ<<:HIHI=6II=:>C8AJH>DCD;FJ:HI>DCH;DGI=:>BEDGI6CILDG9HB6N=6K:
BDG:D;6C:;;:8I;DGI=:G:8D<C>I>DCD;LDG9HI=6CI=:>C8AJH>DCD;6D@: Q#DL:K:G I=:
G:HJAIHD;8DBE6G>HDC7:IL::C:;;:8IHD;FJ:HI>DCH6C9D@:QL=:CI=:DI=:GFJ:HI>DCL6H
>C8AJ9:9>CI=:H6B:A:8IJG:K>9:DH=DLI=6I6D@:Q>HBDG::;;:8I>K:I=6C6FJ:HI>DC  

/=>HE6E:GEGDEDH:96B:I=D9ID<:C:G6I:=JBDG8DCH>HI>C<D;D@: Q/HJ@@DB> Q6C9
FJ:HI>DCH;DG>BEDGI6CILDG9H>CA:8IJG:H8:C6G>DH 9:BDCHIG6I:97NILD6K6I6GH ID>BEGDK:
I=:EGD7A:BD;P7DG:9DBL>I=DC 9:B6C9A:8IJG:H Q/LD:ME:G>B:CIHL:G:8DC9J8I:9ID
K:G>;NI=:K6A>9>IND;I=:EGDEDH:9B:I=D9 $IL6H;DJC9I=6II=:699>I>DCD;:>I=:G6PD@:QDGFJ:HI>DCB69:I=:E6GI>8>E6CIHA:HH7DG:96C96AHDH=DL:9I=6II=:>C8AJH>DCD;6D@:Q
6C9FJ:HI>DC;DG>BEDGI6CILDG9HB6N=6K:6C:;;:8IDCI=:G:8D<C>I>DCD;LDG9H #DL:K:G 
H>C8:I=:8DCI:CID;I=:K>9:DHJH:9>CI=::ME:G>B:CIHL6HDCAN8D<C>I>K:H8>:C8: >I>H
C:8:HH6GNID:M6B>C:L=:I=:GI=:EGDEDH:9B:I=D9>H6AHD:;;:8I>K:;DGA:6GC>C<DI=:G
8DCI:CI : < B6I=:B6I>8H !JGI=:GBDG: H>C8:I=:A:8IJG:K>9:D>CI=>H:ME:G>B:CIA6HI:9;DG

380

H=DGI:GI=6CILDB>CJI:H >I>HJC8A:6GL=:I=:G>IL>AA7:6CDC 9:B6C9A:8IJG:I=6I86C7:
EG68I>86AAN $I>HC:8:HH6GNID8DC9J8I:ME:G>B:CIHJH>C<A:8IJG:K>9:DHI=6I6G:H>B>A6G>CA:C<I=IDI=:DC 9:B6C9A:8IJG:HI=6I6G:68IJ6AANJH:9 $C699>I>DC I=::K6AJ6I>DCD;
7DG:9DBG:H>HI6C8:L6HHJ7?:8I>K: 6C96CD7?:8I>K::K6AJ6I>DC>HCDIN:IEDHH>7A: $I>H
C:8:HH6GNIDK:G>;NI=:JH:D;E=NH>DAD<>86A>C9>8:H HJ8=6HI=:CJB7:GD;7A>C@H $C699>I>DC 7:86JH:>C8AJ9>C<B6CNH:IHD;D@:Q6C9/HJ@@DB>Q>C8G:6H:HI=:I>B:D;I=:
A:8IJG:K>9:D6C9>H8DCH>9:G:9>BEG68I>86A L:9>9CDIH:I69DJ7A:=JBDG8DC9>I>DC>CL=>8=
ILDH:IHD;D@:Q6C9/HJ@@DB>QL:G:>C8AJ9:9>CI=::ME:G>B:CIH #:C8: >I>HC:8:HH6GN
ID8DC9J8I699>I>DC6A:ME:G>B:CIHL>I=69DJ7A:=JBDG8DC9>I>DCID8DBE6G:7:IL::C:;;:8IH
D;FJ:HI>DC6C9PD@:QL=:C6CDI=:GPD@:Q>H>C8AJ9:9>CI=:H6B:A:8IJG:K>9:D  ">@24!
&?,?@><@::169:B70/20,9/@9/0=>?,9/492:1.:77020>?@/09?>=02,=/492>;0.4,7
>@;;:=?0/@.,?4:9 :8;,=4>:9:149;0=>:9,9/:9/08,9/.7,>>0>:@=9,7:1?30,.@7?D:1/@.,?4:9"!(!)%&'+   ,?>@>348,+#=,.?4.,7=0;:=?=02,=/492?:,9:9/08,9/.7,>>B4?3::2707,>>=::8
&09>3@(94A0=>4?D:3:,2,6@096D@5:&3:3:@  
4D:=4=,64,>02,B,+:>34:&#0=1:=8,9.00A,7@,?4:9:1,@?:8,?4.20>?@=0
2090=,?4:9>D>?08@>492-4/4=0.?4:9,7&' :93@8,9:4/=:-:?'30 9/99@,7:910=09.0
:1?30,;,90>0&:.40?D1:==?414.4,79?0774209.0
*,?,9,-0!,6,2,B,9/: +@6,B,')0=41D492?300110.?>:1.3,92492?30
,;;0,=,9.0:1?3070.?@=0?:,1,A:=4?0.3,=,.?0=4970.?@=0,;,90>0&:.40?D1:=491:=8,?4:9,9/
&D>?08>49/@.,?4:9 +,8,>34=:+,24>34?,43,=,+,8,8:?:&602,D,+@8:=:@>>09?09.0>
093,9.0808:=D,9/49/@.0-0?,,9/2,88,;:B0=49?303@8,91=:9?,7.:=?0C:@=9,7:1
!0@=:90?):7
;;  &@E@64@9,:4@-@?,+,?:
93,9.492/4,7:24..:88@94.,?4:9B4?3,@/409.0
49;7,99492:1;=0>09?,?4:9?,76?3=:@23F ,9E,4G>.=4;?80?3:/'30 ?399@,7 00?492:1
,;,90>0:294?4A0&.409.0&:.40?D;; +,>34:&=,64
090=,?4:9:1<@0>?4:9>09?09.0>@>492.:9A0=>,?4:9.:=;@>1:=
,@?:8,?4.2090=,?4:9:170.?@=0>.09,=4::970.?@=0>@->?4?@?4:9>D>?08'30 =/99@,7:910=09.0:1?30,;,90>0&:.40?D1:==?414.4,79?0774209.0
)=:4/&?@/4:3??;>A=:4/.:8>?@/4:,>?)40B
 
:0:9?3??;>:0:9?.7:@/,>?)40B
 
?090&?@/4:3??;> ?090.:8>?@/4:,>?)40B
 
";09=:,/.,>?0=&:1?B,=03??;>:->;=:50.?.:8,>?)40B
 
,84D,+"39:' ,?>@-,=,&,>34:6,
:.3:?064?,4B,>D>?0894:60=@4E@.34
>:@9D@@?48492'30>>:.4,?4:91:=!,?@=,7,92@,20#=:.0>>492 ?3;;  +:>34/,+,24B,=, 
9,@?:8,?4.8,9E,4/4,7:2@0.=0,?492>D>?089?0=9,?4:9,7
:@=9,7:1110.?4A0924900=492"97490&&!):7 !: &;0.4,7>>@0;;   
D:6,>3::=;@>:43D:3??;>.7=/9495,7,.5;-..B51=0<74>?3?87,>?)40B
 
@/:'+,8,8:?: ,?>@8:?:+;;7D492.:9/4?4:9,7=,9/:81407/>?:,;,90>0
8:=;3:7:24.,7,9,7D>4>#=:.00/492>:1?30
:910=09.0:98;4=4.,7 0?3:/>49!,?@=,7
,92@,20#=:.0>>492 !#;; '>@.34D, :2@=0&!4>34E,64"3?,!,6,2,B,&0A07:;492.:=;@>:1
,;,90>0.7,>>=::870.?@=0>;00.3.:9?09?>#=:.:1?30&4C?39?0=9,?4:9,7:910=09.0:9
,92@,20%0>:@=.0>,9/A,7@,?4:9% ;;   
@548@=,34-,&"3>: 0C4.,7,9/2=,88,?4.,710,?@=0>:1>;:609,9/B=4??09
,;,90>049.:9?=,>?C;7:=492,70C4.,7;=:147492,;;=:,.3?:.:8;,=492>;:609,9/B=4??09.:=;:=,#=:.:1?30)?3&#9?0=9,?4:9,7:910=09.0&;00.3,9/:=;:=,   ',?0-0@54?,+!43:92:D:>34,?>@B,:=;@>3??;BBB.:=;@>1?.:8?0=8>
,>?)40B
 
,9.0=>3??;>BBB7,9.0=>5;8D;,20,>?)40B
 

381

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Adapting Noticing Framework to Analyze
Learner's Reasoning in VR-simulated
complex scenarios
Herold PCa*, Chandan DASGUPTAb
Educational Technology IDP, IIT Bombay, Indiab
Educational Technology IDP, IIT Bombay, India*herold@iitb.ac.inaAbstract: Car drivers learning to make decisions in Indian on-road scenarios ischallenging. We designed and simulated a complex on-road scenario using virtual
reality technology to help learners experience alternate perspectives of their driving
actions. Our research focused on the cognitive influences on decision-making andadapted the noticing framework as the analytical lens. Four participants drove a car on
virtual Indian city roads accompanied by a virtual conversational agent to elicit theirreasoning while making decisions, followed by a stimulated recall protocol. We
recorded each participant's actions and timestamped conversations with the agent andresearcher. We inductively coded the noticing episodes and interpreted six types ofconceptual relationships between the designed elements in the VR scenario. These
interpreted codes could help us analyze the changes in a learner’s noticing patterns
after taking alternate perspectives. This paper highlights the usefulness of the adapted
noticing framework for future studies with multiple complex VR scenarios.
Keywords: VR simulation, Noticing framework, On-road scenarios, Learner's
reasoning analysis

1. Introduction
A decision made by a human in complex scenarios requiring a time-bound response involves
reasoning processes as characterized by naturalistic decision-making models (G. A. Klein,1993). A vital component of that process involves the learner noticing and assessing thecritical cues within the complex situation. In our research, we contextualized a complex
scenario based on the frequently occurring events on Indian public city roads, simulated in a
Virtual Reality environment. Existing work on VR-based decision-making and situationassessment has majorly modeled extreme accident situations as complex scenarios(Benvegnù et al., 2021) rather than frequently occurring events. VR interventions withaffordance to take alternate perspective-views of an event (Kang et al., 2012) for decisionmaking have used ethical/moral frameworks (Sütfeld et al., 2017) to analyze their actions.
These analytical lenses include both cognitive and affective influences on decision-making.
Since capturing the affective responses of learners while driving is challenging, we focusedour research on cognitive analytical methods. This scoping led us to the noticing framework in
the education research domain. The existing noticing frameworks are predominantly derived
in the teacher education and learning sciences field within a classroom setup (Amador et al.,2021). In this paper, we chose the noticing framework (Lobato et al., 2012) from learningsciences due to the possibility of mapping the conceptual element in our VR scenarios withthe adapted components of the framework. In the VR intervention we developed,"PERSPECS," the learners/participants drive a VR-simulated car on Indian public city roads,
making quick decisions at critical junctures, followed by taking an alternate perspective viewof their actions. Our broader research objective is to investigate the effects of virtual reality
perspective-taking on the learner's noticing behaviors in complex scenarios while driving a car
on Indian roads. But, in this paper, we investigated a scoped-down research question: "How382

does the adapted noticing framework help capture the learners' assessment of critical cues
within the complex VR scenarios?".

2. Noticing framework
Research on Noticing as a construct is derived from Goodwin's work on professional vision(Goodwin, 1994). For our study, we are considering the definition from the learning sciencesgroup (Lobato et al., 2012), which says - Noticing refers to selecting, interpreting, and working
with particular features or regularities when multiple sources of information compete forstudents' attention. While assessing a complex situation on Indian roads, noticing the criticalcues from various sources of information across the sequence of events is essential for
decision-making (G. Klein, 2008). Lobato's noticing framework maps well with our context,
which consists of the following components: (i) Conceptual Relationships, (ii) Perceptual
Features, (iii) Focusing interactions, (iv) Task features, and (v) The Nature of the Activity. We
adapted the framework by modifying the domain-specific definitions for each component
detailed in the following sub-section.

2.1 Framework adaptation
In our context, a 'Conceptual Relationship' is a relationship between different elements in acomplex scenario that can be abstracted to signify a concept. The 'Perceptual Features' in ourcontext help drivers orient toward the conceptual elements in the situation and notice theconceptual relationships between them. 'Focusing Interactions' refers to the virtual
conversational agent (co-passenger) interactions with the participants, detailed in the next
section, which can emphasize the various conceptual relationships. The 'Task Features' are
the multiple on-road complex scenarios designed. The 'Nature of Activity' refers to theparticipatory roles of the driver and other actors in the VR environment, which regulates their
possible actions. Particularly, the 'Conceptual Relationships" forms the basis of analyzing thenoticing episodes coded from the participant's conversations with the agent, detailed in thediscussion section.

3. Design of VR intervention
The noticing framework and the user study interviews conducted by us with ten car drivers
commuting in Indian cities guide the design of the VR-simulated complex scenarios. Theinterview method was semi-structured, with questions to recall and reflect on their driving
experience, for example – “Can you share any instance where you had to break a traffic rule
but felt your decision was right?” The VR-simulated car also includes the design of aconversational agent guided by the scaffolding framework of pedagogical agents (Azevedo etal., 2004). We developed the scenario with the "Unity-3D" software. The scenario is
experienced in VR using an "HTC-VIVE" head-mounted display and controlled using aphysical steering wheel and brake/accelerator, as shown in Figure 1.

3.1 VR-simulated Complex scenario
A complex scenario is a scoped version of complex problems defined by (Funke, 2010),
characterized by Complexity, Connectivity, Dynamics, and In-transparency in his work on
cognition. In our research, we scope our scenarios that imbibe the characteristics of complex
problems. In such complex scenarios, the decision-maker needs to notice and assess many
elements involved in events that are changing dynamically and respond within 3-4 seconds.
This response action results in immediate consequences on fellow human beings in that publicscenario. The elements of each complex scenario are primarily characterized into the following
four types – (i) Action Target - The action target of a scenario is humans who could bear theconsequences of a participant's driving actions, (ii) Action Rule - The action rules of a scenario
communicate the road rules to be followed by the driver in that context. (iii) Action Persuader

383

- The vehicles in the participant's vicinity which persuade or pressure the participants to makea quick decision, and (iv) Action Choice Primer - The vehicles in the participant's vicinity that
could prime them on the available choices to act in that context. We derived thesecharacteristics for each design element in the VR scenario from the affinity-map inferencing
done on user study interviews.

Figure 1. Participant driving through the VR scenario (left) with the agent (middle) inthe VR study setup (right).
A brief to the "Old man scenario" shown in Figure 1 – "You are on the way back to yourhome in your car to attend a dinner you had planned, and you have stopped in front of a traffic
signal as the signal is RED (Action Rule). A few pedestrians and an old man (Action Target)
are crossing the road. As the old man is halfway across the road nearing your car, the signal
turns GREEN, the vehicles by your side start moving (Action Choice Primer), and vehicles
behind you start honking (Action Persuader)." In this public scenario, the complexity is inassessing the situation with multiple events unfolding. The response time to decide and act inthis complex scenario is around 3-4 secs. We validated this scenario using a survey
questionnaire with 30 Indian drivers, who rated multiple Indian on-road scenarios based on
the following factors - familiarity with them and level of difficulty in deciding an action.

3.2 Conversational agent
The Wizard-of-Oz method (Dahlbäck et al., 1993) is adopted to deliver the appropriate
dialogues and animation gestures of the co-passenger to the participant, where a researcher
will use the desktop keyboard to trigger them, as shown in Figure 1. This method allows theresearcher to trigger multiple dialogues like - “Hey, back there, how did you decide what to
do?” based on the response from the participant.

4. Method
The primary research question we are analyzing in this study is – "How does the adapted
noticing framework help capture the learners' assessment of critical cues within the complex
VR scenarios?". The participant's conversation with the conversational agent during the VRintervention and the participant's conversation with the researcher during the stimulated recall(Lyle, 2003) are coded as per the adapted noticing framework to interpret patterns for the
primary research question. We used qualitative analysis with an inductive coding method(Cohen et al., 2017) to derive inferences.

4.1 Participants and procedure
The participants for the study were adults in the age range of 21 - 29 years with a minimum
car driving experience of one year on Indian roads. Four participants volunteered for the study
and signed the necessary consent forms approved by the institutional review board on
campus. None of the participants had prior experience in using a VR car simulation.
Participants take 3-level training inside the PERSPECS VR environment for 15 minutes to get
accustomed to the car controls, gain familiarity with the city buildings and roads, and getfamiliar with the co-passenger avatar and self-avatar within the VR environment. After training,384

the participant experiences the complex scenario, i.e., the "old man scenario," as per thefollowing sequence of steps for another 15 minutes. Firstly, as detailed in the design section,
the participant drives the car, encounters the traffic lights with the old man and otherpedestrians crossing the road, and acts at the critical decision-making point. After the action,
the co-passenger has a conversation right after the critical decision-making point. Following
the conversation, the participant experiences their action replay from the old man's
perspective inside the VR environment (taking a perspective view is part of the broader
research objective and not detailed in this paper), followed by a conversation with the copassenger again. Ending the VR experience, the participant removes the VR HMD. The
researcher then replays the screen recording on the desktop and conducts a stimulated recall
protocol to reiterate their reasoning at the critical points in the scenario for 7-10 minutes.

4.2 Data sources and analysis
The data sources include (i) The video recording of the participant experiencing the VRintervention, (ii) The audio recording of the participant conversing with the co-passenger andthe researcher, and (iii) The screen recording of the entire VR action-play by the participant.
We used a qualitative analysis with an inductive coding method to derive inferences for our
research question- "How does the adapted noticing framework help capture the learners'
assessment of critical cues within the complex VR scenarios?". The transcripts between theco-passenger, participant, and researcher are encoded where the participant notices aconceptual relationship (CR) between the elements within the VR scenario. For instance, theconversation between the co-passenger and the participant in Table 1 is a noticing episode,which we then coded inductively into their various types.
Table 1. A conversation excerpt between the co-passenger and a participant
Speaker
Co-passenger
Participant-3

Dialogue
What all thoughts were there in your mind before acting backthere?
I thought he is old; he was quite slow, so I thought I should give
him time to cross the road.

In this conversation, the participant noticed the conceptual relationship of walking pacebetween the old man and the other pedestrians crossing the road. Noticing a conceptual
relationship becomes one of the reasons for the participant's decision to give time for the old
man to cross the road. Hence, similar conversations are "noticing" episodes that form theanalysis unit. Following this, we inductively interpreted the types of noticing episodes andcategorized them as shown in Table 2. We refined our categories through repeateddiscussions, where we examined their descriptions.

5. Results and Discussion
We encoded the conversations between the four participants, co-passenger, and researcher
resulting in 29 noticing episodes. Out of these, ten noticing episodes were after the criticaldecision point, at an average of 2.5 episodes per participant, and 19 noticing episodes wereafter perspective-taking, amounting to an average of 4.75 episodes per participant. We
interpreted these noticing episodes and categorized them into six types based on theconceptual relationships noticed between the designed elements of the VR scenario. The
following subsection details the categorized noticing episodes with examples of suchreasonings spoken out by the four participants.

5.1 Noticing episodes with conceptual relationship
The six types of noticing episodes were interpreted and categorized based on the conceptual
relationships between the designed elements in the VR scenario are summarized in Table 2.

385

The top two types of noticing episodes observed across the four participants are "You Driving(YD) + Action Rule (AR)" with nine episodes and "Action Persuader (AP) + Action Rule (AR)"with six episodes. The following dialogue is an instance of the noticing episode interpreted as
the type "Action Persuader + Action Rule" from the reasoning conversations of participant 4 –"Ya, like this, even in the real world it happens, and I usually stop my vehicle, and if the vehicle
behind don't notice I sometimes roll the window and signal using my hand that 'wait' so theyusually understand. or sometimes I signal from the rearview mirror the vehicle behind can see
my eyes when I give an expression, they usually understand and wait that something is there
in front and they avoid honking. But if it was a teenager, they honk again and again."
Table 2. Types of noticing episodes interpreted with description.
Noticing types

Dialogue

Action Target (AT) +You Driving (YD)

When the participant reflects and talks onconceptual relationships like the old man's (AT)response in relationship with their vehicle's (YD)
behaviors towards him and its immediate
consequence towards the old man.
When the participant reflects and talks onconceptual relationships like the behavior of thecar behind (honking) (AP) in relation to old man's
behavior (AT) or traffic signal (AR).
When the participant reflects and talks about
conceptual relationships, like other vehicle's
response (ACP) in relation to old man's (AT)behavior or traffic signal (AR).
When the participant reflects and talks onconceptual relationships between their car (YD)
and behaviors of the honking vehicle behind (AP)or other vehicles by the side (ACP) or change intraffic signal light (AR).
When the participant reflects and talks onconceptual relationships between an object inthe scenario and retrospective accounts (R) of
that object from previous experiences.
When the participant reflects and talks onconceptual relationships like old man's (AT)behavior in relationship with other pedestrians
while crossing the road (like walking pace) or in
relation to changes in traffic signal light (AR).

Action Persuader (AP) +Action Target (AT)/Action Rule (AR)Action Choice Primer (ACP)+ AT/ARYD + AP/ACP/AR

Retro (R) + AT/AP/ACP

AT + ARThe types of noticing episodes inductively interpreted and categorized in Table 2 basedon the conceptual relationships will help critically analyze the participant's reasoning while
making decisions in complex scenarios. For our broader research objective of investigating
the effects of virtual reality perspective-taking, this approach will allow us to explore thechange in noticing patterns across multiple VR scenarios. For instance, in the "Old man
scenario," the noticing episodes of type "Action Target + You Driving" were interpreted
differently by the participant before and after perspective-taking. One of the dialogues ofParticipant 4 before perspective-taking is – "Now what do I say about that, signal was RED,but then the old man was there, so I just stopped, and also there is a rule that you have to give
first priority to the pedestrian. I am in a car so I can go easily but they are walking so". On thecontrary, the dialogue of Participant 4 after perspective-taking is – "Actually in that view I wasjust judging myself, so at that time the car shook a bit, because at that time when I was moving
right it was going too much right and likewise for left, So if I were an old man I would nevercross the road seeing someone shake the car like this, thinking that they might hit me. Also,386

the car came back and forth and I felt from the old man's view that if it was the old man, thenhe would have got scared.". These coded noticing episodes with six categories indicates thatthe adapted noticing framework used to design the VR intervention is meaningful and valuable
for more extensive studies with multiple complex VR scenario in our context. The number ofparticipants is a limitation of the study, and if the number is higher, there is a possibility ofinterpreting more types of noticing episodes in the future. One of the challenges in adapting
the noticing framework is to identify the appropriate conceptual elements within our context
and map them to the definition of noticing framework components.

6. Conclusion
Using advanced virtual reality technology, we developed complex scenarios that can show
alternate perspectives to help car drivers learn decision-making in complex on-road scenarios.
This paper discusses how we adapted the noticing framework, developed a complex VR
scenario, and inductively interpreted the participant's reasoning by categorizing the noticing
episodes. Four participants drove a car on virtual Indian city roads accompanied by a virtual
conversational agent to elicit their reasoning while making decisions. We interpreted thenoticing episodes into six types of conceptual relationships between the designed elements inthe VR scenario. In future work, we can use the identified noticing characteristics to identify
and analyze the patterns in noticing behaviors before and after takings perspective views
across multiple complex scenarios in the Indian road context. This paper demonstrates theusefulness of the adapted noticing framework for such contexts and can be used by Learning
Sciences and Education researchers for VR-based interventions.

References
Amador, J. M., Bragelman, J., & Superfine, A. C. (2021). Prospective teachers’ noticing: A literature
review of methodological approaches to support and analyze noticing. Teaching and Teacher
Education, 99.
Azevedo, R., Cromley, J. G., & Seibert, D. (2004). Does adaptive scaffolding facilitate students Õ ability
to regulate their learning with hypermedia ? q. 29, 344–370.
Benvegnù, G., Pluchino, P., & Gamberini, L. (2021). Virtual Morality: Using Virtual Reality to Study
Moral Behavior in Extreme Accident Situations. 316–325.
Cohen, L., Manion, L., & Morrison, K. (2017). Research Methods in Education. Research Methods in
Education, 916.
Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993). Wizard of Oz studies — why and how. KnowledgeBased Systems, 6(4), 258–266.
Funke, J. (2010). Complex problem solving: A case for complex cognition? Cognitive Processing, 11(2),133–142.
Goodwin, C. (1994). Professional Vision. American Anthropologist, 96(3), 606–633.
Kang, S., Chanenson, J., Cowal, P., & Weaver, M. (2012). Advancing Ethical Decision Making in Virtual
Reality. Emotion (Washington, D.C.), 12(2), 364–370.
Klein, G. (2008). Naturalistic decision making. Human Factors, 50(3), 456–460.
Klein, G. A. (1993). DECISION MAKING IN ACTION: MODELS AND METHODS. In Ablex publications.
Lobato, J., Rhodehamel, B., Hohensee, C., & Hohensee, C. (2012). “ Noticing ” as an Alternative
Transfer of Learning Process “ Noticing ” as an Alternative Transfer of Learning Process.
8406(May).
Lyle, J. (2003). Stimulated Recall: A report on its use in naturalistic research. British Educational
Research Journal, 29(6), 861–878.
Sütfeld, L. R., Gast, R., König, P., & Pipa, G. (2017). Using virtual reality to assess ethical decisions inroad traffic scenarios: Applicability of value-of-life-based models and influences of time pressure.
Frontiers in Behavioral Neuroscience, 11(July), 1–13.

387

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in Education.
Asia-Pacific Society for Computers in Education

Analysis of algorithmic strategy
development in the development ofcomputational thinking of upper
elementary school students
Xiaowen WANG, Pinqi HU & Guang CHEN*
Faculty of Education, Beijing Normal University, Beijing, China*guang@bun.edu.cn
Abstract: Computational thinking has become a thinking ability that we need to masterin our daily lives. Programming is a way to cultivate computational thinking. This study
focuses on the algorithmic links of senior primary school students in programming
activities, puts forward a hierarchical table of algorithmic strategies, and analyzes theirstrategic development paths according to their programming behaviors.In this paper,
Swift Playgrounds, developed by Apple Company, is selected as the tool ofprogramming activities, and three levels are selected as the teaching content in thethree theme activities of Learning Programming 1 in Swift Playgrounds software. The
target audience of the activities is the senior (fourth to sixth grade) students in primaryschools. This paper studies the design of the micro-generation method, carries out
programming activities based on computational thinking for students through pre-test,
practice, post-test and migration stages, and collects data by interview and classroom
observation so as to analyze the development path of students' algorithm strategies indifferent stages and the change track of algorithm strategies in each stage.This study
provides some support for teachers' teaching in programming teaching activities.
Keywords: Programming teaching activities, computational thinking, micro-genetic
method, algorithm strategy

1. Introduction
In the context of the rapid development of information technology, society has put forward
new requirements for people's abilities and accomplishments. People should learn tounderstand some problems encountered in social production, life and learning from theperspective of computer science, and be able to use the basic principles of computer scienceand advanced technical tools to solve complex and difficult problems (Tedre & Denning,2016).
Computational thinking is a key research topic in computer education. Different
researchers have different definitions of computational thinking, but the definition ofcomputational thinking proposed by Jeannette, which includes five elements: algorithms,
decomposition, abstraction, generalization and debugging, has been widely recognized byresearchers all over the world. Jeannette believes that computational thinking includes
thinking activities such as problem-solving, designing systems, and understanding humanbehavior. This is not only a skill that scientists need to possess, but also a skill that every oneof us needs to possess (Jeannette, 2006).
As one of the ways to cultivate computational thinking, programming education cancultivate students' systematic problem-solving and strategic thinking abilities. In the process ofprogramming, students can not only learn some basic knowledge but also learn logicalmethods and problem-solving strategies through specific problem situations, so it becomes a
powerful starting point for cultivating computational thinking (Winslow et al. ,2018).

388

According to the cognitive development level and thinking development process ofprimary school students, primary school students above grade three are graduallytransitioning from image generalization level to abstract generalization level, and theirreasoning ability is also in a stage of rapid development. They can independently and logicallydemonstrate more complicated judgments and processes. Therefore, at this stage, students'
computational thinking can be cultivated through programming teaching. For example,
Saez-Lopez et al.(2016) through on-the-spot evaluation of the effect of using Scratch's visualprogramming language in classroom practice, explore whether students have madesignificant progress in learning programming concepts, logic and computational practice;
Zhao & Shute(2019) asked students to play "Penguin Go" video game in the classroom, thatis, to create a program to guide penguins to cultivate students' cognition and attitude towards
the development of computational thinking skills through a regional approach; On the other
hand, Chen-Chung et al.(2011) designed a simulated game to let students learn and use
programming knowledge to solve situational problems, and explored the influence of
simulated games on learning experience and problem-solving strategies related toprogramming knowledge. Pellas & Peroutseas(2016) showed students their participation in a
3D multi-user game environment by combining Second Life with Scratch4SL's 2D
programming environment and used them to develop computational thinking skills incollaborative problem-based programming tasks.
To sum up, most of the current research on computational thinking focuses on thedevelopment and effect research of computational thinking activity courses, and the researchon students' behavior in the process of computational thinking is relatively lacking. This study
focuses on the cultivation of students' algorithm strategies in computational thinking activities.
Starting with the basic programming knowledge, it studies the strategies and behaviors usedby students when using programming knowledge to solve problems, and provides support for
front-line teachers' programming teaching process.
Therefore, this study intends to solve the following problems:(1) What are the algorithm strategies that students mainly use at different stages ofdifferent programming activities?
(2) What are the development paths of students' algorithms and strategies at different
stages of the same programming activity?

2. Methods
In this study, the micro-generation method is used to study the changing process ofstudents' algorithm strategies in the programming process, which is an exploratory preliminary
study. This study mainly analyzes the development of algorithm strategy among seniorprimary school students in Swift Playgrounds-based programming activities, and further
analyzes the path of students' strategy development according to the algorithm strategy grade
table.
The whole teaching process of programming activities is divided into three thematic
activities: command, function and For loop. Each thematic activity is divided into four stages:
pre-test, exercise and post-test. After the completion of the three thematic activities, acomplete programming project is carried out and used as a test in the migration stage. And
through interviews and classroom observation to collect data. The interview method is mainly
used to interview students after each stage in the process of teaching by the micro-generation
method so that students can orally express what strategies are used to complete this part ofthe challenge and why such strategies are used to solve the problem. Classroom observation
is mainly used to track and record the development of students' algorithm strategies in classso as to further analyze students' algorithm strategies.
The research object of this study is the students in senior grades (grade four to grade six)in primary schools, and they are recruited through online registration. A total of 15 students
were recruited, including 12 boys and 3 girls, aged between 10 and 12, with an average age of
10.9 years and a standard deviation of 0.59. There are 12 fourth-grade students, 2 fifth-gradestudents and 1 sixth-grade student. During the implementation of the activity, a total of 15
students participated, but 2 students failed to complete all the courses. After the389

implementation of the activity, a total of 15 students' learning data were received, of which 13
were valid. Among the valid data collected, from the point of view of school distribution, 9people are studying in primary schools in Beijing, while 4 people finish their usual studies athome. In terms of gender, there are 10 boys and 3 girls, with an average age of 10.2 years anda standard deviation of 0.38.
This experiment is divided into three courses, and the content taught in each course isexactly the same. During the class, each student is recorded on an iPod Touch. At the sametime, the iPad used by the students turns on the screen recording function, and acorresponding number of graduate teaching assistants are equipped for stage interviews andbehavior tracking.
In this experiment, there are 15 subjects, who have completed 12 levels of programming
game challenges in 3 theme activities, and designed 19 interview questions in the pre-test,
practice, post-test and migration stages. Based on this, the obtained data is encoded.
Combined with the specific programming behavior of middle school students in this researchcourse, the algorithm strategies used by students in this programming process are
summarized and classified, and according to the path and order of children's cognitive
development from low-level strategies to high-level strategies, three experts in the field of
psychology are invited to divide students' algorithm strategies into four levels, namely,
zero-level strategies, first-level strategies, second-level strategies and third-level strategies.
According to the different development characteristics of each level strategy, combined withstudents' specific algorithm behavior, the different level strategy characteristics of algorithm
strategy are defined, and according to the video analysis and interview answers of 13 subjects'
programming activities in Study 2, three experts teach coders to encode independently, finally
improve the algorithm strategy level, and analyze the reliability according to the consistency
percentage formula proposed by Holsti in content analysis. The r (reliability) of the algorithm
strategy rating table is 0.9, and the reliability of the evaluation records of three judges, A, B
and C, is greater than 0.9, which shows that the evaluation result of the main judge can beused as the result of content analysis, which achieves the consistency of coders. The strategy
level rating table is shown in the following table:
Table 1. Level Table of Algorithm Strategy
Algorithm Strategy Level
Definition
In the process of the algorithm, we rely on our own subjective
Level 0
judgment and experience to complete the algorithm.
On the basis of subjective judgment, simple methods are used
to complete the level goal, and the simple methods are limited
Level 1to the commands involved in the level goal.
On the basis of simple methods, use appropriate methods toLevel 2complete the level goal, but it is limited to simpler scene goals.
On the basis of appropriate methods, draw inferences fromothers and think independently about the best method needed
Level 3
to complete the checkpoint goal.

3. Results and Discussion
This chapter will analyze the data collected in the previous section. The study mainly uses
qualitative analysis to investigate the changes in students' strategies and behaviors in theprocess of programming. It mainly uses encoding the video of students' entry into theclassroom, and analyzing the interview contents in the pre-test, practice, post-test andmigration stages to discuss the changes in students' strategies in the programming process.
According to the micro-generation method, the breadth analysis of the same programming
activity in pre-test, practice, post-test and migration stages is carried out. Mainly through thethree theme activities, students' individual development routes are carried out, and theindividual development roadmap is drawn according to the development of children's strategic390

level at different stages. Based on this road map, this paper analyzes the migration degree ofstudents' algorithm strategies of three theme activities in the migration stage.

3.1 The development route of individual algorithm strategy in command theme
activities.
From the roadmap of individual development of the algorithm, it can be seen that fivepeople in the algorithm strategy of command theme activity have developed from Level 0 to
Level 2; One person developed from Level 0 to Level 2 and then dropped to Level 0 and thento Level 2; Two people developed from Level 0 to Level 2 and then fell to Level 1 and then to
Level 2; Five people directly developed from Level 1 to Level 2. There are four strategic
development paths in total.
From the development path of algorithm strategy, all students use low-level strategy inthe pre-test stage of command theme activities. In the practice stage, students can easily
master this method after teachers teach the method of counting grids and planning theshortest route, so the second-level strategy is used the most frequently in the post-test stage,
but a small number of students still use low-level strategy. In the migration stage, students'
algorithms all develop into high-level strategies, which also shows that in the algorithm
strategy of command sentences, low-level strategy is a monotonically decreasing
development model, and high-level strategy.

Figure 1. Roadmap of individual development of command theme activity algorithm strategy

3.2 The development route of individual algorithm strategy in function theme activities.
From the roadmap of individual development of the algorithm, it can be seen that four
people in the algorithm strategy of function theme activity have developed from Level 1 toLevel 3; Five people have developed from Level 1 to Level 3 and then dropped to Level 1; Fourpeople developed from Level 1 to Level 2 and then fell to Level 1. There are three strategic
development paths in total.
In the development path of algorithm strategy, all students use low-level strategy in the
pre-test stage. Through the function method teaching in the practice stage, students' algorithm
strategy has reached the second and third levels, but in the migration stage, Level 2 has
dropped to Level 0, the most frequently used strategy level is the first-level strategy, and thethird-level strategy is also used by students. It shows that in this part of the function, students
do not fully grasp the definition and usage of the function and learn to draw inferences fromothers. The usage times of low-level strategies and high-level strategies are evenly distributed,
and the students' algorithm strategies are not developing from low-level strategies tohigh-level strategies, but the development modes of low-level strategies and high-levelstrategies fluctuate and alternate constantly.

391

Figure 2. Roadmap of individual development of function theme activity algorithm strategy

3.3 The development route of the overall algorithm strategy for loop theme activity.
From the roadmap of individual development of the algorithm, it can be seen that one
person in the algorithm strategy of “For loop” theme activity has developed from Level 1 toLevel 2; There are 12 people who have always maintained the level of secondary strategy.
There are two strategic development paths in total.
In the theme activity of “For loop”, students' algorithm and strategy have maintained ahigh level from the pre-test stage to the migration stage, even though the most frequently usedstrategy in the post-test stage is the first-level strategy. This is because in the post-test stage,
it is necessary to customize the function and use the loop to write the code. According to thedevelopment map of algorithm and strategy in the function part, students can master thedefinition and usage of For loop skillfully in this part. So on the whole, the development ofadvanced strategies is constant.

Figure 3. Roadmap of Individual Development of Algorithm Strategy for Cyclic Theme Activity
Synthesizing the development paths of algorithm strategies of the three theme activities,
we can get the following results: the proportion of algorithm strategies used by middle school
students in command activities and “For loop” activities is getting higher and higher, and theoverall trend is from low-level strategies to high-level strategies, but in the later period,
low-level strategies have not all disappeared, and some students still use low-level strategies
to solve problems; The strategy development path of the function part shows this point moreclearly. The algorithmic strategy of the function part develops alternately. Even if students
learn more advanced function knowledge in the practice stage, more students still choose
lower-level codes to replace the function strategy in the migration stage, which shows thatstudents' cognitive strategies do not completely develop from low-level strategies to advanced
strategies, but students subconsciously choose their most skilled and direct strategies by
relying on their own knowledge and pre-concept experience in different situations.

4. Conclusion
This study uses the micro-generation method to study the changes of students' strategic
behavior in programming activities, and makes a preliminary exploration in programming
education. Through this study, it was found that students use different algorithm strategies
when completing different programming activities. In such simple programming activities ascommand statements and For loop, students can use advanced strategies taught by teachers
to complete tasks, which shows that students' algorithm strategies are constantly developingfrom low level to high level, and the migration effect of algorithm strategies is obvious.
However, in such difficult programming activities as function statements, the algorithm
strategies used by students before and after teacher intervention are not the methods taught
by teachers, but their own subconscious strategies, which shows that the algorithm strategies
have not developed from low level to high level in this link, and the transfer effect is not
obvious. Therefore, the development of cognitive strategies is complicated, and students needcertain cognitive resources for learning strategies, and choose the most appropriate cognitive
strategies to solve problems according to specific situations.

392

The results of this study also show the differences in the development of students'
cognitive strategies in mathematics. For example, in the "function" activity of this study, theteacher taught more advanced methods, but some students who had not been exposed toprogramming before did not master this method, so they still used their own methods in thesubsequent task level, and in the cognitive development of arithmetic, all students can always
complete the corresponding tasks according to the methods taught by the teacher. It showsthat some students don't have the pre-concept of programming activities, so they can't finish
coding and debugging according to the statement logic in programming in a short time.
However, arithmetic activities are activities that students have been exposed to sincechildhood, and students can combine the relevant experience and pre-concept in the processof growth with the methods taught by teachers, and finally complete the corresponding tasks.
Therefore, on the whole, the development of children's cognitive strategies is complex
and specific. Complexity is reflected in the fact that students do not completely abandon the
low-level strategies formed by previous conceptual experience in the process of solving
problems after learning more advanced strategies, but form a competitive relationshipbetween low-level strategies and high-level strategies in the process of thinking. Althoughusing more advanced strategies may increase the efficiency of solving problems, they are stillin the process of accepting and understanding the more complicated and difficult advanced
strategies they have just learned, which requires a lot of cognitive resources. It is impossible tocomplete a flexible application, so the specificity of cognitive strategy development is formed:
students choose the most suitable strategy according to different environments to achieve thegoal of solving problems.
There are also some shortcomings and defects in the design and implementation of thisstudy. From the experimental link and process, this experiment focuses on the development
process of students' algorithm strategy in programming activities, that is, based on whether
they successfully complete the checkpoint task, while ignoring whether students have written
wrong and redundant codes that will not affect the final task. Therefore, in the future researchand development of this kind of programming software, the process can be designed morestrictly, and if irrelevant code appears, it should also be prompted or challenged to fail, so as toensure that students can successfully write and run the code not because of luck or subjective
judgment. In addition, the selection of subjects in this experiment is not rigorous. Somestudents have already used Swift Playgrounds before taking part in the activity, so the results
obtained in the pre-test stage and practice stage are not much different from those obtained in
the post-test stage and migration stage, which leads to the unconvincing effect of theexperiment.

References
Tedre, M., & Denning, P. J. (2016). The long quest for computational thinking. Proceedings of the 16th
Koli Calling International Conference on Computing Education Research, 120–129.
Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking throughprogramming: What is next for K-12? Computers in Human Behavior, 41, 51–61.
https://doi.org/10.1016/j.chb.2014.09.012
Jeannette, M. W. (2006). Computational thinking. COMMUNICATIONS OF THE ACM, 49(3)Winslow, B., Danielle, B. H., Katherine, J. N., Ken, P., Natalie, F., Camilla, N. J., Byron, L., Patrick, L., &Kasia, M. (2018). Active Learning Environments with Robotic Tangibles: Children's Physical and
Virtual Spatial Programming Experiences. TLT, 11(1)Saez-Lopez, J., Roman-Gonzalez, M., & Vazquez-Cano, E. (2016). Visual programming languages
integrated across the curriculum in elementary school: A two year case study using "Scratch" infive schools [Article]. COMPUTERS & EDUCATION, 97, 129-141.
http://doi.org/10.1016/j.compedu.2016.03.003
Zhao, W., & Shute, V. J. (2019). Can playing a video game foster computational thinking skills?
COMPUTERS & EDUCATION, 141, 103633. http://doi.org/10.1016/j.compedu.2019.103633
Chen-Chung, L., Yuan-Bang, C., & Chia-Wen, H. (2011). The effect of simulation games on the learningof computational problem solving. Pergamon, 57(3)Pellas, N., & Peroutseas, E. (2016). Gaming in Second Life via Scratch4SL: Engaging High SchoolStudents in Programming Courses. JOURNAL OF EDUCATIONAL COMPUTING RESEARCH,
54(1), 108-143. http://doi.org/10.1177/0735633115612785

393

" !   !" " "   #" !
#" !"$  #" ! #"           !#&) $   $ !$#( $ 
(,-$-.- !)+ + -$/  #()&)"$ ,($/ +,$-2)!).-# +( &$!)+($ 0+ ( &&)!$ ( ($/ +,$-2)! &$!)+($ +% & 2 
8A+813-> ?=- /.?"   < 1; 1587:<)6< <7 8:-8):- <0- ?7:4,C; A7=<0 )6, <0- .=<=:- ?7:3.7:+- ?1<0.=6,)5-6<)4 367?4-,/- 7. :<1.1+1)4 6<-441/-6+-  6.7:5)4 ;-<<16/; ;=+0 );
5=;-=5;7..-:/:-)<7887:<=61<1-;160-4816/A7=<0)6,<0-/-6-:)48=*41+4-):6)*7=<6<01;8)8-:?-?144,1;+=;;<0-,-;1/67.)$1:<=)4=5)6@01*1<<0)<)15;<7+755=61+)<- <7 <0- 8=*41+ )*7=< <0- +)8)*141<1-; )6, 158)+< 7.  <0:7=/0 <-+06747/1-;=;-,16>1:<=)40=5)6; 

-,=+)<1764-):616/1616.7:5)4;-<<16/;>1:<=)40=5)6 
<>303-3+68>/6631/8-/3=+09?8.+>398+6>/-289691C>2+>3=><+8=09<73819?<A9<5+8..+36C
63@/= >3=-<3>3-+6>9:</:+</0?>?</1/8/<+>398=A3>2,+=3-589A6/.1/90 >3=+6=9-<3>3-+6
09<:+</8>=>96/+<8+,9?>+=>2/C+</90>/8>2/./-3=3987+5/<=09<>2/3<-236.</8H=?=/90 809<7+6=/>>381==?-2+=7?=/?7=900/<?83;?/9::9<>?83>3/=>9</+-29?>>9>A9
1/8/<+>3986/+<8/<=E:+</8>+8.-236.386/+<8381+,9?> 8>23=:+:/<A/.3=-?==>2/./=31890+)3<>?+6?7+8B23,3>>2+>+37=>9-977?83-+>/>9>2/:?,63-+,9?>>2/-+:+,363>3/=+8.37:+->90>2<9?12>/-2896913/=?=/.38@3<>?+62?7+8= '2//B23,3>3=
./=31809<=-3/8-/7?=/?7=A3>2@3=3>9<=,/>A//8>9 C/+<96.+--97:+83/.,C>2/3<:+</8>=9<1?+<.3+8= '2//B23,3>-?<</8>6C38-6?./=+8?7,/<90+->3@3>3/=>2+>-/8>/<+<9?8.
29A@3<>?+62?7+8=?>363D/=>9</+63D/0?8->398+63>3/==3736+<>92?7+8==?-2+=</-9183D/
0+-3+6/B:</==398= *236/>2//B23,3>3==>366/B:+8.381>2<9?12</=/+<-2+8../@/69:7/8>
:+<>90>2//B23,3>2+=+6</+.C,//8:369>/.+>A9<53817?=/?7=A3>22?8.</.=90@3=3>9<=      
'2/)3<>?+6?7+8B23,3>38-6?./=+8?7,/<9038>/<+->3@/+->3@3>3/=A2/</>2/@3=3>9<=-+8
A+65?:/B:/<3/8-/>2/.<3@/8>/-2896913/=?=/.>92/6:+@3<>?+62?7+8-2+<+->/<>2385
0//6=:/+5+8.+->635/+</+62?7+8+8.</06/->?:98>2/37:63-+>398=98>2/3<9A863@/= '2/-?<</8>37:6/7/8>+>39890>2//B23,3>38-6?./='$& +8.$++)+$++)+>A9
+->3@3>3/=A2/</@3=3>9<=38>/<+->A3>2+-9?:6/900+-3+6/B:</==398</-9183>398=$+-.&)' $(A2/</>2/@3<>?+62?7+8>/66=495/+8.?=/7?6>379.+6,/2+@39<+8+6C=3=/ 1 
0<97@3=398+8.=://-2>9380/<30>2/@3=3>9<=635/>2/495/ (,$("#)0, A2/</@3<>?+6
2?7+8-98@/<=/A3>2>2/@3=3>9<=A236/>2/?=/<38>/<0+-/.3=:6+C=@3=?+63D+>398=90.+>+
:<9-/==381,/238.>2/=-/8/=/ 1 2/+.:9=/+8.0+-3+66+8.7+<5><+-5381=://-2
</-9183>398+8.$&&2 +8+->3@3>C.3=-?==/.38./>+3638>2/09669A381=/->398 (8635/6/+<8381>+5381:6+-/3809<7+6=/>>381==?-2+=-6+==<997=+5/C-2+66/81/38
./=318381/.?-+>398+6/B:/<3/8-/=383809<7+6=/>>381==?-2+=7?=/?73= ("" ' (- 8>23=0<//-293-//8@3<987/8>=A3>27+8C/B23,3>=>9-299=/0<97@3=3>9<=90>/8=:/8.63>>6/>37/./-3.38130>2/CA+8>>938>/<+->A3>2+8/B23,3> '23=7/+8=>2+>>2/./=31890>2//B23,3>
=29?6. +>><+->>2/@3=3>9<=H+>>/8>398+8.:3;?/>2/3<38>/</=>>938>/<+->A3>23> 3@/8>2+>
>2/>+<1/>6/+<8/<:9:?6+>3983=>9 C/+<96.-236.</8+8.>2/3<+--97:+8C3811?+<.3+8=
A/./=318/.>2/@3<>?+62?7+8>9,/+-2+<+->/<90=3736+<+1/>9>2/C9?81@3=3>9<= (8635/

394

09<7+66/+<8381=/>>381=A3>2>/+-2/<=>9=-+0096.>2/6/+<8381+8.+8=A/<;?/=>398 >2/
6/+<8381/B:/<3/8-/38+7?=/?78//.=>9,/=/60/B:6+38+,6/A3>29?>>2/8//.09<2?7+8
+==3=>+8-/9<38>/<@/8>398 3@/8>2+>9?<@3=3>9<=+</:<37+<36C>9 C/+<96.-236.</8
A3>26373>/.</+.381+,363>CA/./=318/.>2/38=><?->398=?=/.38>2//B23,3>38=37:6/
6+81?+1/= '2/38=><?->398=+</13@/8@/<,+66C38=>/+.90+=>/B>.3=:6+C/.98>2/=-<//8 *236/>2/=.<3@381>2/38>/1<+>/.@3<>?+62?7+8+</-97:6/B/B:6+3838129A=?-2
-97:6/B=C=>/7=A9<53=89=7+660/+>/3>2/<:+<>3-?6+<6C>9+8+?.3/8-/90C9?81+1/
A3>29?>/8138//<381,+-51<9?8.38+=29<>+79?8>90>37/38+7?=/?7=/>>381 */2+@/./@/69:/.+%./>/->38>/<:</></=:98.0<+7/A9<5>9366?=><+>/>2/23126/@/6:<9-/==90
29A,/238.@3<>?+62?7+8=A9<5>9=/<@/+=>2/?8./<6C381=><?->?</09<=-+0096.381>2/
?8./<=>+8.38190=?-2:<9-/==<//8A+6.<+59A=53?<>*+81 
#8/90>2/03<=>/B23,3>+->3@3>3/=>2+>2+=198/>2<9?123>/<+>398=90./=31837:6/7/8>+>398
+8.?=/<>/=>3813=$&&2 8>23=+->3@3>C>2/@3=3>9<=2/6:>2/@3<>?+62?7+86/+<8>9</-9183D/>2/3<=366C+8.=/<39?=0+-/= '2/+->3@3>C3=9<1+83D/.3803@/:2+=/=
8>2/383>3+6 "&$#'&#":2+=/>2/@3<>?+62?7+81<//>=>2/@3=3>9<=+=>2/CA+65?:>9
>2//B23,3>/ 1 F$-# + ).&2).&$% -)-+2( -$/$-20$-#' + ,,-# + .--)(-),-+-G+8.,<3/06C38><9.?-/=>2/>+=5/ 1 F)+-#$,-$/$-2'")$("-)., 
-)& +(')+ ).-2).+!$& 1*+ ,,$)(, + & -' 3))'$(-)2).+! G '23=:2+=/-6/+<6C7+<5=>2/,/138838190>2/+->3@3>C 89?</+<63/<>/=>381A2/8>23=
:2+=/A+=89>38-6?./.>2/C9?81@3=3>9<=A/</?8-6/+<+,9?>A23-2:2+=/>2//B23,3>
A+=38+8.A2+>>2/C=29?6..9>9=>+<>3> 8>2/&# &#":2+=/>2/@3<>?+62?7+81?3./=>2/@3=3>9<=>9:9=/=366C+8.
=/<39?=0+-/=+8.-+:>?</=37+1/=90>29=/0+-/=+=><+38381.+>+=/>31?</ '9:
/0> '2/@3<>?+62?7+8:<9@3./=>2/C9?81@3=3>9<=A3>2-6/+<38=><?->398=/B+7:6/=
+8./8-9?<+1/7/8>>92/6:>2/7>2<9?129?>>23=:2+=/ '2/@3<>?+62?7+803<=>:<9@3./=
+8/B+7:6/=366C0+-/37+1/F)2).%()0#)0-)'% ,$&&2!  + 4,0#-'2
,$&&2! &))%,&$% G>2/8+=5=>2/@3=3>9<=>9:9=>=366C0+-/=F0).&&$% -)& +(0#-2).+,$&&2! &))%,&$%  ).&2).'% ,)' ,$&&2! ,-)# &*' & +( G?<381>2/.+>+-+:>?<381>2/@3<>?+62?7+8-98>38?/=>9=:/+5>9>2/@3=3>9<=F * $(",$&&2 & +( -- +$!(, &&)!2).+$!! + (-,$&&2 1*+ ,,$)(, G&?-2:2<+=/==/<@/89>
986C+=A9<.=90/8-9?<+1/7/8>=,?>+6=9+=38.3-+>398=>2+>>2/-+:>?<381:2+=/3==>366
9819381 >>2//8.90=366C0+-/37+1/=-+:>?<381>2/@3<>?+62?7+8-98-6?./=>23=
<9?8.90-+:>?<381+8.38><9.?-/=>2/8/B><9?8.F)0+ &&2&$% 2).+,$&&2! ,
-#$(%#/  ()."#--)& +(0#-2).+,$&&2! &))%,&$% 4&$% -)& +(()-# +)( )!2).+ 1*+ ,,$)(, + $,'2, +$).,!  (2).'% , +$).,! , G
8>2/# #"%&$'&#":2+=/>2/@3<>?+62?7+86/+<8=0<97>2/37+1/=-+:>?</.
+8.,?36.=+-6+==303/<>9</-9183D/>2/@3=3>9<=H=366C+8.=/<39?=/B:</==398= 8+..3>398
>9-977?83-+>381@/<,+66C>9>2/@3=3>9<==?-2+=F -' -% ')' (--)& +(-# 
$!! + (  -0 (2).+, +$).,! ,(,$&&2! ,G>2/?=/<38>/<0+-/+6=9:6+C=
+837+>398=A2/</38.3@3.?+6:3->?</=+</23126312>/.>998/,C98/>9=37?6+>/>2+>>2/
@3<>?+62?7+83=/B+738381+8.6/+<83810<97/+-237+1/ 8>2/# %&":2+=/>2/@3<>?+62?7+8><3/=>9</-9183D/>2/@3=3>9<=H0+-3+6
/B:</==398=?=381>2/6/+<8/.79./6F+2'%$(".(#)!$!! + (-,$&&2(, +$).,
! ,-), $!'"))- - +'$($("0#$#)( 2).+ '%$("G31?</ '9:
-/8>/<+8.'9:%312> '23=:2+=/6+=>=+<9?8. =/-98.= 8>2/038+6 &#":2+=/>2/@3<>?+62?7+8</@3/A=>2/37+1/=</-9183D/.+==366C
+8.=/<39?=0+-/=A3>22312-9803./8-/6/@/6=31?</ 9>>976/0>+8.9>>97-/8>/<F'*+ --2)(!$ (--#-2).0 +  $(",$&&2$(-#$,*#)-) )2)."+) ,-#$,
*#)-)&))%,$'$&+-)-# ,$&&2! ,2).,#)0 '  +&$ + G+8.>2/98/=A3>269A
-9803./8-/6/@/6/ 1 31?</ 9>>97%312>F##+-$' !$".+$(").-$!2).0 + '%$(", +$).,! )+,$&&2! $(-#$,*$-.+ #2)2).-#$(%,-+.""&  G '2/
19+690>2/</@3/A=3=>92/6:>2/@3=3>9<=</06/->29A><+38381.+>+37:+->=>2/:/<09<7+8-/=+8.>2/6373>+>39890>2/=>+>/90>2/+<>/ 1 2+<.>93./8>30C0+-3+6
/B:</==398=89>=//38138>2/><+38381.+>+=/> 395

'23=+->3@3>C-+:>?</=>2/@3=3>9<38:?>=@3++=3816/:2C=3-+6,?>>98A23-2+669A=>2/@3=3>9<=
>9=>+<>9<</=>+<>>2/+->3@3>C+8.+A/,-+7A23-2-+:>?</=>2/@3=3>9<=H0+-3+6
/B:</==398= !/.3+$3:/3=?=/.09<0+-/</-9183>398><+-5381 ?1+</=3/>+6  '2/
0+-3+6/B:</==398-6+==303/<3=37:6/7/8>/.?=381!9,36/"/>)998-/998-/
'2/@3<>?+62?7+8H==://-23==C8>2/=3D/.38</+6>37/?=381!3-<9=90>>/B>>9=://-2 '2/=://-21/=>?</+8.63:=C8-2+</:<9-/.?<+66C1/8/<+>/.38</+6>37/>2<9?12") //
+8.!+<=/66++8./</,/66+ 2977/>/>+6 ,+=/.98>/B>90>2/=://-2 

$".+ &-<//8-+:>?</=90>2/.300/</8>:2+=/=38>2/$&&2 +->3@3>C90>2/)3<>?+6
?7+8B23,3> )* !->2/@3<>?+62?7+81+>2/<=37+1/=90+@3=3>9<H==366C+8.=/<39?=
0+-/=+=><+38381.+>+ )*$& ($"#-+0>/<?=381>2/><+38381.+>+>9,?36.+
=366C =/<39?=/B:</==398</-9183D/<>2/@3=3>9<>/=>=29AA/663>=:/<09<7+8-/=  )--)')0>2/@3<>?+62?7+8</@3/A=>2/</-9183D/./B:</==398=A3>2>2/@3=3>9< "9>/+
7/7,/<90>2/</=/+<-2>/+7:9=/.+=>2/@3=3>9<38>2/=/366?=><+>398= '2/$&&2 +->3@3>C2+=</-/8>6C=>?.3/.>2<9?12+.+C/B23,3>+>+=-3/8-/7?=/?738
>2/(83>/.&>+>/=A3>29@/< @3=3>9<= #,=/<@+>398=0<97>2/:<94/->H=>23<.:+<>C
38./:/8./8>/@+6?+>398>/+738.3-+>/>2+>>2//B23,3>-</+>/.23126/@/6=90@3=3>9</81+1/7/8>+8..A/66>37/-97:+</.>99>2/</B23,3>=+>>2/7?=/?7 #>2/<+->3@3>3/=0<97>2//B23,3>=?-2+=$++)+$++)++8.$+-.& )' $(2+@/,//8:369>/..?<381=-3/8-/
0+3<=+>69-+62312=-2996= &>?.3/=98>2/=/+->3@3>3/=A3>2>+<1/>+?.3/8-/=+>=-3/8-/
7?=/?7=+</?8./<A+C>91+>2/<.+>+98+1/+::<9:<3+>/8/==90>2/=/+->3@3>3/=/ 1 -+8
>2/C9?817?=/?7@3=3>9<=?8./<=>+8.>2/495/=>96.,C>2/@3<>?+62?7+8-97/.3+8+8.
>2/1?3./.38>/<+->398=>92/6:@3=3>9<=1<+=:=>2/,/238.@3<>?+62?7+8= 
'23=</=/+<-2A+==?::9<>/.,C>2/(83>/.&>+>/="+>398+6&-3/8-/9?8.+>398?8./<<+8>
  8C9:38398=038.381=+8.-98-6?=398=/B:</==/.38>23=7+>/<3+6+</>29=/90
>2/+?>29<=+8..989>8/-/==+<36C</06/->>2/@3/A=90>2/"+>398+6&-3/8-/9?8.+>398 
:--6?)4,:)37?;31=:<"%)6/  -<-+<6<-:8:-< -;876,.:)5-?7:3<7/:7=6,<0-,-;1/67.;<=,-6<169=1:A16<7;A;<-5; !"  ! 
#" 
776+-776+-  7*14--<$ $#" #  "% !%"%" ! % ""!"" &"
--)6,):;-44)! 76>-:*)4*-0)>17:/-6-:)<7:.7:-5*7,1-,+76>-:;)<176)4)/-6<;6
:7+--,16/;7.6<-441/-6<$1:<=)4/-6<;
B
!8:16/-:0755-<&=')6,):;-44)!  -:-*-44))=<75)<1+/-6-:)<1767.676>-:*)4*-0)>17:
.7:>1:<=)40=5)6;6 !"   " ">74=5- =/):-;1")6/);0+4)6)0)6#*7?-2))A;(0)6/0)6/
'76/--  -,1)818-.:)5-?7:3.7:*=14,16/8-:+-8<176818-416-;

396

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Analyzing Learning Patterns and Potential
Interventions in First-Year Compulsory
Course at an Online University
Yasuhisa KATOa*
Professor, Tokyo Online University, Japan*kato.yasuhisa@internet.ac.jpaAbstract: Online universities are becoming increasingly popular for their flexibility,
especially for adult learners. However, the autonomy of online learning can lead tochallenges in maintaining motivation and engagement. This study at Tokyo Online
University examines the learning patterns of first-year students in a mandatory
academic literacy course. It analyzes the relationship between students' learning
history and grades to identify patterns of success and failure in order to develop
targeted intervention strategies. The literature review discusses MOOC attrition rates,
self-regulated learning strategies, and the relationship between learning patterns andacademic performance. The analysis clusters students based on engagement andperformance, reveals distinct patterns between successful and unsuccessful students,and highlights early course engagement as critical. The study recommends introducing
information on successful study strategies during new student orientation and sending
targeted messages to disengaged students.
Keywords: Online universities, Learning patterns, Intervention strategies

1. Introduction
Online universities offer flexible, on-demand courses that are ideal for adult learners whowant to retrain or upgrade their skills, freeing them from the constraints of time and place. As
higher education increasingly moves online, it is important to understand the challenges andopportunities involved. A key challenge in online learning is the need for individualautonomy, self-regulation, and sustained motivation, without which learners may abandon
their studies, a prominent issue in MOOCs (Hone & El Said, 2016).
Previous research highlights key factors for online student success, including time
management and social presence (Kizilcec, Pérez-Sanagustín, & Maldonado, 2017;
Richardson et al., 2017). This study aims to build on these findings to develop targetedintervention strategies for online students, inspired by Furukawa's (2022) analysis of MOOC
student data. Despite advances in MOOC research, studies of learning behaviors in fouryear online undergraduate programs are limited. Online students tend to be more intrinsicallymotivated and less socially connected than traditional college students (Kato, 2021; Kato,
2022b). Early engagement in courses correlates with higher final grades, as shown in a firstyear course at an online university (Kato, 2022a). This paper focuses on analyzing learning
patterns and exploring potential interventions for students, rather than detailing specificintervention implementations.

2. Literature Review
Studies of learning progress and attrition in online learning, particularly in MOOCs, areextensive. The average MOOC enrolls approximately 43,000 students and has a completion
rate of 6.5% (Jordan 2014). Literature reviews indicate that the rigid structure of MOOCs
may not be appropriate for students with diverse needs, and those who take courses
selectively or slowly may be considered dropouts (Onah et al., 2014). Learning analytics

397

have been used to identify successful and unsuccessful learning patterns in blended
environments (Tempelaar et al., 2015). A systematic review highlighted the importance ofunderstanding different learning patterns in online higher education and suggested thatfactors that contribute to success in traditional settings may be less effective online(Broadbent and Poon, 2015). Kizilcec et al. (2017) explored the relationship between selfregulated learning strategies, learner behavior, and goal attainment in MOOCs. Furukawa(2022) analyzed MOOC programming learning logs and identified clusters of early learners
to support based on characteristics such as number of video views and programming skills.
Kato (2022a) found that early course completion has a significant impact on final grades,suggesting that early intervention for students who are falling behind could potentially reduce
dropout rates.

3. Research Questions
The research questions are as follows:x What learning patterns are associated with successful completion of a course during thefirst half of the quarter?
x What learning patterns are associated with failure during the first half of the quarter?

4. Analysis of Learning Patterns
Figure 1 displays a scatterplot comparing students' first and eighth class completion dates
with final scores, where a score of 60 or above, marked by a red line, signifies success. The
graph indicates some early starters dropped out, while most who completed eighth classpassed.
Figure 2 shows students clustered by first class completion date and final score:
Group A started early, scored high, and did not need intervention. Group B started late,scored high, and benefited from the intervention. Group C started on time but failed andappeared to need time management intervention. Group D is a group that started slowly andlost credits, and how to intervene is an issue for the future.

Figure 1. Final scores and completion datesof the 1st and 8th classes.

Figure 2. Grouping of Students by Date ofFirst-Class Completion and Final Score.

Figures 3 and 4 show the full periodic learning histories for Groups A and C, whocompleted their first study within three weeks. The axes represent a quarter's schedule andprogress percentage, with 100% representing completion of all 8 classes. Due to similar
trends, the analysis divides the students into two groups: those who passed (A and B) andthose who failed (C and D). Clustering was done using the Python module "sklearn".

5. Future Discussion398

In the future, the following information should be presented at new student orientation prior
to the start of classes to help students understand their learning options and consider theirown learning styles:
1. Take your first class as soon as possible
2. There are several ways to take classes. There are several ways to proceed withthe class, either regularly or all at once in the first half of the class.
3. It is possible to catch up in the second half even if you fall behind in the first half.
4. If you don't start at the very beginning, you may not be able to finish.

Figure 3. Learning progress of Group A.

Figure 4. Learning progress of Group C.

Acknowledgements
This research was supported by JSPS KAKENHI under Grant Numbers JP22K12303. I
would like to thank the students at Tokyo Online University for their cooperation in this study.

References
Furukawa, M. (2022). A Study on Learning Behavior Analysis for Reducing Dropout in Online Course.
Ph.D. Thesis, College of Knowledge and Library Sciences, Tsukuba University (in Japanese).
Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study.
Computers & education, 98, 157–168.
Jordan, K. (2014). Initial trends in enrolment and completion of massive open online courses. The
International Review of Research in Open and Distributed Learning, 15(1).
Kato, Y. (2021). Analysis of Learning Motivations and Consideration of Appropriate Interventions forOnline Students. European Association for Practitioner Research on Improving Learning(EAPRIL).
Kato, Y. (2022a). Analysis of Learning Patterns and Final Grades in an Online University and a Studyof Appropriate Intervention Methods for Students. World Conference on Computers in Education,
Hiroshima, Japan.
Kato, Y. (2022). Yearly Trends in Student Motivation to Learn at an Online University and Comparisonby Academic Year. European Association for Practitioner Research on Improving Learning(EAPRIL).
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies
predict learner behavior and goal attainment in Massive Open Online Courses. Computers &education, 104, 18–33.
Onah, D. F. O., Sinclair, J., & Boyatt, R. (2014). Dropout rates of massive open online courses:
behavioral patterns. EDULEARN14 proceedings, 1, 5825–5834.
Richardson, J. C., Maeda, Y., Lv, J., & Caskurlu, S. (2017). Social presence in relation to students’satisfaction and learning in the online environment: A meta-analysis. Computers in humanbehavior, 71, 402–417.
Tempelaar, D. T., Rienties, B., & Giesbers, B. (2015). In search for the most informative data for
feedback generation: Learning analytics in a data-rich context. Computers in human behavior,
47, 157–167.

399

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Comparative Study of Traditional andAugmented Reality-Based Engineering
Drawing Instruction: Effects on
Visualization Skills and Cognitive Load
Ajay Shankar TIWARIa, Kaushal Kumar BHAGAT b*
Advanced Technology Development Centre, Indian Institute of Technology Kharagpur,West Bengal, 721302, India*kkntnu@hotmail.com

a,b

Abstract: Augmented reality (AR) is utilized in education to enhance students' skills
and learning experience. We developed an AR-based application for engineeringdrawing (ED) courses, evaluating its impact on 3-D visualization skills and cognitive
load. The study involved 392 first-year engineering students, with a control group using
conventional methods and an experimental group using the AR application. Results
showed that the experimental group outperformed the control group in 3-D visualization
skills and experienced lower cognitive load. Students responded positively to theapplication, indicating that AR can enhance learning performance and experience inengineering education.
Keywords: Augmented reality, marker-based, cognitive load, system usability,
engineering drawing

1. Introduction
Augmented reality (AR) technology enhances education by providing an interactive
experience where computer-generated information augments the real world (Azuma, 1997). It
helps students to visualize abstract concepts, increases motivation, and develops 3-D
visualization skills. This study focuses on using AR in an Engineering Drawing (ED) course toimprove spatial abilities and reduce cognitive load. A marker-based AR application wasdeveloped and evaluated using the System Usability Scale (SUS). Student feedback wascollected to identify strengths and weaknesses of the application for future improvements. The
specific research questions (RQ), this study intends to answer are as follows:x RQ1: Can AR based learning improve students' visualization of 3-D objects in ED
courses compared to the conventional instruction method?
x RQ2: Can AR-based learning in ED reduce students' cognitive load relative toconventional ED learning?
x RQ3: Is the augmented reality-based application easy to use for the ED course?
x RQ4: How satisfied are students with the AR-based learning method?

2. Related work
2.1 Augmented reality and its role in field of Engineering Drawing
AR integrates computerized information into the real world and has been implemented invarious disciplines, including education, recreation, tourism, and healthcare. In education, AR
has helped in developing immersive and interactive learning environments (Kaufmann &Schmalstieg, 2003). It enhances the classroom by providing real-time response, simulations,and visual aids, increasing student engagement and understanding. AR has improved learning
outcomes and motivation across different educational contexts, making difficult ideas moreaccessible (Chang & Hwang, 2020). Interactive AR simulations has enabled students to400

manipulate virtual objects and observe their behavior, simplifying the comprehension of
abstract concepts (Bacca et al., 2014). ED is an important course in engineering education,
but students struggle with visualizing 3-D objects from 2-D drawings (Pando Cerra et al., 2020).
There have been studies where AR has enhanced learning by providing interactive
experiences, generating 3-D models, offering visual aids, and enabling practical experience(Danakorn Nincarean et al., 2019; Martín-Gutiérrez et al., 2019). AR has also assisted inboosting motivation, engagement, and enjoyment in ED courses (Wei et al., 2011).

2.2 Cognitive load theory
Cognitive load theory (CLT) suggests that reducing the cognitive load enhances learning.
Multimedia, like images and videos, can increase cognitive burden, but interactive media canfacilitate learning (Sweller et al., 1998; Mayer & Moreno, 2003). AR decreases cognitive loadby providing an immersive and interactive learning environment (Billinghurst & Duenser, 2012).
Lee (2012) found that AR significantly reduced cognitive load in science learning compared to
traditional methods. However, the effectiveness of AR in reducing cognitive load depends on
system design and learners' prior knowledge (Huang et al., 2020).

3. Methodology
3.1 Research design and sample
A pilot survey was conducted to identify challenging topics in ED. Participants included 53
undergraduates and 18 experts. The survey revealed difficult areas such as solids projection,
cross-section, points, planes, and auxiliary planes. A true-experimental design was employedwith 392 randomly selected students (291 males, 101 females) aged 19-22. The experimentalgroup used an AR app, while the control group followed traditional methods. Each groupconsisted of 196 students. In this study, we developed a marker-based AR application using
Unity1 software to enhance 3-D visualization skills in an ED course. The application included
sections like orthographic view, projection of solids, and cross-section of solids. Users
selected sections, pointed their device's camera at markers, and viewed 3-D models. They
could interact with the models by rotating, moving, and resizing them. The application provided
various shapes and angles for visualization. Users were recommended to explore all sections
within a week to fully utilize its features (see Figure 1).

Figure 1. User Interface of application.

1

https://unity.com/

401

3.2 Instruments
We administered a pre- and post-test containing 8 test items each to evaluate 3-D visualization
skills, validated by subject expert (see Figure 2). Students were given 10 minutes to complete
this test. Thereafter, we used a cognitive load questionnaire for analyzing students' extraneous
cognitive load and intrinsic cognitive load during the experiment (Hwang et al., 2013). SUS,
created by Brooke (1996), was used for evaluating the usability of the AR application. It
consisted of ten items on a five-point Likert-scale. Students were given 25 minutes to complete
the post-test questionnaires. The Cronbach's value for all questionnaires was greater than 0.7,
which is acceptable (Barrett, 2001). We used a questionnaire to obtain qualitative feedbackon our AR application. It consisted of 10 subjective questions through which students could
discuss their experience using the application, thereby providing more support to our findings.
The questions were like: “Do you have any previous experience of using AR app?”, “Howmuch would you rate the app? “, “Did you find the AR useful?” etc.

Figure 2. Example of questions included in pre-test and post-test.

Figure 3: Experimental design402

3.3 Procedure
Figure 3 depicts the experimental procedure of the investigation. The study consisted of threestages: (1) Both groups were obliged to complete the pre-test within the allotted time; (2) theexperimental group was instructed to use the application to study ED for one week; and (3)both groups were required to complete the post-test. The experimental group was
subsequently required to complete an SUS questionnaire and provide qualitative feedback.

4. Results
We examined the data using descriptive statistics. The learning outcomes of the experimentalgroup and the control group were compared using t-test methodology. We used SPSS version21 for result analysis.
An independent t-test was employed to analyze the results, and the effect size
(Cohen's d for various sample sizes) was computed to demonstrate the extent of difference
between the groups. In the pre-test, there was no significant difference between the control
and experimental groups (t = 0.30, p > 0.05). In contrast, the post-test revealed a significant
difference between the control and experimental groups (t = 11.14, p < 0.05). In addition, we
conducted a t-test on the learning gains of both groups. There was a significant difference inperformance between the control and experimental groups (t = 6.64, p < 0.05; see Table 1).
Table 1: Means and standard deviations of the pre- and post-test for each group.
Group
Pre

Mean
4.37

SD
2.31

MD
0.07

t-value
0.30

Controlgroup
Experimental 4.44
2.29
group
Post
Control
4.34
2.68
2.01
11.14*
group
Experimental 6.41
1.11
group
Gain
Control
0.02
3.21
1.93
6.64*
group
Experimental 1.96
2.51
group
Note. * p <.05. SD = Standard deviation, MD = mean differencep
0.78

Effect Size
0.03

0.00

1.00

0.02

0.67

An independent t-test was conducted to examine the impact of the AR-based learning
approach on the intrinsic and extraneous cognitive load of students. The results showed asignificant difference between the two groups in terms of mental load (t = 16.78, p < 0.05) andmental effort (t = 15.86, p < 0.05; see Table 2). These outcomes implied that the AR-based
application had a significant impact on the students' intrinsic and extrinsic cognitive loads.
We calculated SUS score using technique proposed by Bangor et al. (2008). The total
SUS score was 92.1237. Figure 4 depicts the SUS analysis and feedback obtained by
students.
Table 2: Means and standard deviations of cognitive levels of each group.
Mental
load

Group
Mean
Control
2.86
group
Experimental 2.02
group

SD
0.59
0.37

403

MD
0.84

t
16.78*

p
0.00

Effect size
1.76

MentaleffortControl
3.05
0.81
1.05
15.86*
group
Experimental 2.00
0.45
group
Note. * p <.05. SD = Standard deviation, MD = mean difference

0.00

1.60

Figure 4: Summary of SUS data and Qualitative feedback from students

5. Discussions
The study assessed the effect of AR technology on improving students' 3-D visualization
abilities in ED courses compared to conventional methods of studying ED. We created an AR
application and encouraged the experimental group to study ED using the application while
the control group studied ED using the traditional method. Four research questions wereaddressed in the study.
The first addressed the impact of AR-assisted ED learning on students' 3-Dvisualization skills. The findings demonstrated that, as compared to traditional approaches,
AR significantly improved 3-D visualization skill in the ED. This result was aligned with earlier
research where AR was used to improve visualization skill of students from architecture
background (Escudero et al., 2016). In addition, they were able to visualize points and lines invarious planes, which was not possible during classroom instruction.
The effect of AR technology in reducing cognitive load in ED tasks is the focus of thesecond research question. The study found that students who exercised an AR-based
application experienced substantially lower intrinsic and extraneous cognitive load comparedto those who used traditional instruction methods. This result surpassed previous researchthat showed AR-based learning can minimize extraneous cognitive load in multimedia science(Lai et al., 2019).
The third research question addressed the usability of AR application for ED course.
The average SUS score obtained revealed that our application is user-friendly and appropriate
to ED courses. Previous research evaluating the utility of AR has shown equivalent results,
indicating that AR-based applications can be employed in ED classrooms (Wijaya et al., 2019;
Martin-Gonzale et al., 2010).
The final research question focuses on student satisfaction with the AR application. As
indicated in the results section, students who had prior experience with AR used theapplication with ease, as did those who were experiencing AR for the first time. The
application's nature as a marker-based AR application allowed users to avoid the difficulties
associated with marker-less AR applications (Wijaya et al., 2019).

6. Conclusions and future work
This study determined that AR-assisted learning in ED was able to enhance students' 3-Dvisualization skills. The results also showed that students who used AR to study ED

404

experienced less cognitive load than those who used conventional methods. The application
was subsequently evaluated using the SUS scale. It was also revealed that the application isuser-friendly and that students can study ED using this application. In their response, studentssuggested that the software can be used as an alternative to Auto-CAD because it allows
users to create custom designs. This turned out to be one of the limitations encountered by
our application. In addition, the application was only compatible with handsets running theAndroid operating system and not iOS.

References
Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators and Virtual Environments,
6(4), 355-385.
Bacca, J., Baldiris, S., Fabregat, R., & Graf, S. (2014). Augmented reality trends in education: asystematic review of research and applications. Educational technology & society, 17(4), 133-149.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability
scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594.
Barrett, P. (2001). Assessing the reliability of rating data.
Billinghurst, M., & Duenser, A. (2012). Augmented reality in the classroom. Computer, 45(7), 56-63.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4-7.
Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding
approach on students’ scientific project performance and perceptions. Computers &Education, 125, 226-239.
Danakorn Nincarean, A., Phon, L. E., Rahman, M. H. A., Utama, N. I., Ali, M. B., Abdi Halim, N. D., &Kasim, S. (2019). The effect of augmented reality on spatial visualization ability of elementary
school student. International Journal on Advanced Science Engineering Information
Technology, 9(2), 624-629.
Escudero, D. F., Domínguez, E. R., & Valls, F. (2016). Motivation and academic improvement using
augmented reality for 3D architectural visualization. Education in the Knowledge Society, 17(1),45-64.
Huang, C. L., Luo, Y. F., Yang, S. C., Lu, C. M., & Chen, A. S. (2020). Influence of students’ learning
style, sense of presence, and cognitive load on learning outcomes in an immersive virtual reality
learning environment. Journal of Educational Computing Research, 58(3), 596-615.
Hwang, G. J., Yang, L. H., & Wang, S. Y. (2013). A concept map-embedded educational computer
game for improving students’ learning performance in natural science courses. Computers &Education, 69, 121–130.
Kaufmann, H., & Schmalstieg, D. (2003). Mathematics and geometry education with collaborative
augmented reality. Computers & graphics, 27(3), 339-345
Lai, A. F., Chen, C. H., & Lee, G. Y. (2019). An augmented realityǦbased learning approach to enhancing
students’ science reading performances from the perspective of the cognitive load theory. British
Journal of Educational Technology, 50(1), 232-247.
Lee, K. (2012). Augmented reality in education and training. TechTrends, 56(2), 13..
Martín-Gutiérrez, J., Saorín, J. L., Contero, M., & Alcañiz, M. (2010, July). AR_Dehaes: an educational
toolkit based on augmented reality technology for learning engineering graphics. In 2010 10th
IEEE International Conference on Advanced Learning Technologies (pp. 133-137). IEEE.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning.
Educational psychologist, 38(1), 43-52.
Pando Cerra, P., Gracia Rodríguez, J., Fernández Álvarez, H., & Busto Parra, B. (2020). Combining
multimedia and self-assessment CAD tools in an interactive web environment to learn engineeringdrawing. Interactive Learning Environments, 28(1), 81-94.
Sweller, J., van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and instructional
design. Educational psychology review, 10(3), 251-296.
Wei, X., Weng, D., Liu, Y., & Wang, Y. (2015). Teaching based on augmented reality for a technicalcreative design course. Computers & Education, 81, 221-234.
Wijaya, A. C., Munandar, M. W. A., & Utaminingrum, F. (2019, September). Usability testing ofaugmented reality for food advertisement based on mobile phone using system usability scale.
In 2019 International Conference on Sustainable Information Engineering and Technology(SIET) (pp. 266-269). IEEE.

405

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Measuring Self-regulated Learning
Processes in Computer Science Education
Elizabeth B. CLOUDEa*, Ryan S. BAKERa & Maciej PANKIEWICZbaPenn Center for Learning Analytics, University of Pennsylvania, USA
bInstitute of Information Technology, Warsaw University of Life Sciences, Poland*ecloude@upenn.edu
Abstract: Self-regulated learning (SRL) is important for computer science education.
Yet, students often do not have SRL skills to benefit their learning. In this study, we
examined 187 (n=187) students’ SRL behaviors while they built programs with anautomated feedback tool. Anchored in Winne and Hadwin’s (1998) COPES model of
SRL, our results showed that novices used more operators to debug compiler errors,
while more experienced programmers used more operators to debug non-compiler
errors. Finally, a random forest classifier showed that prior knowledge was the mostimportant COPES feature predicting learning gain, followed closely by the student’s
perceived programming ability, use of evaluations with the automated feedback tool,and operators used to debug non-compiler errors on failed programs.
Keywords: Computer Science Education, Self-regulated Learning, Learning

1. Introduction
Computer science (CS) education requires students to develop self-regulated learning (SRL)
skills (Prather et al., 2018); yet many students, particularly novices, do not have the SRL
skills to benefit their performance (Arakawa et al., 2021; Loksa et al., 2016). SRL requires
monitoring and adapting learning processes and strategies to purse a goal. COPES (Winne& Hadwin, 1998), a widely adopted framework, emphasizes that feedback is the central
mechanism driving SRL, adaption, and learning.
Many CS educators rely on automated assessment tools since they offer automatic
feedback and evaluation (Chen et al., 2020), creating a feedback loop between the student
and automated tool, possibly opening doors to support SRL. Arakawa et al. (2021) examined
whether students struggled to initiate SRL as they programmed with an automated feedback
tool. Results showed that often students ignored the feedback from the tool altogether,perhaps due to a lack of SRL skills (Arakawa et al., 2021). Loksa and others (2016)
examined relations between coding errors and SRL strategies collected using a think-aloudprotocol. While students programmed with an automated feedback tool, the results showed
novices used SRL infrequently and often ineffectively for debugging errors. In contrast,
experienced programmers used SRL during programming and they made fewer errors.
Marwan et al. (2022) and Ko et al. (2019) found similar results to Loksa et al. (2016). Novice
and experienced students differed in their use of SRL strategies and coding errors with anautomated feedback tool. In sum, prior programming knowledge plays a role on how SRL
manifests with automated feedback tools and its relation to code quality. More research isneeded to explore relations between prior knowledge, SRL processes with an automatedfeedback tool, and learning outcomes in introductory programming.

2. Methods
Data were collected from 245 CS undergraduates (n=245) during a mandatory, firstsemester programming course (C#) at a large University in Poland. Due to missing data forinterest variables, students (n=58) were removed from our analysis, resulting in a subset of406

187 students (n=187; 39% female). The course covered 146 tasks on 8 basic CS education
topics. The automated assessment tool, runcodeapp.com (Pankiewicz, 2020) allowed
students to submit code on a pre-defined set of test cases. Feedback was provided as a
total score, compiler errors, and detailed results for each test case. To view detailed
feedback for each test case, students had to click on a selected test case.
Participants completed a 5-point Likert scale (1=low, 5=high) to report theirprogramming ability. An 8-item, multiple-choice pre-test was also administered to measuregeneral programming knowledge (Mdn=38.75). The Likert scale and pre-test measures wereused to assign students to a novice or more experienced group. A Wilcoxon test found pretest scores were significantly higher for students who reported more programming
experience (Mdn=65.88) than novices (Mdn=16.25), W=8079, p<.001. As such, students
were assigned to the more experienced group (n=87) if they reported a higher programming
ability (3-5), or to the novice group if low (1 or 2; n=100). A 9-item, multiple-choice test
evaluating knowledge of the course’s programming topics was administered at the middle(Mdn=55.56) and end of the semester (Mdn=66.67). Normalized learning gain (NLG) wasused to calculate a normalized score of the max possible change from mid- to post-test(M=.09, SD=.38; see Marx & Cummings, 2007). To explore SRL with an automatedfeedback tool and its relation to learning, we grounded our work in Winne and Hadwin’s(1998) COPES model (see Table 1). Log data on code and interactions with the automatedfeedback tool were collected.
Table 1. COPES construct descriptions and operational definitions.
COPES

Description

Conditions

Cognitive or external resources

Operators
Products
Evaluations
Standards

Operational definitions

x Self-reported programming ability.
x General programming knowledge scores.
Primitive information processing The proportion of consecutive test cases formechanics
code that scored <100: contained either a)compiler or b) non-compiler errors.
Knowledge shaped by operators Not analyzed in current study.
Feedback on discrepancies Average clicks on results of resubmitted
between products & standards
code from the automated feedback tool.
The criteria of success
Criteria for perfect program solutions.

3. Results
Do novices and more experienced programmers differ in the evaluations andoperators used to debug failed programs with compiler and non-compiler errors?
The Benjamini and Hochberg method controlled for type I errors. A Mann-Whitney test foundmore experienced (Mdn=.5) and novice programmers (Mdn=.63) marginally differed in how
many operators they used to debug compiler errors, W= 3687, adj. Į=.05, p=.07122. The
second test found that more experience programmers used more operators to debug
programs with non-compiler errors (Mdn=7.29) than novices (Mdn=6.96), W=5220.5, adj.
Į=.025, p=.01845. A separate Mann-Whitney found more experienced (Mdn=1.03) andnovice programmers (Mdn=1.04; p=.3852) did not differ in the use of evaluations on failed
programs with the automated feedback tool (p>.05).

Which COPES features are important for classifying if students learned?
For RQ 2, a random forest classifier was built to predict whether a student learned, from midto post-test using COPES variables. For that classifier, we split students into a group thatlearned (NLG>0) and a group that did not learn (NLG=<0). Data were partitioned into a
75:25 split for training and testing. To strike a balance between the best AUC and the lowest

407

error rate, grid search was used for the 1) number of trees grown and 2) COPES features
used (Table 1) to grow each tree. Ten repetitions of 10-fold cross-validation (in the training)were used. The random forest classifier achieved an AUC ROC of .63 (95% CI [.54-.72] andCohen’s kappa of .2553, with 2 features per decision tree. The important features forclassifying NLG were, first, conditions, general programming prior knowledge, and second,
perceived programming ability. The third most important feature was the evaluations
students used during programming, followed by the operators used to debug non-compiler
errors. The operators used to debug compiler errors had a mean importance of 0.

4. Discussion
Novice programmers do not utilize SRL effectively with automated feedback tools in CS
education (Ko et al., 2019; Loksa et al., 2022; Marwan et al., 2022). In this paper, we
explored the role of prior knowledge on SRL and its relation to learning outcomes. The first
RQ found that novices used operators more to debug compiler errors on failed programs. In
contrast, more experienced programmers used more operators to debug non-compiler
errors. By contrast, the groups did not differ in the evaluations of failed programs.
The second RQ found that the most important COPES feature for predicting students’learning gain was their level of general programming knowledge prior to the course. Thiswas followed by their perceived programming ability, how often they evaluated failed
programs with the automated feedback tool, and use of operators to debug non-compiler
errors. The number of operators used to debug compiler errors was not part of the predictorsof students’ learning. Although syntax plays important role at the initial stage, its importancedecreases over time. This work advances our understanding on the role of priorprogramming knowledge on SRL with an automated feedback tool and its relation onlearning outcomes in CS education. This research has two limitations. First, the use of log
data to study COPES, possibly missing other COPES behaviors that occur beyond systeminteractions, and 2) products were not collected, possibly missing additional data on SRL
during programming.

References
Arakawa, K., Hao, Q., Greer, T., Ding, L., Hundhausen, C. D., & Peterson, A. (2021). In situ
identification of student self-regulated learning struggles in programming assignments. In
Proceedings of the 2021 ACM Technical Symposium on Computer Science Education (pp. 467473). ACM.
Chen, H. M., Nguyen, B. A., Yan, Y. X., & Dow, C. R. (2020). Analysis of learning behavior in an
automated programming assessment environment: A code quality perspective. IEEE Access, 8,167341-167354.
Ko, A. J., LaToza, T. D., Hull, S., Ko, E. A., Kwok, W., Quichocho, J., ... Pandit, R. (2019). Teaching
explicit programming strategies to adolescents. In Proceedings of the 2019 ACM technicalsymposium on computer science education (pp. 469-475). ACM.
Loksa, D., & Ko, A. J. (2016). The role of self-regulation in programming problem-solving process andsuccess. In Proceedings of the 2016 ACM Conference on International Computing Education
Research (pp. 83–91).
Marwan, S., Akram, B., Barnes, T., & Price, T. W. (2022). Adaptive immediate feedback for blockbased programming: Design and evaluation. IEEE Transactions on Learning Technologies,
15(3), 406-420.
Marx, J. D., & Cummings, K. (2007). Normalized change. American Journal of Physics, 75(1), 87–91.
Pankiewicz, M. (2020). Move in the right direction: Impacting students’ engagement with gamification
in a programming course. In EdMedia+ Innovate Learning (pp. 1180-1185). AACE.
Prather, J., Becker, B. A., Craig, M., Denny, P., Loksa, D., & Margulieux, L. (2020). What do we think
we think we are doing? Metacognition and self-regulation in programming. In Proceedings of the2020 ACM Conference on International Computing Education Research (pp. 2-13).
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J.
Dunloksky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277304). Erlbaum.

408

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Construction of a Japanese Language
Learning Support System for Learning
Semantic Negotiation
Satoru Kogurea*, Akira YOSHIDAb, Yashihiro NOGUCHIa, Koichi YAMASHITAc,
Tatsuhiro KONISHIa & Makoto KONDOaaFaculty of Informatics, Shizuoka University, Japanb
Graduate School of Integrated Science and Technology, Shizuoka University, Japanc
Faculty of Business Administration, Tokoha University, Japan*kogure@inf.shizuoka.ac.jp
Abstract: Semantic negotiation techniques are important for beginners of Japaneselearner. In this study, we investigated how semantic negotiation is handled in a learning
environment and then reclassified it. We developed a Japanese language learning
support system that can handle semantic negotiation using text input. We report theresults of a questionnaire survey administered to three Japanese language teachers
and an experimental evaluation of a foreign student.
Keywords: Semantic negotiation, Japanese language learning, role-play, cooperative
learner agent

1. Introduction
The process of interaction between participants in a conversation, such as clarification
requests and confirmation checks, to understand the intentions of the other party, is called
semantic negotiation (SN) (Varonis, & Gass, 1985; Long, 1996). SN is important for second
language (L2) learners, who are likely to experience frequent communication problems in thelanguage they are learning. However, Miyazaki pointed out that in Japan, "research resultson SN are not reflected in Japanese language education" (Miyazaki, 2020). In fact, oursurvey of instructional materials used in Japanese language education (Three A Network,2016) revealed that most students did not use SN. We also administered a questionnaire
survey to Japanese language teachers. We found that three out of the four teachers did notteach SN in their classes because it was not a part of the curriculum. Thus, SN is notconsidered important in the field of Japanese language education. To solve these problems,
we constructed an environment that enables self-taught learners to study SN in Japanese.
The objectives of this research are (1) to propose a learning scenario that enables beginnerlearners of Japanese to learn SN as a means of adjusting to problems that occur in dialogue,and (2) to construct a text-based Japanese dialogue system that helps students to learndialogue and SN in Japanese through conversation-based experiences with a cooperative
learner agent (CLA).

2. Semantic Negotiation
The classification of SN in Japan has not been well established (Yokobayashi, 1991;
Fujinaka, 1996; Igari, 1998). In this study, we newly categorized and defined SN withreference to previous studies that deal with the classification of SN. The categories of SN
defined in this study can be divided into those conducted by the learner and those conductedby partner. SNs for learner include (A) reduction strategy and (B) achievement strategy. (B)strategy can be divided into (B1) speech strategy and (B2) listening strategy. In this study,409

we focused on (B1) speech strategies and classified them in detail. (B1) speech strategies
can be divided into (a) direct appeal, (b) indirect appeal, (c) non-linguistic, (d) implicit signalof uncertainly, (e) confirmation of partner's understanding, (f) code switching, (g) literaltranslation, (h) paraphrase, (i) substitution, (j) generalization, (k) neologism and (l)restructuring. SNs for partner include (i) error corrections, (ii) contributors/completions, (iii)elaborations, (iv) confirmation checks and (v) clarification requests. Among the negotiation of
meaning conducted by the learner, (A) the "reduction strategy" is not dealt with because it isdifficult for the system to recognize it. In this study, we primarily addressed (B) "achievement
strategies". Among (B) "accomplishment strategies," (B2) the "listening strategy" is limited tocases in which the system is not sure of the meaning of the other party's utterance and theuser wants to confirm whether his/her understanding is appropriate, according to thespecification of the system for text-based dialogues. Therefore, (B2) "listening-back strategy"
is a SN, in which the learner responds to the other party's utterance with a question, and theother party responds to the SN by answering the question. For (B1) "speech strategies," thisstudy covers SN (a), (e), (f), (g), (h), (i), (j), (k), and (l). We did not use SN (b), (c), and (d)because they cannot be realized in a text-based learning support system.
To investigate how much SN is actually taught in the field of education, we
administered a questionnaire survey to four teachers who teach Japanese at Shizuoka
University. In terms of learning materials, one respondent used "dialogue including SN
without explicitly stating it" while another respondent used "dialogue including SN with
explicitly stating it" as a learning material, and two respondents did not use "dialogue
including SN." This indicates that SN is not always taught in Japanese language study. In
the classes that used "dialogue including SN without explicitly stating it," only (i) error
corrections and (iv) confirmation checks were used, and the learners' (B1) "speech
strategies" were not covered. On the other hand, the lessons that involved "dialogue that
includes SN" as the learning material dealt with (A) reduction strategies, (B1) speech
strategies, and (B2) strategies for listening back (i) "corrections," (ii) "contributions andcompletions," (iv) confirmation checks, and (v) clarification requests. When focusing on the"speech strategies," only one of the four respondents used the material including thesestrategies in their classes. Two respondents who do not employ SN stated that they do notteach it because it is not part of the curriculum. This indicates that SN, or speech strategies,
are not taught in Japanese language learning.

3. Composition of a Japanese Language Learning Support System Targeting
Semantic Negotiation
In this study, we constructed a text-based dialogue system that supports SN and allows forfree input. The system is implemented in three parts: (1) scenario selection, in which thelearner chooses a scenario to be learned with reference to learning goals, learning content,
and dialogue situations; (2) a dialogue portion, in which the learner interacts with the agent
according to the situation provided; and (3) feedback, in which the system evaluates thelearner's dialogue from the teacher's perspective. In this system, in order to "set up a lesson
specialized for the learning of a certain SN," expressions that trigger the detection of the useof that negotiation are registered in advance in the contents prepared by the teacher. The
number of SN that a learner can learn acquire simultaneously is limited to one; hence, there
is no error in the detection of SN. The teacher needs to prepare two pieces of content inadvance for each lesson: "situation contents" and "SN contents". "Situation content" includes
the image file name of the situation, the caption, the learner's assumed utterances and the
CLA's utterances, as well as the results of the dependency parsing of the learner's
utterances. SN content includes the type of SN to be studied, the words and phrases thattrigger the negotiation, the learner's sentences assumed used in the negotiation, and theresults of the analysis of the engagement of these sentences.

410

4. Experimental Evaluation
To evaluate the usability of the system, we asked three Japanese language teachers whoteach Japanese to foreign students and one foreign student who is studying Japanese to
use the system. In an experiment with three teachers, we first (1) explained SN to theteachers, (2) asked them to view slides and videos on how to use the system, and (3) asked
them to view slides and videos on how to create content for this system. From the results of
the questionnaire, we obtained a rating (3.00 out of 5) that the system is somewhat effective
in helping students learn SN. On the other hand, we received 2.67 and 2.33 in the "would
you want your students to use it" and "would you use it in class" categories, respectively.
These results indicate that there is room for improvement of the system. In the experimentwith one international student, we administered (1) a pre-questionnaire, (2) an explanation of
SN and system usage, (3) a tutorial on system usage, (4) the first learning session of SN, (5)the second learning session of SN, and (6) a post-questionnaire. This foreign student had
not learned SN explicitly in class, but was already familiar with SN itself. We received ascore of 4 and a score of 5 (on a 5-point scale) in the evaluation of whether the system was
useful for learning SN and whether we would like to use the system in the future,
respectively.

5. Conclusion
We developed a Japanese language learning support system that enables self-study of SN,
based on the fact that SN techniques are very important for beginners in Japanese, but havenot been adopted in the broader field of Japanese language education. The results of thequestionnaire survey of two Japanese language teachers suggested that the system itself is
effective, but when asked if they would want to use it in their classes, they gave it a score ofthree or less out of five. In the future, it will be necessary to conduct experimental
evaluations with a larger number of Japanese language learners and to develop an
authoring tool that enables teachers who are not familiar with IT systems to create content.

Acknowledgements
This work was supported by JSPS KAKENHI Grant Number JP21K12172.

References
Fujinaka, K. (1996). A study on the communicative competence of beginner-intermediate learners ofJapanese: a consideration of communication strategies as a speaker, Bulletin of the international
center for Japanese language, 6, 51-69. (in Japanese)
Igari, M. (1998). A study on communication strategies in oral tests. Bulletin of international student
center, Yokohama national university, 5. (in Japanese)
Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. C.
Ritchie, & T. K. Bhatia (Eds.), Handbook of second language acquisition 413-468, New York:
Academic Press.
Miyazaki, S. (2002). Issues of semantic negotiation in second language acquisition research. Waseda
University Japanese Language Research, 1, 71-89. (in Japanese)Three A Network, (2016). Minna-no-Nihongo novice I Teaching Guide, Three A Network. (inJapanese)
Varonis, E. M. ,& Gass, S. (1985). Non-native/Non-native Conversations: A Model for Negotiation of
Meaning. Applied Linguistics, 6(1), 71-90.
Yokobayashi, H. (1991). Communication strategies used by beginning and intermediate Japanese
language learners. Proceedings of the 1991 spring meeting of the association for teaching
Japanese as a foreign language. (in Japanese)411

Shih, J. L. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in Education.
Asia-Pacific Society for Computers in Education

Creating Meaningful Connections: The
Role of Simultaneous Multi-Situational
Learning in Knowledge Contextualization
and Application
Chih-Yen CHENa, Su-Hang YANGb, Meng-Xuan XIEa, Yi-Chuan FANc, Jen-Hang WANGc*& Gwo-Dong CHENaa Department of Computer Science and Information Engineering, National Central University, Taiwanb Department of Hospitality Management, Chien Hsin University of Science and Technology, Taiwan
c Research Center for Science and Technology for Learning, National Central University, Taiwan*harry@cl.ncu.edu.tw
Abstract: This study investigates the effectiveness of a simultaneous multi-situational
learning system with embodied recognition to enhance learners' outcomes. Traditionalsituational learning systems provide a single situation for knowledge explanation orapplication, which limits learners' ability to fully immerse themselves in the learning
environment and receive immediate feedback. To address this, a simultaneous
multi-situational learning system was proposed to create a digital space with multiple
situations, allowing learners to immerse themselves in different scenarios
simultaneously and facilitating reflective learning. An experimental study with 60
undergraduate students in a Hospitality Japanese course compared a control groupusing the traditional situational learning system with an experimental group using the
proposed simultaneous multi-situational learning system. Results showed the
proposed system significantly improved learning outcomes, understanding, andinterest in learning. It enables the prompt creation of situational learning environments,
contextualizes knowledge, and fosters reflective learning with real-time feedback.
Overall, this study contributes to the field of education by providing a more effective
approach to situational learning that can be easily implemented in classroom settings.
Keywords: Situational learning, embodied cognition, Simultaneous multi-situation

1. Introduction
1.1 Backgroundġ
Situational learning is an approach that allows learners to immerse themselves in situations
where knowledge may be applied, thereby strengthening the connection between knowledge
and its practical applications in real-world situations. Brown, Collins, and Duguid (1989)
emphasized the importance of learning in an environment that corresponds to its situations
and showed that knowledge can be contextualized. Consequently, researchers explored how
to design a situational learning environment to enhance learners' learning outcomes, and they
showed that situational learning is an effective method helping learners to acquire knowledge(Herrington & Oliver, 2000). On the other hand, creating an appropriate situational
environment in the real world can be time-consuming and complicated, making situational
learning rarely be applied in traditional classrooms (Dede, 2009).
However, with the development of technologies, virtual environments can beconstructed more easily, leading the provision of situational learning methods in virtualenvironments. Wu et al. (2015) have proposed an approach in which real human images are
combined with virtual scenes to enable learners to not only immerse themselves in thesituation but also see their own appearance and inspect performance in the virtual412

environment. This approach can be particularly useful in establishing a situational learning
environment promptly in classrooms and facilitating reflective learning.
In addition, embodied cognition theory shows that learners can acquire knowledge
through the interaction between their bodies and the environment (Wilson, 2002). Furthermore,some research suggested that embodied interaction in virtual space can improve user
engagement (Lindgren et al., 2016). Recently, Wang et al. (2020) demonstrated that the
integration of body recognition services in digital learning environments allows for learners tointeract using embodied behaviors, Furthermore, computers can function as personal
instructors, identifying inadequate aspects of posture and emotional expression during thelearning process, and guiding learners to adopt correct behaviors (Xie et al., 2022).
However, during a learning process, the context is not limited to only one situation. The
Mantle of the Expert (MoE) theory, proposed by Heathcote and Bolton (1994), suggests thatwhen learners are asked to present a professional role by the instructor, they must learn
relevant knowledge and imagine a situation to achieve their goal. As a result, they will need toplay a variety of different roles when they face different events that change at the same time.
Additionally, the Legitimate Peripheral Participation (LPP) theory (Lave & Wenger, 1991)
states that the best way to learn complex skills is to become a member of a problem-solving
team. Therefore, it is necessary to understand the status of different roles in different
situations. As demonstrated, multiple scenarios are essential to explain various situations
during situational learning.

1.2 Motivation
As aforementioned, it is evident that current studies typically restrict learners' learning to asingle situation in which they only witness the application of knowledge but do not fully grasp
how it is contextualized. However, with the advancements in technology, particularly in thefields of AI, GPU, memory, and related technologies, situational learning systems can now be
combined with recognition services. This integration facilitates the provision of situational
feedback by detecting learners' embodied behavior while simultaneously presenting multiple
situations. Consequently, learners can immerse themselves in multiple scenarios
simultaneously, enabling them to contextualize and apply their knowledge in various contexts.

1.3 Research Objectives
This study aims to build a learning system that can identify users' embodied behavior andprovide them with the situation of knowledge explanation, the situation of knowledge
application, and the situation of post-problem-solving. The main goal is to verify whether thissystem can improve learners learning performance by allowing them to immerse insimultaneous multi-situational scenarios.

1.4 Research Question
This study investigates whether the proposed simultaneous multi-situational learning system,
which generates different situations for the needs of knowledge learning and service andenables the coexistence of various situations in a digital space, can lead to better learningachievement compared to traditional single-situation learning systems. Therefore, theresearch question is: "Could the simultaneous multi-situational learning system outperform
traditional single-situation learning systems in terms of learning achievement?"

2. Related Work
2.1 Situational Learning413

Situational learning is a valuable approach that emphasizes the interaction between learners
and the environment to acquire knowledge. Brown et al. (1989) emphasized that knowledge isacquired through interaction with contexts of the environment, making situational learning avaluable approach to strengthen the connection between knowledge and practical application.
Gagne et al. (2005) also argued that learning is most effective when it is applied to real-lifesituations. To improve learners' thinking, exploration, and problem-solving abilities, situational
learning aims to immerse learners in various situations for exploration and reflection. Scholars
have proposed various methods of situational learning, including authentic learning anddrama-based learning. Authentic learning combines learning content with real-world issues,
allowing learners to learn and practice at the same time, which has shown to improve learning
outcomes (Ornellas et al., 2018). Drama-based learning involves learners collaborating toperform a drama, which can increase learner engagement and promote active learning(Dawson & Lee, 2018).
Additionally, previous research on situational learning has used different approaches,such as situational classrooms and simulated environments. The English Village is an
example of a situational classroom that transforms a traditional classroom into a foreign
language learning environment (Kelch, 2011). Similarly, the Clinical Skills Center of the
National Taiwan University Hospital has created simulated spaces for performing surgeries
and medical examinations to allow learners to become familiar with the process of seeing adoctor. With advances in technology, situational learning has been combined with multimedia
technology to simulate learning situations in real environments. For instance, Chou et al.
(2012) developed a computer game that combines game-based learning with situational
learning, enabling elementary school students to learn how to react during earthquakes.
Moreover, digital spaces built in the classroom based on drama-based learning enable
learners to perform dramas and learn in the situation (Wu, 2015). All of the above confirm thefeasibility of digital learning platforms for situational learning.

2.2 Embodied cognition
Recent technological advancements have made it possible to create realistic environments ina digital way. Studies have indicated that situational learning in digital reality can improvelearning outcomes (Giasiranis & Sofos, 2016). In recent years, human-machine interaction
has become more diverse, with additional sensors or recognition systems detecting users'
embodied behaviors. Linden Lab's Second Life, for instance, uses Windows, Icon, Menu, andPointer (WIMP) for interaction, allowing learners to control virtual avatars through devices andinteract with the virtual world through WIMP. With the development of VR and AR, users canexperience more immersive interactive experiences (Buttussi & Chittaro, 2017). Researchsuggests that using VR in education can enhance learners' learning motivation and improvetheir performance, outperforming observation and exploration (Kavanagh et al., 2017; Kwon,
2019).
Situational learning combined with embodied interaction can be advantageous foreducational applications. However, in most embodied interaction situations, users enter thedigital world as a virtual avatar, which makes it difficult for them to observe their own
performance in the digital world and engage in reflective learning. To address this limitation,
Wu et al. (2015) developed the Digital Learning Theater (DLT), a drama-based digital reality
system that allows learners to immerse themselves in the digital environment through a mirror
image, enabling them to see their own performance during the drama and engage in reflective
learning.

3. Implementation
3.1 Design Concept414

In this study, we designed a simultaneous multi-situational learning system that aims toenhance learners' connection between situations and knowledge. The system consists of foursituations: the "situation of knowledge presentation," "situation of knowledge explanation,""situation of knowledge application," and "situation of post-problem-solving." The "situation ofknowledge presentation" provides learners with the necessary concepts and knowledge
through textbooks and lectures. The "situation of knowledge explanation" is an interactive
mechanism that allows learners to select specific contents for more in-depth learning. The"situation of knowledge application" provides learners with scenarios that allow them to apply
what they have learned to real-world situations. Finally, the "situation of post-problem-solving”
provides a feedback mechanism that allows learners to reflect on and incorporate theknowledge and experience gained from the previous situations into their own learning process.
To implement the system, learners will access the different situations through a digital platform,and each situation will be designed to provide a unique learning experience. The system will
evaluate learners' performance by monitoring their embodied behavior during each situationand providing feedback in the "situation of post-problem-solving." The feedback will be
designed to help learners reflect on their performance and incorporate their learning into future
situations, thereby improving their learning outcomes.

3.2 System structure
The system design of this study consists of three modules: the drama-based simultaneous
multi-situational display system, the simultaneous multi-situational editor, and the database,as shown in Figure 1. Instructors and learners can use the simultaneous multi-situational
editor to manage materials and edit the script stored in the database. When learners perform
in a drama, the simultaneous multi-situational display system retrieves the relevant data fromthe database and overlays multiple scenes, changes scenes, and dresses focus situations.
Finally, the learners can observe their appearance and performance in the digital space via
the displayed video.

Figure 1. System structure
The simultaneous multi-situational learning system comprises three parts. The firstpart, shown in Figure 2, is the data editing system, which allows editors to add events,
materials, scenes, props, and costumes that occur when the event is triggered. The second
part is the embodied situational learning system, which processes the events set in the415database according to the previous editing system. This system mainly involves overlapping
multiple scenes, scene changes, and focus situation dressing. Additionally, the system
recognizes and processes the generated scene and the original situational scene according to
the learner's action response. The third part is the database, which stores the learningmaterials and scripts based on the Hospitality Japanese textbooks. The teaching content
includes the complete reception process of the waiters, starting from the reception, menu
introduction to the checkout. These materials are discussed with the instructors of the
Hospitality Japanese course to help learners learn Hospitality Japanese and the response
actions during the reception process.

Figure 2. Simultaneous multi-situational editor interface

3.3 Real-time embodied cognitive system
The real-time recognition cognitive system consists of two parts, including speech recognition
and body recognition. Figures 3 and 4 illustrate the flowchart of the system.
Speech recognition is the first part of the system, which is processed through the
speech recognition service in Microsoft Azure. The Speech-to-Text API converts audio into
Japanese text, and punctuation processing is performed to minimize errors during content
comparison. The processed results are then compared with the lines, enabling the system tomake a corresponding situational change to the background and props based on therecognized results.

Figure 3. Speech recognition flowchart
The second part of the system is body recognition, which involves training therecognition model using TensorFlow and the Long-term Recurrent Convolutional Networks(LRCN) method. This model combines CNN and LSTM, where Convolutional layers extract
features from each video frame, and these features are sent to the LSTM layers to process thetime relationship between each frame and improve recognition of continuous actions. Thismodel solves the issue that traditional CNN models can only recognize a single frame andsimulates the impact of the user's performance on the situation. After identifying the body416

recognition results, the system makes a corresponding situational change to the backgroundand props based on the recognition results.

Figure 4. Body recognition flowchart

4. Method
4.1 Experimental Hypothesis
To evaluate the effectiveness of the proposed system, the present study tests the following
hypothesis: The use of a simultaneous multi-situational learning system that provides
immediate feedback based on learners' embodied interactions and simultaneously presents
multiple situations is more effective in terms of learning achievement than a traditionalsituational learning system that only provides a single situation. The study aims to compare
the learning outcomes of the two systems to determine whether the proposed simultaneous
multi-situational learning system outperforms traditional situational learning systems in termsof learning achievement.

4.2 Experimental Subject
This study was conducted in collaboration with a local university of science and technology
located. The sample consisted of 60 participants from two classes in the Department of
Hospitality Management. The two classes were randomly assigned to either the experimentalgroup (30 participants) or the control group (30 participants). To ensure comparability between
groups, participants within each class were divided into small groups of six based on theirpre-test scores. The experimental group was exposed to a simultaneous multi-situational
learning system that provided additional situational changes based on learners' embodied
behaviors during learning activities, while the control group received a traditional
single-situation learning system where the instructor changed the situations on the screen
according to learners’ embodied behaviors during the rehearsal of the drama.

4.3 Experimental ProcessThis study employed a six-week experimental process, with one class session per week andtwo hours per class session, which is depicted in Figure 5. The instructor for the two groups
was the same.

417

Figure 5. Experimental Process
During the first week, the learners took a pre-test and received an introduction to thesystem. Over the next two to four weeks, the course content was taught, and group practicesessions were conducted. In the fifth week, the learners performed in a formal setting, and a
post-test and questionnaire were administered during the final week. To enhance the learning
experience, the system was also accessible online, providing learners with the opportunity topractice on their own time and increase their familiarity with the system.
The environmental design of curriculum is shown in Figure 6, where the classroom isdivided into an actor area and an audience area. The screenplay in the casting area differs
between control group and experimental group, where the experimental group used thesimultaneous multi-situational learning system, while the control group used a traditional
single-situation learning system. The control group only depicts the knowledge application
situation in the restaurant scene, while the experimental group presents a multi-situational
screenplay combining two situations: the situation of knowledge explanation for thedescription of ingredients and the situation of knowledge application for the description of therestaurant scene.

Figure 6. The environmental design of curriculum

4.4 Experimental Measurement Tools
To evaluate the effectiveness of the proposed simultaneous multi-situational learning system,
pre-tests and post-tests were conducted to assess the learners' academic knowledge in
hospitality situations, their proficiency in Japanese oral communication, and their familiarity418with Japanese culture. These tests were based on the Hospitality Japanese textbook used in
the course and were designed in consultation with the course instructor. The tests included
true/false, multiple-choice, and fill-in-the-blank questions, with a maximum score of 100 foreach test. Analysis of Covariance (ANCOVA) was employed on the test scores to analyze theimpact of the learning system on the learners' academic performance. Additionally, aquestionnaire was administered using a Likert five-point scale to understand the learners'
perceptions on the system's impact on their motivation and confidence. The questionnaire wasdesigned based on the ARCS model (Keller, 1987) and was validated by expert professors,with a Cronbach's alpha of 0.972, indicating high reliability.

5. Result
5.1 Pre-Test and Post-Test Result
To evaluate the normality of the data, Shapiro-Wilk test was employed for both theexperimental and control groups as the sample size for each group was less than fifty. The
results showed that the experimental group pre-test (p = .120 > .05), control group pre-test (p= .145 > .05), experimental group post-test (p = .111 > .05), and control group post-test (p= .432 > .05) all fulfilled the assumption of normal distribution. To further test the assumption ofhomogeneity of regression coefficients, the results showed that the assumption was met. (F= .015, p = .902 > .05). Moreover, the assumption of the equality of variances was also
supported by the result of Levene’s test (p = .470 > .05).
Based on the above results, ANCOVA was performed. The result shows that the
average and the adjusted mean of the experimental group post-test were 73.673 and 72.293,
respectively; while those of the control group were 65.507 and 66.257, respectively.
Furthermore, there is a significant difference between the two groups (F = 4.303, p = .043
< .05). The partial eta square value was .07, with a range from .06 to .14, which indicates theeffect size is at a medium level (Cohen, 1992). The data indicate that the simultaneous
multi-situational learning system was significantly more effective than the traditional situational
learning system with a single situation in terms of learning performance.
Therefore, the results of this study support the hypothesis that the use of asimultaneous multi-situational learning system is more effective than using a traditionalsituational learning system with a single situation.

5.2 Questionnaire Result
The results of the questionnaire indicated that there were no significant differences in themean scores of the user's system experience, learning motivation, and building confidence
between the experimental and control groups. Nonetheless, both groups provided positive
feedback on the experimental content, demonstrating that multimedia tools can effectively
engage and motivate learners' interest, as well as align with the principles of situational
learning and enhance their motivation to learn.

5.3 Interviews
In order to gain a deeper understanding of the experimental group's experience, interviews
were conducted with four learners and the instructor who participated in the experiment. In
terms of system control, the participants expressed that the simultaneous multi-situational
learning system provided a novel control method. Compared to using mobile phones orcomputers, this method provided a more immersive experience. Some learners also
expressed a desire for the system to point out their errors during learning, which could improvetheir long-term learning performance.
Regarding situational changes, learners indicated that the application of the situation,
explanation of the situation, and the resulting situation after the change can effectively link419

them to the knowledge they want to express. The learning model allowed them to have a
deeper understanding of what they were learning and a better understanding of the textbook
knowledge. Furthermore, the learning model increased their motivation to learn. Someparticipants stated that the system had made them eager to express more diverse knowledge
while others hoped for stronger audio or visual effects.

6. Conclusion
The aim of this study was to investigate the effectiveness of a simultaneous multi-situational
learning system on learners' learning achievement. The proposed system created a digitalspace with multiple situations, allowing learners to perform and reflect on their behavior inreal-time. This approach addressed the limitations of traditional single-situation learning, suchas the difficulty of immersing learners in a single situation and the lack of real-time assessment.
The experimental results demonstrated that the performance of the experimental group was
significantly better than that of the control group in the post-test.
Furthermore, the questionnaire and interview results indicated that the digital spacewith multiple situations enhanced learners' understanding and connection to the knowledge
and increased their interest in learning. Therefore, the contribution of this study is the proposalof a new type of simultaneous multi-situational learning system and the validation of the
hypothesis that a simultaneous multi-situational learning system is more effective than a
traditional situational learning system.
Based on the feedback from the experimental group's interviews, students expressedthat some variations in situations were not engaging enough, future research could focus on
exploring the incorporation of sound effects to enhance the overall attractiveness andeffectiveness of the learning experience. Furthermore, with advancements in Augmented
Reality (AR) and Mixed Reality (MR) technologies, such as the Mixed Reality Toolkit (MRTK)
offered by Microsoft, visual effects could be enhanced to create more realistic multi-situational
environments. Moreover, adding error guidance could further enhance the learning process,allowing learners to instantly correct their posture, pronunciation, facial expressions, and otheraspects of knowledge that are not typically expressed through traditional paper-basedlearning.

Acknowledgements
This study is supported by the National Science and Technology Council, Taiwan.
Grand Number: MOST 109-2511-H-008-004-MY3 and NSTC 112-2811-H-008-006.

Reference
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational
Researcher, 18(1), 32-42.
Buttussi, F., & Chittaro, L. (2017). Effects of different types of virtual reality display on presence andlearning in a safety training scenario. IEEE Transactions on Visualization and Computer Graphics,
24(2), 1063-1076.
Chou, Y. S., Hou, H. T., Yu, M. C., Lee, H. J., Wu, H. S., Yang, Y. T., & Liao, Y. J. (2012). Running
Tommy©: Developing a Digital Adventure Game Based on Situated Learning to Promote Learners'
Concepts of Earthquake Escape. In Digital Game and Intelligent Toy Enhanced Learning(DIGITEL), 2012 IEEE fourth international conference on pp. 156-158.
Dawson, K., & Lee, B. K. (2018). Drama-based pedagogy: Activating learning across the curriculum.
Intellect Ltd.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69.
Gagne, R. M., Wager, W. W., Golas, K. C., & Keller, J. M. (2005). Principles of Instructional Design (5thed.). Wardswoth/Thomson Learning.

420

Giasiranis, S., & Sofos, L. (2016). Production and evaluation of educational material using augmented
reality for teaching the module of “representation of the information on computers” in junior highschool. Creative Education, 7(9), 1270-1291.
Heathcote, D. and Bolton, G. (1995) Drama for Learning: Dorothy Heathcote’s Mantle of the Expert
Approach to Education. Portsmouth, NH: Heinemann.
Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning
environments. Educational Technology Research and Development, 48(3), 23-48.
Kavanagh, S., Luxton-Reilly, A., Wuensche, B., & Plimmer, B. (2017). A systematic review of Virtual
Reality in education. Themes in Science and Technology Education, 10(2), 85-119.
Kelch, K. (2011). Curriculum development in English language teaching: Innovations and challenges for
the Asian context. International Journal of Organizational Innovation (Online), 3(3), 22.
Keller, J. M. (2010). Motivational Design for Learning and Performance: the ARCS Model Approach.
Boston, MA: Springer.
Kwon, C. (2019). Verification of the possibility and effectiveness of experiential learning using
HMD-based immersive VR technologies. Virtual Reality, 23(1), 101-118.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge
university press.
Lindgren, R., Tscholl, M., Wang, S., & Johnson, E. (2016). Enhancing learning and engagement
through embodied interaction within a mixed reality simulation. Computers & Education, 95,
174-187.
Ornellas, A., Falkner, K., & Edman Stålbrandt, E. (2019). Enhancing graduates’ employability skills
through authentic learning approaches. Higher Education, Skills and Work-Based Learning, 9(1),107-120.
Wang, J. H., Chen, Y. H., Yu, S. Y., Huang, Y. L., & Chen, G. D. (2020, July). Digital learning theaterwith automatic instant assessment of body language and oral language learning. In 2020 IEEE
20th International Conference on Advanced Learning Technologies (ICALT) (pp. 218-222). IEEE.
Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625-636.
Wu, W. Y., Luo, Y. F., Huang, D. Y., Huang, C. W., Peng, Y. I., & Chen, G. D. (2015). A Self-Observable
Learning Cinema in the Classroom. In The 23rd International Conference on Computers in
Education (pp. 257-262). Asia-Pacific Society for Computers in Education.
Xie, M. X., Wu, Y. W., Chen, G. D., Wang, J. H., & Yang, S. H. (2022, July). An immersive situationalgroup learning system with body movement and emotion recognition combined with subject
knowledge. In 2022 International Conference on Advanced Learning Technologies (ICALT) (pp.
292-296). IEEE.
Cohen, J. (1992). Statistical power analysis. Current directions in psychological science, 1(3), 98-101.

421

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education     %,1/!*(6%$ -4%/%$
%#-++%,$!1(-,0501%+(,!2*1(-$%
 501%+0!,)! ! /%,$!,   " (*(,' !(/-!)(  !
3$"%&-*$&.3&1'/1/-043*.(".%&%*"34%*&28/3/ .*5&12*38"0".
4
&.3&1'/1../5"3*5&&2&"1$)".%%4$"3*/.*."3"$*&.$&8/3/ .*5&12*38"0".
/;<7D3F:@7 ;E3@=3 L=KAFA G 35 <B"01/!#1  6G53F;A@3> D75A??7@67D EKEF7?E 3D7 ;@5D73E;@9>K 475A?;@9 3 5AD7
873FGD7 A8 ?A67D@ 76G53F;A@3> EKEF7?E  '8F7@ F:7 D75A??7@67D 5A?BA@7@F A8 3
EKEF7?;EF;9:F>K;@F79D3F76AD?;9:F47D7?AF7>K>A53F76I;F:AGF3557EE;@963F38DA?
AF:7D>A53>EKEF7?E ,:;EB3B7DBDABAE7E38D3?7IAD=53>>76$!*;@I:;5:>A53>
76G53F;A@3>EKEF7?E53@IAD=3@6E:3D763F3I;F:39>A43>D75A??7@67DEKEF7?F:3F
EB3@E ?G>F;B>7 76G53F;A@3> ;@EF;FGF;A@E  !@ B3DF;5G>3D 3@ ! 6D;H7@ D75A??7@63F;A@
EKEF7? ;E ;@FD;53F7>K ;@F79D3F76 I;F:;@ 3 ?G>F; @A67 >73D@;@9 ?3@397?7@F EKEF7? +;FG3F76 3F F:7 ;@F7DE75F;A@ A8 >3D97 E53>7 63F3 3@3>KE;E 3@6 B7DEA@3>;L76 76G53F;A@$!* 788;5;7@F>K BDA57EE7E :7F7DA97@7AGE 63F38DA? 6;H7DE7 $%+ I:;>7 7@EGD;@9
63F3 E75GD;FK 3@6 BD;H35K  ,:7 BDABAE76 $!*E 3D5:;F75FGD7 63F3 B;B7>;@7 3@6
BDA57EE;@9 ?75:3@;E?E 3D7 7JB>AD76 ;@ 67F3;> 8A5GE;@9 A@ ;FE E73?>7EE ;@F79D3F;A@
I;F:F:77J;EF;@9;@8D3EFDG5FGD7 K>7H7D39;@9%A@9A5:3@97EFD73?E3@6D7>3F;A@3>
63F343E7E $!* 9G3D3@F77E D73> F;?7 63F3 EK@5:DA@;L3F;A@ E75GD7 EFAD397 3@6
788;5;7@FBDA57EE;@9 !FEG@;CG7!@97EFADFAA>FD3@E8AD?EE7>75F76J(!63F3BA;@FE;@FA3 D7>3F;A@3> F34>7 8AD?3F 4A>EF7D;@9 EKEF7? 8G@5F;A@3>;FK  ,:7 EG557EE8G> >3G@5: A8$!*E7DH7E3E3F7EF3?7@FFAF:7BAF7@F;3>A8!;@7@:3@5;@9B7DEA@3>;L76>73D@;@9
7JB7D;7@57E;?BDAH;@963F3E75GD;FK3@64A>EF7D;@9EKEF7?788;5;7@5K &7H7DF:7>7EE
F:7B3B7D7?B:3E;L7EF:7@7768ADA@9A;@9D7E73D5:EB75;8;53>>K5A@57D@;@9BD;H35K
BD7E7DH;@9 ?75:3@;E?E 3@6 788;5;7@F ?3@397?7@F A8 63F3 :7F7DA97@7;FK  !F
67?A@EFD3F7E$!*EB;HAF3>DA>7;@F:7D3B;6>K7HA>H;@9>3@6E53B7A87$73D@;@93@6
;FE BAF7@F;3> 8AD 8GFGD7 36H3@57?7@FE E53>34;>;FK 3@6 363BF34;>;FK E7FF;@9 3 @7IBD75767@F8AD! BAI7D767$73D@;@9D75A??7@63F;A@EKEF7?E %54-/$0 DF;8;5;3>;@F7>>;97@57!! (AI7D76D75A??7@67DEKEF7?JB>3;@34>7
!7 >73D@;@9$!*0!,,1/-$2#1(-,!@ 3@ 7D3 ?3D=76 4K F75:@A>A9;53> ;@F79D3F;A@ 76G53F;A@ :3E I;F@7EE76 5A@E;67D34>7
FD3@E8AD?3F;A@E ,:77?7D97@FD73>?EA8$73D@;@9%3@397?7@F+KEF7?E$%+7$73D@;@9
B>3F8AD?E 3@6 DF;8;5;3> !@F7>>;97@57 ! 7@:3@576 D75A??7@63F;A@ EKEF7?E :3H7
E;9@;8;53@F>K3G9?7@F7676G53F;A@3>7JB7D;7@57E>7H7@7F3>   ,:7E736H3@57?7@FE
67?A@EFD3F7 BAF7@F;3> ;@ B7DEA@3>;L;@9 >73D@;@9 7JB7D;7@57E ABF;?;L;@9 63F3 E75GD;FK 3@6
3G9?7@F;@9 EKEF7? 788;5;7@5K  AI7H7D 67EB;F7 F:7E7 93;@E A4EF35>7E B7DE;EF EG5: 3E
?3@39;@9 63F3 :7F7DA97@7;FK BD7E7DH;@9 63F3 BD;H35K 3@6 ?3;@F3;@;@9 EKEF7? E53>34;>;FK
A;FE:I3D7>A ,:;EBD7E7@FE35A@F;@G76@7768AD;@@AH3F;H73BBDA35:7E !@ F:;E B3B7D I7 BDABAE7 F:7 0!, 7@FD3$ DF;8;5;3> !@F7>>;97@57 *75A??7@67D$!* 3 57@FD3>;L76 ! BAI7D76 D75A??7@63F;A@ EKEF7? F:3F 8G@5F;A@E I;F:;@ 3 ?G>F;@A670!,6G53F;A@3>70B>3;@34>7!,AA>EEKEF7?>3@393@7F3>    $!*;E
67E;9@76FAF35=>7F:7=7K5:3>>7@97E3EEA5;3F76I;F:!D75A??7@67DEKEF7?E;@7$73D@;@9A887D;@93@AH7>B3F:I3KFAI3D6EB7DEA@3>;L76>73D@;@9 ,:7 $!* EKEF7? EF3@6E 3F F:7 ;@F7DE75F;A@ A8 >3D97 E53>7 63F3 3@3>KE;E 3@6

422

B7DEA@3>;L7676G53F;A@ !F53@?3@3973@63@3>KL73H3EF3DD3KA863F335DAEEH3D;AGE$%+
B>3F8AD?E 8GD@;E:;@9 B7DEA@3>;L76 788;5;7@F 3@6 BD75;E7 D75A??7@63F;A@E FA >73D@7DE $!*BD;AD;F;L7E63F3E75GD;FK3@6BD;H35K5DG5;3>5A@E;67D3F;A@E;@F:79>A43>76G53F;A@3>
E75FAD(A>A@7FE=K,7@7 /;<7D3F:@77F3>   ,:7 B3B7D ;E ?7F:A6;53>>K EFDG5FGD76 FA BDAH;67 3@ 7JF7@E;H7 G@67DEF3@6;@9 A8 F:7$!* EKEF7? 5A??7@5;@9 I;F: 3 D7H;7I A8 D7>3F76 IAD=E ;@ F:7 8;7>6 A8 ! 7@:3@576
D75A??7@63F;A@ EKEF7?E  ,:;E ;E EG5577676 4K 3 5A?BD7:7@E;H7 EKEF7? AH7DH;7I 3@
7JB>3@3F;A@A8F:763F3B;B7>;@73@6BDA57EE;@9?75:3@;E?E3@6367F3;>76355AG@FA8F:7
;?B>7?7@F3F;A@ 3@6 ;@F79D3F;A@ A8 $!*  ,:DAG9:AGF F:7 B3B7D I7 G@67DE5AD7 :AI$!*EGDB3EE7EFD36;F;A@3>EKEF7?5:3>>7@97E3@63@F;5;B3F7E8GFGD736H3@57?7@FE;@F:7
D3B;6>K7HA>H;@96A?3;@A87$73D@;@9  %*!1%$-/)0
,:7 ;@F7DE75F;A@ A8 $73D@;@9 %3@397?7@F +KEF7?E $%+ 7$73D@;@9 3@6 DF;8;5;3>
!@F7>>;97@57 ! :3E 477@ 3 8A53> BA;@F A8 D7E73D5: ;@ D757@F K73DE 'GK3@9 7F 3>   
B3DF;5G>3D>K 5A@57D@;@9 F:7 67H7>AB?7@F A8 D75A??7@63F;A@ EKEF7?E  '@7 A8 F:7 73D>;7EF
EFD;67E ;@ F:;E EB357 I3E F:7 36ABF;A@ A8 5A>>34AD3F;H7 8;>F7D;@9 3@6 5A@F7@F 43E76
D75A??7@63F;A@ 3>9AD;F:?E #AD7@   $G 7F 3>    %3D>;@    AD ;@EF3@57
;?B>7?7@F763B7DEA@3>;L76D75A??7@63F;A@EKEF7?;@7$73D@;@9B>3F8AD?EGE;@935A@F7@F $3>;F:37F3> $3>;F:3+D77<3  43E763BBDA35:I:;5:5A@E;67D76F:7>73D@7DEBDA8;>7E
3@6 F:7;D >73D@;@9 EFK>7E  7EB;F7 F:7 ;@;F;3> EG557EE FD36;F;A@3> ?7F:A6E >;=7 F:7E7 A8F7@
7@5AG@F7D76 5:3>>7@97E EG5: 3E F:7 5A>6 EF3DF BDA4>7? 3@6 E53>34;>;FK ;EEG7E B3DF;5G>3D>K
I:7@673>;@9I;F:3H3EF@G?47DA8GE7DE3@65A@F7@FE7>>#AD7@ $3?7F3>  $;=37F3>   ,A AH7D5A?7 F:7E7 :GD6>7E EG4E7CG7@F D7E73D5: 4793@ FA ;@5ADBAD3F7 ?35:;@7
>73D@;@9F75:@;CG7E;@FAF:767E;9@A8D75A??7@63F;A@EKEF7?E EFG6K5A@6G5F764K G3@9
7F3>  G3@97F3>  7J7?B>;8;7EF:;EFD3@E;F;A@ ,:7KBDABAE763:K4D;6D75A??7@63F;A@
EKEF7?F:3F5A?4;@76F:7EFD7@9F:EA85A>>34AD3F;H78;>F7D;@93@6677B>73D@;@9F75:@;CG7E ,:7EKEF7?I3E4G;>FA@3@GFA7@5A67D3@7GD3>@7FIAD=?A67>F:3F;?BDAH76F:7355GD35K
A8 D75A??7@63F;A@E 4K >73D@;@9 5A?B>7J @A@ >;@73D GE7D ;F7? ;@F7D35F;A@E  ,:;E EFG6K
D7BD7E7@F763EG4EF3@F;3>EF7B8ADI3D68DA?FD36;F;A@3>?7F:A6E67?A@EFD3F;@9F:7BAF7@F;3>
A8 ?35:;@7 >73D@;@9 ;@ 7@:3@5;@9 D75A??7@63F;A@ EKEF7?E 3; 7F 3>    77> 7F 3>   
&G?7DAGE76G53F;A@3>;@EF;FGF;A@EFD36;F;A@3>>K7J:;4;F3D7>G5F3@57FA6;E5>AE7EFG67@F
>73D@;@963F3FA7JF7D@3>7@F;F;7E3F7@67@5K9DAG@676;@5A@57D@ED793D6;@9BD;H35K3@663F3
E75GD;FK  AI7H7D F:7 8G@5F;A@3>;FK 3@6 78875F;H7@7EE A8 D75A??7@67D EKEF7?E 3D7;@7JFD;534>K >;@=76 FA F:7;D 3557EE FA7JF7@E;H763F3E7FE F:7D74K 5D73F;@93 5A@G@6DG? E
EG5:F:7D7;E3BD7EE;@9@776FAEFD;=7367>;53F77CG;>;4D;G?47FI77@63F3GF;>;FKF:3F;EF:7
ABF;?;L3F;A@ A8 D75A??7@67D EKEF7?E F:DAG9: 5A?BD7:7@E;H7 63F3 3557EE 3@6 F:7
BD7E7DH3F;A@A863F3BD;H35K7@EGD;@9F:7BDAF75F;A@A8E7@E;F;H7;@8AD?3F;A@ ,:;E43>3@57;E
B3D3?AG@FFA:3D@7EEF:78G>>BAF7@F;3>A8EG5:EKEF7?EI:;>7EF;>>36:7D;@9FA7F:;53>9G;67>;@7E
3@6>793>D79G>3F;A@E#A4E3 (A>A@7FE=K,7@7  '5:7<37F3>EIAD=;@FDA6G57E
F:7 >A5=5:3;@ A8$73D@;@9$A9E '$$ 3 B>3F8AD? F:3F 5A?BD7:7@E;H7>K D75AD6E >73D@;@9
B3F:I3KE3@635:;7H7?7@FE47KA@6F:7E5AB7A8FD36;F;A@3>FD3@E5D;BFE ,:;EEKEF7?5A>>75FE
63F38DA?6;EFD;4GF76>73D@;@9EKEF7?E3@6366D7EE7EF:75A>6 EF3DF;EEG7;@>73D@;@963F3
3@3>KF;5B>3F8AD?E4K7@34>;@9 F:7 E75GD73@6H7D;8;34>7 FD3@E87D A8 >73D@;@9 D75AD6E35DAEE;@EF;FGF;A@E  DG5;3>>K '$$ ?3;@F3;@E367>;53F743>3@5747FI77@63F3 BD;H35KE7@E;F;H;FK
3@6F75:@A>A9KG@67DE5AD;@9;FE5A??;F?7@FFAD7EB75F;@9;@6;H;6G3>BD;H35K'5:7<37F3>   
!@ F:7 BD7>;?;@3DK B:3E7 F:7 BDABAE76 3DF;8;5;3> ;@F7>>;97@57 ! 6D;H7@
D75A??7@63F;A@EKEF7?:3E477@GF;>;L768ADF735:;@9@9>;E:3E3AD7;9@$3@9G397$FA:;9: E5:AA> EFG67@FE ,3=;;7F 3>     GDF:7D?AD7 3@! 7@:3@576 D75A??7@63F;A@
EKEF7?I;F:E7>8 7JB>3@3FADK53B34;>;F;7E:3E477@GE768AD?3F:7?3F;5E;@EFDG5F;A@3;7F
3>  >3@393@7F3>    

423

,:7E7 36H3@57?7@FE G@67DE5AD7 F:7 BAF7@F;3> A8 57@FD3>;L76 ! BAI7D76
D75A??7@63F;A@EKEF7?E;@7@:3@5;@9B7DEA@3>;L76>73D@;@9;?BDAH;@963F3E75GD;FK3@6
;@5D73E;@9EKEF7?788;5;7@5K  AI7H7DF:7K3>EA:;9:>;9:FF:7@7768AD366;F;A@3>D7E73D5:;@
3D73E>;=7BD;H35K BD7E7DH;@9?75:3@;E?E3@6?3@39;@963F3:7F7DA97@7;FK

*(41&'H7DH;7IA8F:7BDABAE76EKEF7?

 501%+3%/3(%4,:7 ;?B>7?7@F3F;A@ A8 ! D75A??7@67D EKEF7?E I:;5: ;@:7D7@F>K D7CG;D7 EG4EF3@F;3>
3?AG@FEA863F38AD788;5;7@FAB7D3F;A@;EF:7D74K38875F76 !F;EB7DF;@7@FFAG@67DEF3@6F:3F
F:7CG3>;FKA8F:7E7EKEF7?E;E6;D75F>KBDABADF;A@3>FAF:7CG3@F;FKA863F33H3;>34>7:7@573
>3D97D63F3E7F;E3>I3KE36H3@F397AGE!@D7EBA@E7FAF:;E5:3>>7@97I7BDABAE73EKEF7?F:3F3?3>93?3F7EF:7D7CG;E;F7
63F38DA?F:7E76;EFD;4GF76EKEF7?E;@FA3E;@9G>3D57@FD3>;L76EKEF7?3EE:AI@;@;9GD7 ,:7BDABAE76EKEF7?5A@F3;@EFIA?3;@B3DFE+7FEA8?G>F;B>7$EKEF7?E$78FE;673@6
F:7 57@FD3> ! D75A??7@67D D;9:F E;67 I;F: (! 5A@@75F;A@E 835;>;F3F;@9 5A??G@;53F;A@
47FI77@F:7FIAEKEF7?E ,:;EEKEF7?F:7@BDA57EE7EF:7355G?G>3F7663F3F:DAG9:F:7!
D75A??7@67DEKEF7? ,:7BDA57EE76;@8AD?3F;A@;@FGD@;E5:3@@7>76435=FAF:7D7EB75F;H7$EKEF7? F:7D74K 7@EGD;@93 8776435= ?75:3@;E? F:3F7@:3@57EF:7 AH7D3>> >73D@;@9
7JB7D;7@57$+KEF7?

*(41&'H7DH;7IA8F:7$EKEF7?
,:7 $73D@;@9 H;67@57 @3>KF;5E D3?7IAD= $ ;@8D3EFDG5FGD7 E:AI@ ;@ ;9GD7
>3@393@  '93F3   ;E 3@ ;@FD;53F7 5A?BAE;F7 A8 E7H7D3> G@;CG7 5A?BA@7@FE
BDA?;@7@F>K;@5>G6;@93@ $%+36;9;F3>>73D@;@9?3F7D;3>D736;@9EKEF7?EAA=DA>>3>73D@;@9
3@3>KE;EFAA>$A9B3>7FF73@63$73D@;@9*75AD6+FAD7$*+ $*+;E3@AB7@ EAGD57FAA>8AD
?3@39;@9 >73D@;@9 63F3 >7H7D39;@9 F:7 E53>34>7 3@6 :;9: B7D8AD?3@57 53B34;>;F;7E A8
%A@9A8AD63F3EFAD3973@6D7FD;7H3>3@6EGBBADF;@963F33@3>KE;E3@6;@E;9:FE97@7D3F;A@ ,:7$%+3E;?B>;764K;FE@A?7@5>3FGD7AD5:7EFD3F7EF:7B>3@@;@97J75GF;A@3@67H3>G3F;A@424

A8F:75A?BD7:7@E;H7>73D@;@9BDA57EE %73@I:;>7AA=DA>>8G@5F;A@E3E3H7DE3F;>77 4AA=
D7367D  !@FD;9G;@9>K 3>> F:7E7 EKEF7?E 7?;F JB7D;7@57 (! J(! 63F3 I:;5: 3D7
EG4E7CG7@F>K EFAD76 I;F:;@ F:7 $*+  ,:7 BDABAE76 EKEF7? :3E ?G>F;B>7 $ @A67E
5A@@75F76FAF:77@FD3>!*75A??7@67D   7@FD3>!*75A??7@67D,:7 7@FD3> ! *75A??7@67D 3E E:AI@ ;@ ;9GD7  ;E 3 5A?B>7J 5A@EFDG5F 5A@E;EF;@9 A8
@G?7DAGE=7K5A?BA@7@FEEG5:3EF:7*75A??7@67D%A67>,D3;@7DF:7$*+3F343E7F:7
$( 3F343E7 F:7 * 3F343E7 F:7 ! *75A??7@67D 3F343E7 F:7 !@97EFAD 3@6 F:7 !
*75A??7@67D 3BB>;53F;A@  3F3:3DH7EF76 8DA?F:76;EFD;4GF76 $ EKEF7?E F:DAG9: F:7
9D7EE (! ;E 5A@EA>;63F76 I;F:;@ F:7 $*+ $( 3@6 * 63F343E7E  ,:;E EFAD76 63F3 ;E
EG4E7CG7@F>K>7H7D39764KF:7*75A??7@67D%A67>,D3;@7DFAFD3;@F:7?A67>I:;5:;EF:7@
67BAE;F76 ;@ F:7 ! *75A??7@67D 3F343E7  ,:7 !@97EFAD E7DH7E 3E 3 63F3 FD3@E8AD?7D
5A@H7DF;@9 $*+ %A@9A 63F3 ;@FA D7>3F;A@3> 63F3 I:;5: ;E F:7@ EFAD76 ;@ F:7 !
*75A??7@67D3F343E7 ;@3>>KF:7!*75A??7@67D3BB>;53F;A@GF;>;L7EF:;E63F3FABDAH;67
D75A??7@63F;A@EF:DAG9:F:7#A63?3(! 

!1!(.%*(,%!,$/-#%00(,',:7$EKEF7?7@5A?B3EE7EFIAFKB7EA863F3D7>3F;A@3>3@6@A@ D7>3F;A@3> *7>3F;A@3>
63F33D7 ?3@3976 F:DAG9: F:7 %K+)$ 63F343E7 3@6 3D7 BD;?3D;>KGF;>;L76 I;F:;@F:7$%+
AA=DA>> 3@6 $A9 (3>7FF7  !@ 5A@FD3EF @A@ D7>3F;A@3> 63F3 ?3;@>K 5:3D35F7D;L76 4K J(!EF3F7?7@FE3D797@7D3F768DA?7H7DK;@F7D35F;A@I;F:;@F:7E7EKEF7?E3@63D7EG4E7CG7@F>K
EFAD76 ;@ %A@9A  ,:7 BDABAE76 EKEF7? ?AD7AH7D 873FGD7E ?G>F;B>7 $ @A67E
6;EFD;4GF7635DAEEH3D;AGE76G53F;A@3>;@EF;FGF;A@E ,:7JB7D;7@57(!,;@3@(! ;@;F;3>>K;@FDA6G576;@ A@L3>7L:GD5:;>>  E7DH7E 3E 3 5DG5;3> 5A?BA@7@F A8 >73D@;@9 3@3>KF;5E  !F A887DE 3 B>3F8AD? 39@AEF;5
8AD?3>;E? 8AD 53BFGD;@9 7H7@FE F:3F FD3@EB;D7 F:DAG9:AGF 3@K >73D@;@9 7JB7D;7@57  ,:7
BDABAE76 EKEF7? 53B;F3>;L7E A@ F:7 53B34;>;F;7E A8 J(! FA D75AD6 3>> >73D@7D EKEF7?
;@F7D35F;A@E ,:797@7D3F76J(!EF3F7?7@FE3D75A??;FF76FA3$73D@;@9*75AD6+FAD7$*+
I:;5:;E3=7K7>7?7@F;@F:7;@8D3EFDG5FGD78ADEFAD;@93@6D7FD;7H;@9>73D@;@9D75AD6E ,:;E
5A?4;@3F;A@ A8 D7>3F;A@3> 3@6 @A@ D7>3F;A@3> 63F3 EFAD397 ?75:3@;E?E 5AGB>76 I;F: F:736ABF;A@A8J(!8AD7H7@FFD35=;@97@EGD7E5A?BD7:7@E;H763F35AH7D3973@68AEF7DEF:7
BAF7@F;3>8ADD;5:3@3>KF;53>;@E;9:FE ADF:7BGDBAE7A87@:3@5;@9F:7$!*EKEF7?;F;E;?B7D3F;H7F:3FB7DF;@7@F63F3
;EA4F3;@768DA?H3D;AGE$63F3EFAD7EI:;5:I;>>EG4E7CG7@F>K47GF;>;L76;@FD3;@;@9;FE
?A67>E ,:;EBDA57EE;E7EE7@F;3>FA3HA;65A@97EF;A@3@6FA?3;@F3;@3=77@8A5GEA@F:7
?;5DAE7DH;57E3D5:;F75FGD7 !@>;9:FA8F:;E3EKEF7?:3E477@67H;E76F:3F;@5ADBAD3F7EF:7
$3F3$3K7D$$ ,:;EEKEF7?5A?BD;E7EFIABB>;53F;A@(DA9D3??;@9!@F7D8357E(!
7@6BA;@FE@3?769D7EE3@6#A63?3I;F:F:7>3F7D?73@;@975:A;@"3B3@7E7 ,:7E7
67E;9@3F76 7@6BA;@FE 3D7 ;@EFDG?7@F3> ;@ 835;>;F3F;@9 E73?>7EE 63F3 7J5:3@97 47FI77@ F:7$EKEF7?E3@6F:7$!* 
A@E7CG7@F>KF:7$!*EKEF7?:3E477@BDAH;E;A@76I;F:F:734;>;FKFAEG4E5D;47
FA F:7 $*+ %A@9A 63F343E7E A8 F:7 $ EKEF7?E  ,:;E EG4E5D;BF;A@ AB7D3F;A@ ;E
35FG3>;L764K>7H7D39;@9%A@9A5:3@97EFD73?E3873FGD7F:3F835;>;F3F7ED73> F;?763F3
BDA57EE;@9 ,:;E?75:3@;E?7?BAI7DEF:7$!*EKEF7?FA93D@7DD73> F;?763F38DA?F:7
$*+%A@9A3@6;@FGD@D7B>;53F7F:;E63F3;@FA;FEAI@$*+%A@9AF:7D74K7@EGD;@9
35A@F;@GAGE8>AI3@6EK@5:DA@;L3F;A@A863F347FI77@EKEF7?E %AD7AH7D F:7 $!* EKEF7? ;E 67E;9@76 FA7JFD35F D7>3F;A@3> 63F3 8DA? F:7 $
EKEF7?EF:DAG9:F:7$$*++(! ,:77JFD35F7663F3;EF:7@FD3@E87DD763@6E75GD7>K
EFAD76I;F:;@F:7$!*EAI@63F3D7BAE;FAD;7E ,:;E?7F:A6;53>BDA57EE7@EGD7EF:3FF:7
M$!* EKEF7? 5A@F;@GAGE>K :3E 3557EE FA GB63F76 3@6 D7>7H3@F 63F3 @757EE3DK 8AD ;FE1

https://xapi.com/

425

ABF;?3>AB7D3F;A@3@6F:735:;7H7?7@FA8;FE;@F7@676A4<75F;H7E ,:7 $!* @757EE;F3F7E D7>3F;A@3> 63F3 8AD F:7 5A?BD7:7@E;H7 8G@5F;A@3>;FK 3@6
873FGD7E;FA887DE ,A35:;7H7F:;EE7>75F7663F3BA;@FE8DA?F:7D7B>;53F76$*+%A@9A63F3
3D7FD3@E8AD?76;@FA3D7>3F;A@3>F34>78AD?3FGE;@9F:7!@97EFADFAA> ,:7!@97EFADEKEF7?;E
5A?BD;E76 A8 FIA ?3;@ 5A?BA@7@FE D73> F;?7 63F3 5A@H7DE;A@ 3@6 435=8;>>;@9 63F3  !@
E57@3D;AE;@HA>H;@9EKEF7?83;>GD7ADF:7;@5ADBAD3F;A@A8@AH7>J(!63F3BA;@FEF:7435=8;>>;@9
3BB>;53F;A@53@47?3@G3>>K7J75GF76FA835;>;F3F7F:7BDAB393F;A@A8D7>3F;A@3>63F38DA?F:7
$*+%A@9A +.*%+%,1!1(-,!,$,1%'/!1(-,GD;@9F:7;?B>7?7@F3F;A@B:3E7AGD8A5GEI3EA@;@F79D3F;@9F:7$!*EKEF7?I;F:F:7
7J;EF;@9$;@8D3EFDG5FGD7 ,:;E;@F79D3F;A@I:;5:I3E3;?763F835;>;F3F;@9E73?>7EE63F3
FD3@E?;EE;A@ EK@5:DA@;L3F;A@ 3@6 BDA57EE;@9 I:;>7 3>EA 9G3D3@F77;@9 63F3 E75GD;FK 3@6
BD;H35K@757EE;F3F76?7F;5G>AGEB>3@@;@93@6EFD;@97@F7J75GF;A@ ,:7$!*EKEF7?;E43E76A@3?;5DAE7DH;57E67E;9@I:;5:3>>AIE8ADH7DE3F;>;FK
E53>34;>;FK3@6E;?B>;5;FKA8GB63F;@9 !FI3E5D73F763@64G;>F?AEF>KI;F:(KF:A@3H7DE3F;>7
3@6 I;67>K GE76 BDA9D3??;@9 >3@9G397 8AD ! 3BB>;53F;A@E 3@6 F:7 +5;=;F $73D@ ?A6G>7
I:;5:BDAH;67E3I;67D3@97A8?35:;@7>73D@;@9F75:@;CG7E8ADF:7D75A??7@63F;A@ +7FF;@9GBF:79D7EE3@6#A63?37@6BA;@FE8AD63F3;@F7D5:3@9747FI77@F:7$
EKEF7?3@6$!*I3EF:78;DEFEF7B;@;FE;?B>7?7@F3F;A@ ,:7E77@6BA;@FEI7D74G;>FI;F:
F:7 >3E= 8D3?7IAD= I:;5: 3>>AIE 8AD F:7 5D73F;A@ A8 *+,8G> (!E  A>>AI;@9 F:3F F:7$!* EKEF7? I3E E7F GBFA GE7 %A@9A 5:3@97 EFD73?E FA EG4E5D;47 FA F:7 $
EKEF7?E$*+%A@9A63F343E7E ,:;EEG4E5D;BF;A@835;>;F3F7ED73> F;?763F3BDA57EE;@93@6
EK@5:DA@;L3F;A@ 47FI77@ EKEF7?E  ,:7 D7>3F;A@3> 63F3 A4F3;@76 8DA? F:7 $ EKEF7?E
F:DAG9:F:7$$9D7EE(!I3E;@5ADBAD3F76;@FA$!*EAI@63F3D7BAE;FAD;7E /7GE76
(AEF9D7+)$ 3 DA4GEF 3@6 E75GD7 D7>3F;A@3> 63F343E7 EKEF7? FA EFAD7 F:;E 63F3  +75GD;FK
?73EGD7E I7D7BGF ;@ B>357FABDAF75F F:763F3;@5>G6;@97@5DKBF;A@ 3F D7EF 3@6 ;@ FD3@E;F
D79G>3D435=GBE3@6EFD;5F3557EE5A@FDA>E ,:7!@97EFADFAA>I3EF:7@67H7>AB76FA5A@H7DFF:7E7>75F76J(!63F3BA;@FE8DA?
F:7D7B>;53F76$*+%A@9A;@FA3D7>3F;A@3>F34>7EFDG5FGD7 ,:7F75:@A>A9K;E5A?BD;E;@9
FIA ?3<AD 5A?BA@7@FE D73> F;?7 63F3 EK@5:DA@;L3F;A@ 3@6 435=8;>>;@9  ,:7 D73> F;?7 63F3
EK@5:DA@;L3F;A@5A?BA@7@F;EID;FF7@;@(KF:A@ '@F:7AF:7D:3@6F:7435=8;>>;@95A?BA@7@F
D7EBA@E;4>78AD8;>>;@9:;EFAD;53>63F393BEI3E67H7>AB76GE;@9A>3@96G7FA;FE788;5;7@5K
3@6B7D8AD?3@5736H3@F397E;@:3@6>;@9>3D9763F3E7FE ,A97F:7DF:7E75A?BA@7@FE?3=7
!@97EFAD3H7DE3F;>7FAA>8AD63F35A@H7DE;A@3@6EK@5:DA@;L3F;A@ 
-BA@ F:7 EG557EE8G> 7EF34>;E:?7@F A8 F:7 63F3 B;B7>;@7 F:7 5A??7@57?7@F A8 F:7?35:;@7>73D@;@9?A67>FD3;@;@9B:3E78ADF:7$!*EKEF7?I3E;@;F;3F76 ,:;E;@F7@E;H7
BDA57EEI3EG@67DF3=7@A@:;9: B7D8AD?3@575A?BGF;@9EKEF7?EAI;@9FAF:75A@E;67D34>7
3?AG@F A8 63F3 ;@HA>H76  '@7 D7BD7E7@F3F;H7 7J3?B>7 A8 F:7 ?35:;@7 >73D@;@9 ?A67>E
7?B>AK76 I;F:;@ F:7 $!* EKEF7? ;E F:7 3K7E;3@#@AI>7697 ,D35;@9 #,3(KF:A@
43E76;?B>7?7@F3F;A@A8F:7#,?A67> ,A7@EGD7GB FA 63F7;@F79D3F;A@A8@7I>73D@;@9>A9
63F3F:7?A67>;ED78D7E:763FD79G>3D;@F7DH3>EEB75;8;53>>K7H7DK ?;@GF7E;8@7I63F3:3E
477@67F75F764KF:7EKEF7?36D;@3F:7F3>    35:?A67>F:3F;EFD3;@76;EF:7@EFAD76
;@F:7*75A??7@67D3>A@9I;F:3F;?7EF3?BE:AI;@9I:7@F:7?A67>5AG>68;DEF47GE76
FA9;H7D75A??7@6E E;F;E5A@F;@GAGE>K47;@9GB63F76:;EFAD;53>H7DE;A@E@3BE:AFEA8F:7
?A67>3D7EFAD763@653@47GE764KD7E73D5:7DEFAD7B>3KADD75D73F7D75A??7@63F;A@EA8
B3DF;5G>3DEFG67@FE3F3BA;@F;@F;?7 ,:;E53@47GE78G>3>EA;83D7E73D5:@776EFA;@H7EF;93F7
;@ 67F3;> I:K 3@6 :AI D75A??7@63F;A@E I7D7 97@7D3F76 BD7H;AGE>K 4K F:7 ?A67> EA F:3F
7JB>3@3F;A@E 53@ 47 5:75=76 3@6 H3>;63F76  !F 3>EA 3>>AIE 67H7>AB7DE 3@6 D7E73D5:7DE FA
7J3?;@7BDA4>7?EF:3F:3H7A55GDD76;@F:7B3EF ;@3>>KI:7@3>73D@7DGE7EF:7D75A??7@63F;A@B3@7>A@3E3F7>>;F7$EKEF7?
F:;E E7@6E 3 D7CG7EF H;3 #A63?3 FA F:7 D75A??7@67D ?A6G>7 ;@ F:7 $!* EKEF7? 2https://scikit-learn.org/

426

(7DEA@3>;L76D75A??7@63F;A@E3@67JB>3@3F;A@E3D7F:7@97@7D3F7643E76A@F:763F33@6
?A67>E EFAD76 ;@ F:7 57@FD3> 63F343E7 D7>3K76 435= FA F:7 E3F7>>;F7 $ EKEF7? 3@6
6;EB>3K76FAF:7>73D@7D ,:78776435=?75:3@;E?7@EGD76F:3FF:7D75A??7@63F;A@EI7D7
5A@F;@GAGE>K ;?BDAH76 43E76 A@ F:7 >73D@7DE ;@F7D35F;A@E I;F: F:7 EKEF7?  >EA 3E F:7
D75A??7@63F;A@?A67>;E57@FD3>>K>A53F76;F7@34>7ED7E73D5:7DE3@667H7>AB7DEFAGB63F7
3@667B>AK@7I873FGD7E3@6F3D97F76;@F7DH7@F;A@E8AD7H3>G3F;A@I;F:788;5;7@5K;@EF736A8
:3H;@9FAGB63F7?G>F;B>7E3F7>>;F7$EKEF7?E !@EG??3DKF:7;?B>7?7@F3F;A@3@6;@F79D3F;A@A8F:7$!*EKEF7?;@FAF:7$;@8D3EFDG5FGD7 I3E 3 5A?B>7J K7F D7I3D6;@9 BDA57EE  ,:7 EG557EE8G> ;@F79D3F;A@ :3E F:7
BAF7@F;3> FA FD3@E8AD? F:7 >73D@;@9 7JB7D;7@57 4K BDAH;6;@9 B7DEA@3>;L76 788;5;7@F 3@6
7@939;@9! 6D;H7@D75A??7@63F;A@E ,:7?A6G>3D;FKA8F:7EKEF7?3>EAA887DEABBADFG@;F;7E
8AD8GFGD77@:3@57?7@FE3@6E53>34;>;FK3EF:7@7763D;E7E  -,#*20(-,!,$212/%-/),:7 67H7>AB?7@F ;?B>7?7@F3F;A@3@6 ;@F79D3F;A@ A8 F:7$!* EKEF7? I;F: F:7 7J;EF;@9$;@8D3EFDG5FGD7:3E67?A@EFD3F76F:7BAI7DA85A?4;@;@9!?35:;@7>73D@;@93@663F3
BDA57EE;@9 F75:@A>A9;7E FA FD3@E8AD? F:7 >73D@;@9 7JB7D;7@57  ,:7 3D5:;F75FGD7 43E76 A@
?;5DAE7DH;57E BDAH;67E 3@ 788;5;7@F 3@6 E53>34>7 EA>GF;A@ I:;>7 F:7 GF;>;L3F;A@ A8 :;9:
B7D8AD?3@575A?BGF;@9EKEF7?E3@636H3@576?35:;@7>73D@;@9?A67>E>;=7#,835;>;F3F7E
F:7BDA57EE;@9A8H3EF3?AG@FEA863F3FA97@7D3F7B7DEA@3>;L76>73D@;@9D75A??7@63F;A@E ,:7;?B>7?7@F3F;A@A8$!*:3EEG557EE8G>>KF35=>76E7H7D3>5:3>>7@97E;@5>G6;@9
E73?>7EE63F3FD3@E?;EE;A@D73> F;?763F3EK@5:DA@;L3F;A@3@6EFD;@97@FE75GD;FK?73EGD7E
8AD63F3BD;H35K GDF:7D?AD7;F:3EG@67DE5AD76F:778875F;H7@7EEA8(KF:A@8AD67H7>AB;@9!3BB>;53F;A@EF:7788;5;7@5KA8>3E=8D3?7IAD=;@5D73F;@9*+,8G>(!E3@6F:736H3@F397
A8A>3@9;@:3@6>;@9>3D9763F3E7FE ,:7 GE7 A8 F:7 !@97EFAD FAA> 8AD D73> F;?7 63F3 EK@5:DA@;L3F;A@ 3@6 435=8;>>;@9 A8
:;EFAD;53>63F33>A@9I;F:F:78D7CG7@FD78D7E:;@9A8F:7?35:;@7>73D@;@9?A67>7@EGD7EF:3F
F:7 >73D@;@9 D75A??7@63F;A@E 3D7 3>I3KE 43E76 A@ F:7 ?AEF 5GDD7@F 63F3  ,:7 8776435=
?75:3@;E? ;@F79D3F76 ;@FA F:7 $ EKEF7? 3>>AIE 8AD 5A@F;@GAGE ;?BDAH7?7@F ;@ F:7
D75A??7@63F;A@CG3>;FK7@EGD;@93?AD77@939;@93@6B7DEA@3>;L76>73D@;@97JB7D;7@57 GFGD7IAD=A@F:7$!*EKEF7?5AG>68GDF:7D7JF7@6;FE53B34;>;F;7E3@6E53>34;>;FK (AF7@F;3> 6;D75F;A@E ;@5>G67 7@:3@5;@9 F:7 ?35:;@7 >73D@;@9 ?A67>E FA :3@6>7 7H7@ ?AD7
5A?B>7J >73D@;@9 E57@3D;AE 3@6 ;@5ADBAD3F;@9 366;F;A@3> 8AD?E A8 >73D@;@9 63F3 EG5: 3E
4;A?7FD;5AD BEK5:A?7FD;5 63F3 FA D78;@7 F:7 B7DEA@3>;L3F;A@A8 >73D@;@9 D75A??7@63F;A@E ,:7D7?3K3>EA47BAF7@F;3>8AD;@F79D3F;A@I;F:AF:7D$%+EAD>73D@;@9B>3F8AD?EFA4DA367@
;FE;?B35F !@EG??3DKF:7$!*EKEF7?F:DAG9:;FE;@@AH3F;H7GE7A8!3@6?35:;@7>73D@;@9
F75:@A>A9;7EBDA?;E7EFA473FD3@E8AD?3F;H7FAA>;@F:7>3@6E53B7A8B7DEA@3>;L76>73D@;@9 !FE EG557EE8G> ;@F79D3F;A@ I;F: F:7 $ EKEF7? 67?A@EFD3F7E ;FE BAF7@F;3> @AF <GEF 8AD;?BDAH;@9 F:7 >73D@7DE 7JB7D;7@57 4GF 3>EA 8AD 6D;H;@9 8ADI3D6 F:7 8;7>6 A8 ! BAI7D76
76G53F;A@3E3I:A>7 ,:78>7J;4>73@6E53>34>73D5:;F75FGD7A8F:7EKEF7?3>EA7@EGD7EF:3F
;F;EI7>> EG;F768AD8GFGD77@:3@57?7@FE3@6;@F79D3F;A@EBAF7@F;3>>K5D73F;@93@7H7@9D73F7D
;?B35FA@F:776G53F;A@E75FAD #),-4*%$'+%,10,:;EIAD=I3EB3DF>KEGBBADF764K"+(+D3@F ;@ ;68AD+5;7@F;8;5*7E73D5:"(
3@6"(JB>AD3FADK"( # "( 3@6&'"(&(






%&%/%,#%0

>7H7@. %5$3G9:>;@  >7@@*  #A76;@97D# *   !@EFDG5F;A@43E76A@363BF;H7
>73D@;@9F75:@A>A9;7E ".%#//+/'1&2&"1$)/.,&"1.*.(".%*.2314$3*/.  427

36D;@3F: /3@9 (3D6AE2    BK4=F@3557EE;4>7BKF:A@>;4D3DKA843K7E;3@=@AI>7697
FD35;@9?A67>E "1!*501&01*.3"1!*5   
3;0 /3@9% $77! 13@92 #A@90 0;3   +5;7@F;8;5B3B7DD75A??7@63F;A@
EGDH7K &&&$$&22   
77>" $3@97D+ 7@L?7:D% ;BB D7;F;@97D &ND@47D97D   *7E73D5:B3B7D
D75A??7@67D EKEF7? 7H3>G3F;A@ 3 CG3@F;F3F;H7 >;F7D3FGD7 EGDH7K  (DA5776;@9E A8 F:7;@F7D@3F;A@3>IAD=E:ABA@D7BDA6G5;4;>;FK3@6D7B>;53F;A@;@D75A??7@67DEKEF7?E7H3>G3F;A@7>> *  %   #AD7@ 1     +53>34>7 5A>>34AD3F;H7 8;>F7D;@9 I;F: <A;@F>K 67D;H76 @7;9:4AD:AA6
;@F7DBA>3F;A@I7;9:FE +7H7@F:!;@F7D@3F;A@3>5A@87D7@57A@63F3?;@;@9!% 
A;FE:I3D7>A     (DABAE;@9 3@ ;@F79D3F76 D7E73D5: 8D3?7IAD= 8AD 5A@@75F;H;E? -F;>;E;@9
F:7AD7F;53> EK@7D9;7E  .3&1."3*/.", &5*&6 /' &2&"1$) *. 0&. ".% *231*#43&% &"1.*.(   
3; 1  >3@393@   ,3=3?; #   '93F3     7E;9@ A8 3 -E7D !@F7DBD7F34>7 %3F: )G;L*75A??7@67D+KEF7?8AD"3B3@7E7 ;9:+5:AA>+FG67@FE (DA5776;@9EA8F:7F:/AD=E:AB
A@(D76;5F;@9(7D8AD?3@573E76A@F:7@3>KE;EA8*736;@97:3H;AD 
>3@393@ '93F3   $73D@;@93@3>KF;5EB>3F8AD?;@:;9:7D76G53F;A@;@"3B3@ ./6,&%(&"."(&-&.3&"1.*.(  
>3@393@  ,3=3?; # ,3=;; #  3;1  %3<G?63D *  '93F3      0!, +K?4;AF;5
JB>3@3F;A@ 6G53F;A@ +KEF7?  F: !@F7D@3F;A@3> A@87D7@57 A@ A?BGF7DE ;@ 6G53F;A@A@87D7@57(DA5776;@9EBB     
G3@92 :G@9/ '@9,:7@   9D3B: 43E76D75A??7@67DEKEF7?8AD6;9;F3>
>;4D3DK (DA5776;@9EA8F:7@6% ! +<A;@F5A@87D7@57A@;9;F3>>;4D3D;7E
#A4E3   (D;H35K 7@:3@576B7DEA@3>;L3F;A@ /--4.*$"3*/.2/'3)&   #AD7@1  G9GEF 35FAD;L3F;A@?77FEF:7@7;9:4AD:AA63?G>F;8357F765A>>34AD3F;H78;>F7D;@9?A67>  !@ (DA5776;@9E A8 F:7 F: % +!# ;@F7D@3F;A@3> 5A@87D7@57 A@ #@AI>7697
6;E5AH7DK3@663F3?;@;@9BB   $3>;F:3, +D77<3(    (7DEA@3>;E76E7>8 6;D75F76>73D@;@9D75A??7@63F;A@EKEF7? 1/$&%*"
/-043&1$*&.$&   $3? 0  &  .G ,  $7 ,     GA@9       "3@G3DK  66D7EE;@9 5A>6 EF3DF BDA4>7? ;@
D75A??7@63F;A@ EKEF7?E !@ (DA5776;@9E A8 F:7 @6 ;@F7D@3F;A@3> 5A@87D7@57 A@ -4;CG;FAGE
;@8AD?3F;A@?3@397?7@F3@65A??G@;53F;A@BB     $;=3 #A>A?H3FEAE#  36<;78F:K?;367E+   35;@9F:75A>6EF3DFBDA4>7?;@D75A??7@67DEKEF7?E 70&132823&-26*3)"00,*$"3*/.2     $G2 AG2 $;3@" 0;70 13@9)  "3@G3DK A@F7@F 43E765A>>34AD3F;H78;>F7D;@98AD
@7IEFAB;5D75A??7@63F;A@ !@(DA5776;@9EA8F:7,I7@FK &;@F:!A@87D7@57A@DF;8;5;3>
!@F7>>;97@57BB    %3D>;@   %     %A67>;@9 GE7D D3F;@9 BDA8;>7E 8AD 5A>>34AD3F;H7 8;>F7D;@9  %5".$&2 *. .&41",
*.'/1-"3*/.01/$&22*.(2823&-2 '5:7<3 (  >3@393@   -763 '93F3     %3@39;@9 >;87>A@9 >73D@;@9 D75AD6E F:DAG9:
4>A5=5:3;@ &2&"1$)".%1"$3*$&*.&$)./,/(8.)".$&%&"1.*.(    
'GK3@9 2:7@9$ ";3A(   DF;8;5;3>;@F7>>;97@57;@A@>;@7:;9:7D76G53F;A@EKEF7?3F;5
D7H;7I A8 7?B;D;53> D7E73D5: 8DA?  FA    %4$"3*/. ".% .'/1-"3*/. &$)./,/(*&2   (A>A@7FE=K" ,7@7'   (D;H35K3@64;963F3?3=;@97@6E?77F 3".&5.,*.&  ,3=;; #  >3@393@   $;  13@9 1   '93F3     JB>3;@34>7 @9>;E: %3F7D;3>
*75A??7@63F;A@ -E;@9 3@ !@8AD?3F;A@ *7FD;7H3> ,75:@;CG7 8AD $ $73D@;@9   F:
!@F7D@3F;A@3>A@87D7@57A@A?BGF7DE;@6G53F;A@A@87D7@57(DA5776;@9EBB   /;<7D3F:@7! %3E3=A' %AD;?GD31 +3=3;   &5&,/0-&.3/'"3""."(&-&.3
/13", '/1 .2314$3/12 ".%1/%4$3*/. &"-  EF ADG? A@ !@8AD?3F;A@,75:@A>A9K !, BB    428" !   !" " "   #" !
#" !"$  #" ! #"     
           

#!
%## !   !  

!"#)'!#*/-#$+-+),0/'*%*"#"'/0"'#.3+/+*'1#-.'/3,*
#*/#-$+-**+1/'1##.#-!&*""0!/'+*'*/!'#*!#*./'/0/#$+-' #-(-/.*"
!'#*!#.3+/+*'1#-.'/3,*
?385:1 9<51>G1 CCD ;I?D?E 13 :@   $/, 36*2*/(05 6- ,(8505. 6.9   9?9:,4 09 ( )36*2*/(05)(9,+73(:-684-68*655,*:05.3,(85,89,+;*(:065(38,*68+9-8644;3:073,9*/6639$/, 
9?9:,4*8,(:,9(7,84(5,5:8,*68+6-3,(85,8930-,365.3,(8505.(9044;:()3,/(9/,9
65:/,)36*2*/(05=/0*/*(5),(5(3?@,+:605-684:,(*/05.(5+3,(8505.$/097(7,8
78,9,5:9(;9()030:?(5(3?9096-:/, 9?9:,4;905.:/, (26)0,39,5,;809:0*9
=0:/ :=6 ;9,8 .86;79 9:;+,5:9 (5+ :,(*/,89 $/, 9:;+? ,<(3;(:,9 :/, ;9()030:? 6-
<(806;9 -,(:;8,9 05*3;+05. :/, ()030:? :6 <0,= 3,(85,8 +(:( -864 4;3:073, 9*/6639
4(5(., 7,840990659 <09;(30@, (5(3?:0*9 +,80<,+ -864 *655,*:,+ 3,(8505. 36.9 (5+
786<0+,(**,99:63,(8505.4(:,80(39;9,+(:<(806;99*/6639 ;8-05+05.9/0./30./::/,
9;**,99,9 6- :/,   9?9:,4 05*3;+05. 9:865. 7,8-684(5*, 05 (8,(9 9;*/ (9*65909:,5*? 8,(3=683+ 8,3,<(5*, (5+ ;9,8 *65:863 6=,<,8 3040:(:0659 =,8,
0+,5:0-0,+ 05 ,8868 /(5+305. (5+ :/, (<(03()030:? 6- *6478,/,590<, /,37 (5+
+6*;4,5:(:065&,*65*3;+,)?,47/(90@05.:/,5,,+-68-;:;8,=682:6(++8,99:/,9,
3040:(:0659 (5+ 9;..,9:9 76:,5:0(3 (<,5;,9 -68 04786<,4,5: <,8(33 :/09 8,9,(8*/
*65:80);:,9:6:/,+,<,3674,5:6-(;9,8-80,5+3?(5+780<(*?*659*06;973(:-684:/(:*(5-(*030:(:,30-,365.3,(8505.(5+,5/(5*,,+;*(:065(3+(:(9/(805.(5+(5(3?909    36*2*/(05 ,(8505. 6.9 %9()030:? ,;809:0* <(3;(:065
+;*(:065

+859>3B51C59>D8514?@D9?>?6D538>?<?7I9>D51389>71>4<51B>9>781CB5CE<D549>D85>554
6?B 1 C53EB5 1>4 5669395>D CICD5= D? 3?>>53D <51B>5BCL 54E31D9?>1< B53?B4C 13B?CC =E<D9@<5
C38??<C +85<?3;3819>?6%51B>9>7%?7C(%%CICD5=?665BC1@B?=9C9>7C?<ED9?>21C54
?>2<?3;3819>D538>?<?7ID?3B51D51@5B=1>5>D1>4D1=@5B@B??6B53?B4?6<51B>5BCL<965<?>7
<51B>9>75H@5B95>35C(385:1<1>171>,541(71D1 I=19>D19>9>7@B9F13IG89<5
5>12<9>7 41D1 C81B9>7 (%% 81C D85 @?D5>D91< D? B5F?<ED9?>9J5 D85 54E31D9?>1< <1>4C31@5 
(%%@B?F945CF1B9?EC651DEB5C9>D5B6135C1>49>6B1CDBE3DEB5D?CE@@?BD1335CC1>4EC175?6
54E31D9?> 41D1 13B?CC 49665B5>D C38??<C 1DD5>454 2I <51B>5BC 9>3<E49>7 <51B>5B @B?69<5
5>7175=5>D 1>1<IC9C 3?8?BD 49CDB92ED9?> 1>4 1335CC D? <51B>9>7 3?>D5>DC 1C @B5C5>D54 9>
(385:1<1>171>&1:E=41B(71D1(385:1<1>171>(71D1
1 +851B389D53DEB51>4;5I3?=@?>5>DC?6D85(%%CICD5=49CD9>7E9C89D6B?=?D85B2<?3;3819>
21C54C?<ED9?>C +89C4535>DB1<9J54@<1D6?B=ED9<9J5C1>5DG?B;?649CDB92ED54>?45CD?F1<941D5
1>4 B53?B4 <51B>5BC 54E31D9?>1< 13D9F9D95C  ?>C5AE5>D<I (%% 5>CEB5C D85 9>D57B9DI 1>4
C53EB9DI ?6 D85 41D1 79F9>7 <51B>5BC 3?>DB?< ?F5B D859B 54E31D9?>1< B53?B4C G9D8?ED
3?=@B?=9C9>7 ?> DBECDG?BD89>5CC 1>4 B5<9129<9DI  ,>45BCD1>49>7 D85 EC129<9DI ?6 D85 (%%
CICD5=9C3BE391<6?B9DCCE335CC6E<14?@D9?>1>49>D57B1D9?>9>D?54E31D9?>1<5>F9B?>=5>DC "><978D ?6 D89C G5 3?>4E3D54 1 3?=@B585>C9F5 EC129<9DI 1>1<IC9C EC9>7 D85  #1;?2 '95<C5>
!5EB9CD93C '95<C5>   I 5F1<E1D9>7 D85 EC5B 5H@5B95>35 6B?= D85 @5BC@53D9F5C ?6
CDE45>DC1>4D51385BCG8?5>7175G9D8D85CICD5=D?1335CCC81B51>41>1<IJ554E31D9?>1<B53?B4CG519=D?945>D96ICDB5>7D8CG51;>5CC5C1>41B51C6?B9=@B?F5=5>D9>D8545C97>
1>46E>3D9?>1<9DI?6D85(%%CICD5= 429

+89CB5C51B38>?D?><I@B?F945CF1<E12<59>C978DC9>D?D85EC129<9DI?6D85(%%CICD5=2ED
1<C?3?>DB92ED5CD?D852B?145B695<4?62<?3;3819>21C5454E31D9?>1<CICD5=C .9D81C@539693
B5C51B38 6?3EC ?> D85 EC129<9DI ?6 D85 (%% CICD5= 9> =19>D19>9>7 @B9F13I G89<5 C53EB5<I
C81B9>71>41>1<IJ9>754E31D9?>1<B53?B4CG519=D?144B5CCD85B5C51B38AE5CD9?>?6 .81D9CD85EC129<9DI?6D85<?3;3819>?6%51B>9>7%?7C(%%CICD5= +859>3B51C59>D8514?@D9?>?6D538>?<?7I9>D51389>71>4<51B>9>781CB5CE<D549>D85>554
6?B 1 C53EB5 1>4 5669395>D CICD5= D? 3?>>53D <51B>5BCL 54E31D9?>1< B53?B4C 13B?CC =E<D9@<5
C38??<C +85<?3;3819>?6%51B>9>7%?7C(%%CICD5=?665BC1@B?=9C9>7C?<ED9?>21C54
?>2<?3;3819>D538>?<?7ID?3B51D51@5B=1>5>D1>4D1=@5B@B??6B53?B4?6<51B>5BCL<965<?>7
<51B>9>75H@5B95>35C(385:1<1>171>,541(71D1 I=19>D19>9>7@B9F13IG89<5
5>12<9>7 41D1 C81B9>7 (%% 81C D85 @?D5>D91< D? B5F?<ED9?>9J5 D85 54E31D9?>1< <1>4C31@5 
(%%@B?F945CF1B9?EC651DEB5C9>D5B6135C1>49>6B1CDBE3DEB5D?CE@@?BD1335CC1>4EC175?6
54E31D9?> 41D1 13B?CC 49665B5>D C38??<C 1DD5>454 2I <51B>5BC 9>3<E49>7 <51B>5B @B?69<5
5>7175=5>D 1>1<IC9C 3?8?BD 49CDB92ED9?> 1>4 1335CC D? <51B>9>7 3?>D5>DC 1C @B5C5>D54 9>
(385:1<1>171>&1:E=41B(71D1(385:1<1>171>(71D1
1 






+855F1<E1D9?>9>D89CCDE4I9C3?>4E3D54D8B?E78185EB9CD931@@B?138G89389>F?<F5C5H@5BDCCICD5=1D931<<I 5H1=9>9>7 D85 (%% CICD5= 1719>CD D85 5CD12<9C854 EC129<9DI 85EB9CD93C
'95<C5>   +89C 1@@B?138 1<<?GC EC D? 945>D96I @?D5>D91< EC129<9DI 9CCE5C 1>4 @B?F945
B53?==5>41D9?>C6?B5>81>39>7D85CICD5=CEC129<9DI1>4EC5B5H@5B95>35 ">D89C5F1<E1D9?>
G54569>5D856?<<?G9>7EC5B7?1<CD?2531BB954?EDG89<55F1<E1D9>7D85EC129<9DI?6D85(%%
CICD5= #$#"• -95G<51B>9>741D11D49665B5>DC38??<CD?CE@@?BD3EBB5>D<51B>9>713D9F9D95C •
B1>D1>4B5F?;5@5B=9CC9?>CD?F95G41D11D@B5F9?ECC38??<D?D51385B1D3EBB5>D
C38??< • 335CC4979D1<3?>D5>DCCE381CD5HD2??;C1>4<53DEB5C<945C1CB5F9C9?>=1D5B91<C !"• -95G@B?69<5?6CDE45>DC?>@B5B5AE9C9D53?EBC5CD?CE@@?BDD859B3EBB5>D<51B>9>77?1< • -95G D85 5>7175=5>D 49CDB92ED9?> ?6 CDE45>DC 9> 1 @B5B5AE9C9D5 3?EBC5 D?G1B4C
B53?==5>49>71@@B?@B91D5B5F9C9?>13B?CC3?8?BDC • ?=@1B5 D85 5>7175=5>D ?6 CDE45>DC 13B?CC 49665B5>D D9=5C ?B 3?EBC5C D? 45D53D
3?>C9CD5>D2581F9?EB?B1DB9C;D5>45>395C • 335CC4979D1<3?>D5>DCEC542ICDE45>DC9>1@B5B5AE9C9D53?EBC5D?E>45BCD1>4D85
45@D8?6<51B>9>7?B1CC5CC=5>D     49F5BC57B?E@?6@1BD939@1>DCG1CC5<53D54D?B5@B5C5>DD85D1B75DEC5B7B?E@C?6D85(%%
CICD5= CDE45>DC 1>4 D51385BC  +? =9>9=9J5 291C @1BD939@1>DC G5B5 B53BE9D54 6B?= F1B9?EC
54E31D9?>1< 9>CD9DED9?>C D81D 3?E<4 @?D5>D91<<I EC5 D85 (%% CICD5=  66?BDC G5B5 =145 D?
5>CEB5 1 F1B954 B5@B5C5>D1D9?> 9> D5B=C ?6 D51389>7 1>4 <51B>9>7 5H@5B95>35 1C G5<< 1C
61=9<91B9DIG9D82<?3;3819>D538>?<?7I1>4<51B>9>7=1>175=5>DCICD5=C ">D?D1<D51385BC
1>4CDE45>DC@1BD939@1D549>D89C5F1<E1D9?>   
+?@B?F945@1BD939@1>DCG9D8181>4C?>5H@5B95>35?6D85(%%CICD5=C@539693651DEB5C1>4
6E>3D9?>1<9D95C D81D CE@@?BD D85 EC5B 7?1<C G5B5 45=?>CDB1D54 +85C5 651DEB5C 9>3<E454 D85
129<9DI D? F95G <51B>5B 41D1 6B?= =E<D9@<5 C38??<C =1>175 @5B=9CC9?>C 6?B 41D1 1335CCF9CE1<9J5 1>1<ID93C 45B9F54 6B?= 3?>>53D54 <51B>9>7 <?7C 1>4 @B?F945 1335CC D? <51B>9>7

430

=1D5B91<CEC5413B?CCF1B9?EC54E31D9?>1<9>CD9DED9?>C 

'%0-#1C82?1B4F95G/CDE45>D0

'%0-#*38??<CF95G/CDE45>D0





'%0-#?EBC5CF95G/CDE45>D0

'%0-#)5B=9CC9?>CF95G/CDE45>D0'%0-# %51B>9>73?>D5>DCF95G
/CDE45>D0

'%0-# %51B>5B@B?69<5F95G/D51385B0

'%0-# )B9?B5>7175=5>DF95G
/D51385B0

'%0-# ?=@1B55>7175=5>DF95G
/D51385B0



431





 

?B5138EC5B7B?E@1C5D?6AE5CD9?>>19B5CG5B545F5<?@5421C54?>D85 #1;?2'95<C5>
!5EB9CD93C6?BEC129<9DI5F1<E1D9?> +85C585EB9CD93CC5BF541C1C5D?67E945<9>5C6?B1CC5CC9>7
D85EC129<9DI?69>D5B13D9F5CICD5=C 13885EB9CD93G1C141@D54D?D853?>D5HD ?6D85(%%
CICD5=1>4C@539693AE5CD9?>CG5B56?B=E<1D54D?5F1<E1D5D85CICD5=C@5B6?B=1>351719>CD
5138 3B9D5B9?>  EBD85B=?B5 D85 AE5CD9?>>19B5C 19=54 D? 71D85B 2?D8 AE1>D9D1D9F5 1>4AE1<9D1D9F56554213;6B?=@1BD939@1>DCB571B49>7D859B5H@5B95>351>4@5B35@D9?>?6D85(%%
CICD5=C EC129<9DI +? =9D971D5 291C5C 9> D89C CDE4I G5 45F5<?@54 3<51B 1>4 E>291C54
AE5CD9?>>19B5CG9D85H@5BD9>@ED1>4@9<?DD5CD9>7 <D8?E78D85CDE4I8141<9=9D54C1=@<5C9J5
G893816653DC75>5B1<9J129<9DID85B5CE<DCCD9<<?665BF1<E12<59>C978DC9>D?D85(%%CICD5=C
EC129<9DI

+8541D13?<<53D546B?=D85AE5CD9?>>19B5CG5B51>1<IJ54D?1CC5CCD85EC129<9DI?6D85(%%
CICD5= 97EB5C1>4 C8?GD85B5CE<DC?6D851>1<IC9C?6D85 AE1>D9D1D9F541D1 B5DB95F54?>
1C31<5%9;5BD6?B5138?6D85CICD5=651DEB5C6?B5138EC5B7B?E@            
+?9>D5B@B5DD85B5CE<DC?6D8585EB9CD935F1<E1D9?>3?>4E3D54?>1@?9>D%9;5BDC31<5G569BCD
31<3E<1D5D85=51>?6D85C3?B5C79F5>2ID85@1BD939@1>DC6?B5138?6D85 #1;?2'95<C5>
85EB9CD933B9D5B91 ?B5138=51>C3?B5G51CC97>1=51>9>71C6?<<?GC•  4 2!#((#*/ +85?<<CICD5=@5B6?B=C5H35@D9?>1<<IG5<<?>D89C3B9D5B9?>•   4   ++" +85 (%% CICD5= 75>5B1<<I =55DC D85 85EB9CD93 7E945<9>5C 1>413895F5C1C1D9C613D?BI<5F5<?6EC129<9DI6?BD89C3B9D5B9?>•   4   '- +85 (%% CICD5= C8?GC C?=5 EC129<9DI 9CCE5C 1>4 B??= 6?B
9=@B?F5=5>D9>=55D9>7D8585EB9CD937E945<9>5C6?BD89C3B9D5B9?>•
 K  )??B +85 (%% CICD5= 81C C97>96931>D EC129<9DI 9CCE5C 1>4 45F91D5C
3?>C945B12<I6B?=D8585EB9CD93@B9>39@<5C6?BD89C3B9D5B9?>• 4  #-3++- +85(%%CICD5=C5F5B5<I<13;CEC129<9DI1>4619<CD?144B5CCD85
85EB9CD937E945<9>5C6?BD89C3B9D5B9?>432            
.5B5@?BDD85B5CE<DC?6D855F1<E1D9?>C?6F1B9?EC651DEB5C?6(%%2I5138EC5B7B?E@25<?G

(F5B1<<D8585EB9CD935F1<E1D9?>3?>4E3D542ID85D51385B7B?E@B5F51<54175>5B1<<I@?C9D9F5
@5B35@D9?>?6D85(%%CICD5=9>D5B=C?6EC129<9DI +85<51B>5B@B?69<5F95GB5359F54@?C9D9F5
B1D9>7C13B?CC=?CD?6D855F1<E1D5485EB9CD93C9>4931D9>717??4<5F5<?63?>C9CD5>3I5BB?B@B5F5>D9?> B53?7>9D9?> ?F5B B531<< 1>4 =1D38 D? D85 B51<G?B<4 3?>D5HD  +85 F9C929<9DI EC5B
3?>DB?<6B554?=6<5H929<9DI5669395>3I1>415CD85D931>4=9>9=1<9CD1C@53DC?6D859>D5B6135
G5B5G5<<B571B454B5359F9>7B1D9>7C?6 ?B89785B !?G5F5BD85B59CB??=6?B9=@B?F5=5>D
9>D5B=C?685<@4?3E=5>D1D9?>G8938G1CB1D541C619B=   +855F1<E1D9?>?6D85
@1CD5>7175=5>DF95G3?=@1B55>7175=5>DF95G1>4<51B>9>73?>D5>DCF95G1<C?C8?G54
75>5B1<<I@?C9D9F5B5CE<DCG9D83?>C9CD5>DB1D9>7C12?F5 6?B=?CD85EB9CD93C '?D12<ID85
6<5H929<9DI  5669395>3I 85EB9CD93 C3?B54 8978 9> D85 3?=@1B5 5>7175=5>D F95G =    
9>4931D9>7 9DC 56653D9F5>5CC  (F5B1<< D85C5 69>49>7C CE775CD D81D D85 (%% CICD5=
45=?>CDB1D5C@B?=9C9>7EC129<9DIG9D8C@5396931B51C945>D969546?B@?D5>D91<5>81>35=5>DC@1BD93E<1B<I9>D5B=C?6@B?F949>725DD5B85<@1>44?3E=5>D1D9?>B5C?EB35C6?BD51385BC

 

+85 85EB9CD93 5F1<E1D9?> 3?>4E3D54 G9D8 D85 CDE45>D 7B?E@ 45=?>CDB1D54 ?F5B1<< @?C9D9F5
@5B35@D9?>C ?6 D85 EC129<9DI ?6 D85 (%% CICD5=  "> D85 CDE45>DC 41D1 F95G D85 CICD5=
B5359F548978B1D9>7C6?B3?>C9CD5>3I1>4CD1>41B4C= =1D38D?D85B51<G?B<43?>D5HD
= F9C929<9DI= 1>415CD85D931>4=9>9=1<9CD45C97>= 9>4931D9>7D81DCDE45>DC6?E>4D85C51C@53DCG5<<45C97>541>456653D9F5 +85CICD5=1<C?@5B6?B=54G5<<9>
D5B=C?6EC5B3?>DB?<1>46B554?==  B53?7>9D9?>?F5BB531<<=  1>46<5H929<9DI
1>45669395>3I=  !?G5F5BD85B5G5B51B51C945>D969546?B9=@B?F5=5>D@1BD93E<1B<I
9> B53?7>9J9>7 4917>?C9>7 1>4 B53?F5B9>7 6B?= 5BB?BC =   1>4 @B?F949>7
3?=@B585>C9F585<@1>44?3E=5>D1D9?>=  *9=9<1B<I9>D8541D1@5B=9CC9?>F95GD85
CICD5=5H8929D54CDB5>7D8C9>3?>C9CD5>3I1>4CD1>41B4C= EC5B3?>DB?<1>46B554?=
=     F9C929<9DI =    1>4 15CD85D93 1>4 =9>9=1<9CD 45C97> =     !?G5F5B

433

D85B5 G5B5 ?@@?BDE>9D95C 6?B 9=@B?F5=5>D 9> B53?7>9J9>7 4917>?C9>7 1>4 B53?F5B9>7 6B?=
5BB?BC= 1>4@B?F949>73?=@B585>C9F585<@1>44?3E=5>D1D9?>=  ">D85
<51B>9>73?>D5>DCF95GD85CICD5=B5359F54@?C9D9F5B1D9>7C6?B3?>C9CD5>3I1>4CD1>41B4C= EC5B3?>DB?<1>46B554?==  5BB?B@B5F5>D9?>=  B53?7>9D9?>?F5B
B531<< =    F9C929<9DI =     1>4 15CD85D93 1>4 =9>9=1<9CD 45C97> =    !?G5F5B9=@B?F5=5>DCG5B5>554549>B53?7>9J9>74917>?C9>71>4B53?F5B9>76B?=5BB?BC=   1>4 @B?F949>7 3?=@B585>C9F5 85<@ 1>4 4?3E=5>D1D9?> =    (F5B1<< D85
5F1<E1D9?> @B?F9454 F1<E12<5 9>C978DC 9>4931D9>7 D81D G89<5 D85 (%% CICD5= 45=?>CDB1D54
CDB5>7D8C9>F1B9?ECEC129<9DI1C@53DCD85B5G5B5C@5396931B51CCE381C5BB?B81>4<9>71>4
85<@ 4?3E=5>D1D9?> D81D B5AE9B54 1DD5>D9?> 1>4 9=@B?F5=5>D  +85C5 69>49>7C 8978<978D D859=@?BD1>35?6144B5CC9>7D85C53?>35B>CD?5>81>35D85?F5B1<<EC5B5H@5B95>356?BCDE45>DC 

">3?>3<EC9?>D89C@1@5B@B5C5>D54185EB9CD935F1<E1D9?>?6D85<?3;3819>?6%51B>9>7%?7C
(%%CICD5=6?3EC9>7?>9DCEC129<9DI6B?=D85@5BC@53D9F5C?62?D8CDE45>DC1>4D51385BC +855F1<E1D9?>B5F51<54C5F5B1<CE335CC5C?6D85(%%CICD5=9>3<E49>7CDB?>7@5B6?B=1>35
9>1B51CCE381C3?>C9CD5>3IB51<G?B<4B5<5F1>35EC5B3?>DB?<1>415CD85D9345C97> +85C5
69>49>7C45=?>CDB1D5D85CICD5=C@?D5>D91<D?56653D9F5<ICE@@?BD<965<?>7<51B>9>71>46139<9D1D5
D85C81B9>7?654E31D9?>1<B53?B4CC53EB5<I !?G5F5B<9=9D1D9?>CG5B5945>D96954@1BD93E<1B<I
9> D5B=C ?6 5BB?B 81>4<9>7 1>4 D85 1F19<129<9DI ?6 3?=@B585>C9F5 85<@ 1>4 4?3E=5>D1D9?>
B5C?EB35C EDEB5G?B;C8?E<4@B9?B9D9J5144B5CC9>7D85C5<9=9D1D9?>CD?6EBD85B5>81>35D85
CICD5=C EC129<9DI  449D9?>1<<I D85 C=1<< C1=@<5 C9J5 1>4 @?D5>D91< 291C5C 9> @1BD939@1>D
C5<53D9?><9=9DCD8575>5B1<9J129<9DI?6D8569>49>7C9>D89CCDE4I !?G5F5BD859>C978DC719>54
6B?=D89CCDE4I9>3?>:E>3D9?>G9D8@B5F9?ECB5C51B381>4D85?B5D931<3?>C945B1D9?>C@B?F945
1F1<E12<56?E>41D9?>6?B6EBD85B5H@<?B1D9?>C1>4B569>5=5>DC?6D85(%%CICD5=CEC129<9DI 
EDEB5 CDE495C G9D8 <1B75B 1>4 =?B5 49F5BC5 C1=@<5C G?E<4 85<@ F1<941D5 1>4 5HD5>4 ?EB69>49>7C 

+89CG?B;G1CCE@@?BD549>@1BD2ID856?<<?G9>77B1>DC#1@1>*?395DI6?BD85)B?=?D9?>?6*395>35 #*)* B1>D9>94 6?B 5<<?GC #) $#    #) !   #) !
H@<?B1D?BI #) $    #) !   1>4 '5G >5B7I 1>4 ">4ECDB91< +538>?<?7I
5F5<?@=5>D(B71>9J1D9?>'(E>45B B1>D#)')1>4 B1>D#)')   

?*2/6--   '0,32,  A3:4(55  /(::0    #*/86,+,8 %  ,90.5 (5+0473,4,5:(:0656-(3,(8505.(5(3?:0*9:66320:-68:,(*/,89,1.+ ),$"1! 0',+ )#!&+,),%3
,!'#03  
0,39,5  78035/(5*05.:/,,>73(5(:68?76=,86-;9()030:?/,;809:0*95.,!##"'+%/,$0&#
!,+$#.#+!#,+1* + !0,./'+,*-10'+%3/0#*/77   
*/,1(!.)6 ?,3,8,##3(5(.(5 .(:(
)36*2*/(0505,+;*(:065
9?9:,4(:0*8,<0,=(5+78(*:0*(3*(9,9:;+0,9!!#//  */,1(!3(5(.(5 .(:(
(99,994,5:",9;3:965:/,36*2*/(0565*,7:;(38(4,=6825.0'$'!' )+0#))'%#+!#'+"1! 0',+,/0#./ +" 0#.# ('+%#/1)0/,.(/&,-/+" 10,.' )/ +"1/0.3 +" ++,2 0',+ . !(/ . !0'0',+#./5 +" ,!0,. ) ,+/,.0'1*  ."+0#.+ 0',+ ),+$#.#+!#1.& *1)3 4.,!##"'+%/ .077/(4#7805.,85:,85(:065(3!;)309/05.
*/,1(!3(5(.(5%,+( .(:(
(5(.05.30-,365.3,(8505.8,*68+9:/86;./
)36*2*/(05#/# .!& +". !0'!#'+#!&+,),%3+& +!#"# .+'+%    */,1(!3(5(.(5%,+( .(:(
5<,9:0.(:05.",3,<(5*,6-!8068,(8505.(:(655,*:,+:/86;./:/,36*2*/(055 .,!##"'+%/,$0&#.,!##"'+%/,$0&# 0&+0#.+ 0',+ )
,+$#.#+!#,+,*-10#./'+"1! 0',+ +%(,(& ') +"77   434

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Program Visualization System Supporting
Teacher-Intended Stepwise Refinement
Koichi YAMASHITAa*, Hiroki SOMAb, Satoru KOGUREb, Yasuhiro NOGUCHIb,
Raiya YAMAMOTOa, Tatsuhiro KONISHIb & Yukihiro ITOHc
aFaculty of Business Administration, Tokoha University, Japanb
Faculty of Informatics, Shizuoka University, Japanc
Shizuoka University, Japan*yamasita@hm.tokoha-u.ac.jp
Abstract: This study portrays a learning support system that visually represents the
structure of stepwise algorithm refinement, also known as the structure of programcode abstraction, aligned with the teacher's instructional objectives. We focus on thevarying needs of learners in programming education and consider the possibility thatdifferent levels of stepwise refinement may be required based on their knowledge andexperience. Novice learners with limited programming experience may need programcode abstractions in finer steps, rather than the abstraction of numerous statements
simultaneously. For learners with a certain level of experience, abstraction withexcessive steps may hinder their holistic understanding of the entire program from ahigher perspective. In this study, we developed a program visualization system thatempowers teachers to customize the visualizations of the structure of stepwise
abstractions and the data structures utilized by the target program according to theirinstructional intentions. We introduced the proposed system into an actual class andevaluated its effectiveness by measuring the degree of improvement in learners'
understanding based on tests.
Keywords: Programming education, program visualization
refinement and abstraction, educational authoring toolsystem,stepwise

1. Introduction
To date, various program visualization (PV) systems have been developed to support the
programming learning of novice learners (Sorva, Karavirta, & Malmi, 2013). We had
developed a PV system called Teacher's Explaining Design Visualization Tool (TEDViT) andconducted several classroom sessions introducing this system into actual classes (Kogure etal., 2014). There are two features of TEDViT that distinguish it from other existing systems:
first, it enables teachers to customize PVs based on their objectives, and second, it enables
learners to observe and compare two PVs with different abstraction levels of target data
structures. However, it cannot provide visualizations at different levels of abstraction forprogram-code that represents concrete operational procedures for the target domain worldof the data structures to be processed.
In programming education, stepwise refinement has been used to design programcodes by gradually decreasing the level of abstraction from highly abstract algorithms.
Moreover, there have been various attempts to support learners' understanding of programs
by focusing on stepwise abstraction used to understand the algorithm behind program codeby gradually increasing the level of abstraction of the code. In a classic study, Shneiderman& Mayer (1979) summarized the procedures that expert programmers undertake during code
reading. These steps include chunking some statements within the target program,recognizing the function of the chunk, and progressively assembling the chunks to form
larger units. The chunking process is continued recursively until the entire program isunderstood. This implies that learners need to perform stepwise abstraction by chunking
concrete operation procedures to understand a program. However, the literature suggests

435

that novice learners may frequently struggle to comprehend a program owing to difficulties in
effectively chunking the code.
In this study, we developed a system that can visualize program behavior by drawing
target program codes that represent concrete processing procedures and problem analysis
diagrams (PADs) that represent algorithms with a higher level of abstraction than programcode, enabling users to gradually change abstraction level of PADs. The abstraction level of
PAD here refers to the size of blocks of program code to be chunked, with larger chunked
blocks being considered highly abstract and smaller chunked blocks being considered lessabstract. This system is an extension of TEDViT that enables teachers to define the chunks
to be included in algorithm visualization. Learners understand chunks differently dependingon their knowledge and experience. Hence, our system, which allows teachers to define
chunking structures, is expected to provide PVs that better align with learners' understanding
compared to existing systems with fixed chunking structures determined by developers.

2. Related Works
2.1 TEDViT
Price, Baecker, & Small (1998) categorized users of PV systems into the following four roles:
User/viewer who observes the world visualized by the system; visualizer/animator who
defines the visualization of the target program's behavior; software developer who develops
the PV system; and programmer who designs the program that is the target of visualization.
A typical existing PV system assigns the role of user/viewer to the learners,
visualizer/animator and software developer to the PV system's developer, and programmer
to the learners and teachers.
As mentioned earlier, we had developed a PV system called TEDViT, which assigns
the role of visualizer/animator to the teachers. TEDViT is a system that enables teachers todefine drawing rules to customize the visualization suitable for each target program and forthe target user/viewer. When a teacher explains a program behavior to novice learners using
visualizations, the teacher may sort out, highlight, and lay out some drawing objects from thetarget data structure. The teacher may also attach descriptive natural language texts tosome objects to assist students' understanding and may draw pointers and array indexers
using arrows. We consider these visualizations to be based on a policy that the teacher
determines according to the learners' background knowledge and degree of understanding,the properties of the target program, etc. We refer to these visualization policies and theconcepts that the teacher intends to explain using these policies as the teacher's "intent of
instruction." TEDViT is, therefore, a PV system that can reflect teachers' intent of instruction
regarding visualizations by assigning them the role of visualizer/animator.
TEDViT enables visualizers/animators to create or edit a configuration file for PV
creation based on their intent of instruction independently from the target program file.
Configuration file comprises a set of drawing rules, each of which is a comma-separated
value (CSV) entry consisting of a condition part and an object part. The condition partdefines the condition to fire the drawing rule. The object part defines the operation to edit thetarget object (i.e., "create," "delete," and "update" are available) and the attributes necessary
to draw the object such as the object type, position, color, and corresponding variables.
TEDViT generates PVs by scanning configuration files, interpreting the visualization policy
represented by a set of drawing rules, and visualizing drawing objects accordingly.
Users/viewers can observe the program behavior based on the PVs. The supporting
programming language is C.

2.2 Programming Learning Support Based on Stepwise Refinement and Abstraction
In general, a PV system visualizes the target domain world involved in program processing
and aims to support learners in understanding the program's meaning by enabling them toobserve the changes in the domain world caused by the program. Each statement is used as436

a unit of meaning, that is, a unit that causes changes in the domain world. Although PV
systems can visually support the understanding of the meaning of each statement, they may
not be able to provide sufficient support for abstracting the meaning of multiple statements
and understanding it, and hence, understanding the meaning of the entire program.
The stepwise refinement technique has been widely used in programming education.
The procedure summarized by Shneiderman & Mayer (1979) implies that program
comprehension can be achieved by the opposite operation, namely, stepwise abstraction
from the detailed meaning of individual statements to abstract meaning that spans multiple
statements. Here, abstraction is the interpretation of the meaning of chunking and chunks ofthe program, and refinement is the decomposition of chunks into smaller chunks. In learning
programming, it is also necessary to understand the meaning of programs through theabstraction of operational procedures and refinement of algorithms; this is not supported by
PV systems. Based on this idea, there have been several attempts to aid novice
programming learners by supporting the stepwise abstraction of programs and pseudocode(Shinkai & Sumitani, 2007; Watanabe, Tomoto, & Akakura, 2015; Kogure et al., 2013).

3. Proposed System
In this study, we focused on the possibility that the learners may need different granularities
of chunking in programming learning based on stepwise refinement depending on theirknowledge and experience. For novice learners with minimal programming experience, small
chunks of program code have to be abstracted at multiple levels of abstraction, rather thanchunking a large amount of code simultaneously. For learners with a certain level ofexperience, chunking extremely granularly may hinder their holistic understanding of theentire program from a higher perspective. Yamashita et al. (2016) highlighted that there are
two possible types of abstraction of sequences of operations: simple grouping and process
generalization. In our teaching experience, learners with insufficient knowledge of CS andmathematics often struggle to understand chunking, particularly when it involves process
generalization. In such cases, teachers could customize the chunking granularity andexplanation of the meaning of the chunks based on the differences in learners' knowledge
and background. Moreover, the differences in the chunking granularities would have acertain effect on their visualization policy of the target domain world.
In this study, we developed a PV system that enables teachers to freely define andvisualize the chunks of program code and their structures based on their intent of instruction.
The chunks of program code are visualized as PAD-based algorithmic representations. Our
system can enable users to increase the abstraction granularity of each chunk object andvisualize more detailed PADs. It builds upon TEDViT, a program visualization (PV) systemthat empowers teachers to customize the visual representations of the domain world
according to their instructional objectives. Therefore, our system provides teachers with theflexibility to define and customize the visualizations of the data structures involved inprogram processing. Figure 1 provides a screenshot of the proposed system.
Our system generates a learning environment with four main visualization areas. In
area (A), the target program code is visualized, and an executing statement or chunk ishighlighted. In area (B), a PAD-based algorithm representation is visualized. Area (C)visualizes the currently executed operation, displaying the text of the executing chunk. Area
(D) visualizes the domain world, drawing the data structure of the program's processing
target based on the definition file edited by the teacher as in TEDViT.
Each element of the PAD in area (B) corresponds to an abstracted chunk ofstatements in a certain range of the program code. The correspondence between eachelement of the PAD and each range of the program code can be identified by the number
assigned to each element of the PAD and each range of the program code. In addition,
buttons such as "+" and "-" are provided at the top of each element of the PAD. By clicking
these buttons, the chunk of each element can be displayed in one more step in detail, as
shown in Figure 2. As the algorithm representation visualized in area (B) includes chunking
structures, it cannot strictly be called a PAD. We call the algorithm representation visualized437

by the proposed system "extended PAD." In addition to the aforementioned features, someelements of the extended PAD have buttons for "step in function" and "step over function,"
resembling typical debuggers. Some elements also have buttons for "execute all loop" and"execute one round of loop" depending on the processing contents.

Figure 1. Screenshot of the proposed system.

Figure 2. Visualizing a chunking structure in one step more detailed.

4. Evaluation
To assess the effectiveness of our system, we conducted a classroom practice byincorporating it into an actual programming course. The hypothesis to be verified is whether

438

our system could help learners to understand the behavior of programs and algorithms incode reading learning better than an existing PV system, TEDViT. The actual class whereby
the systems were introduced was an on-demand class offered to second-year university
students majoring in computer science. The number of participants in our class was 47, andthey had 1-2 years of programming experience. Two types of sample programs, quick sortand merge sort, were prepared for code reading. All participants had already learned quick
sort in the previous semester but had not learned merge sort.
In the practiced class, we had the participants perform code reading learning in theorder of quick sort first and merge sort second. In each code reading learning, they took a
pre-test before learning with PV systems, and then performed two learnings of code
readings, one with TEDViT and the other with our system. They took tests immediately after
each of the two learnings, which were positioned as a mid-test and a post-test, respectively.
To reduce order effects, we divided the participants into two groups, Groups A and B, witheach group learning with PV systems in reversed order. We also reversed the order of use of
the system in the two learnings of quick sort and merge sort for each group as well. After all
the learnings were conducted, we conducted a questionnaire survey.
In the code reading learning using TEDViT, participants observed changes in the
logical data structure visualized in the domain world with stepwise execution of eachstatement of the target program. In the learning using our system, participants used the "+"and "-" buttons provided in area (B) of Figure 1 to observe the chunking structure of theextended PAD and simultaneously observed the changes in the domain world with stepwise
execution of each statement or each chunk. The pre-test, mid-test, and post-test consistedof nine questions for the quick sort and 11 questions for the merge sort and were identicalfor each target program. Participants were required to take six tests in the entire class, and a
total of 32 participants (13 in Group A and 19 in Group B) took all six tests.
Table 1 summarizes the average correct answer rates for each test for the 32
participants who took all tests. Underlined scores are the rates for the tests immediately after
learning with our system. Both Groups A and B, regardless of the target program,demonstrated improved scores after code reading learning compared to before learning,suggesting that participants would improve their understanding of the programs. We also
calculated average improvements for learning with TEDViT and for learning with our system
by calculating score differences for all test results compared to the results of the immediately
preceding test. We found the average improvement of correct answer rates for learning with
TEDViT was 9.2% and that for learning with our system was 14.4%. Currently, we have notobtained any statistical results that significantly support our hypothesis that our system
contributes more to learners' understanding of programs than TEDViT. However, theobserved average improvement of correct answer rates for our system, which is 5% higher
than that of TEDViT, suggests a significant degree of effectiveness in enhancing learners'
understanding.
Table 1. Average correct answer rate of pre-, mid-, and post-test
Group
A
A
B
B

N.
13
13
19
19

TargetQuick sort
Merge sort
Quick sort
Merge sort

Pre-test
.554
.608
.332
.526

Mid-test
.662
.746
.668
.768

Post-test
.708
.738
.679
.758

The post-class questionnaire included a 5-point pairwise comparison of whether
TEDViT or our system was easier to learn, with 1 indicating a strong preference for TEDViT,
5 indicating a strong preference for our system, and the median being 3. The mean value forthis item was 3.3 for the 30 participants who responded to the questionnaire, indicating thatparticipants tended to prefer our system to TEDViT. Moreover, the mean values of all theitems for which participants were asked to subjectively evaluate the effectiveness of thefunctions of our system were also higher than the median. Subjective evaluations of the439

learning effectiveness of our system were also favorable, suggesting that participants
positively viewed our system.

5. Conclusion
This study described a PV system that can simultaneously provide learners with processing
procedures and processing objects at different levels of abstraction. The system achieves
this by generating visualizations of the stepwise chunking structure of program code andvisualizations of logical data structure, which are the objects being processed. We assessed
the effectiveness of our system by introducing it in an actual class. As our system is basedon the existing PV system, TEDViT, we asked participants to perform code reading learning
using TEDViT and learning using our system and measured the effects of two learning
activities by administering pre-, mid-, and post-tests. To reduce order effects, participants inthe practice class were divided into two groups, each with an opposite learning sequence. In
addition, two target programs for code reading, quick sort, and merge sort, were prepared,
and the effect of the systems was measured on multiple program codes. The evaluation
results based on the correct answer rates in three tests indicated no statistical difference
between code reading using TEDViT and code reading using our system. However, TEDViT
achieved a 9.2% improvement in correct answer rate on average, while our system achieved
a 14.4% improvement in correct answer rate on average, indicating that the improvement
was more than 5% better for learning with our system. We can conclude that the results
indicate a certain degree of support for the effectiveness of our system.

Acknowledgements
This study was supported by JSPS KAKENHI Grant Numbers JP19K12259 and
JP22K12290.

References
Kogure, S., Fujioka, R., Noguchi, Y., Yamashita, K., Konishi, T., Itoh, Y. (2014). Code reading
environment according to visualizing both variable's memory image and target world's status.
Proceeding of the 22nd International Conference on Computers in Education, 343-348.
Kogure, S., Okamoto, M., Yamashita, K., Noguchi, Y., Konishi, T., Itoh, Y. (2013). Evaluation of analgorithm and programming learning support features to program and algorithm learning support
environment. Proceedings of the 21st International Conference on Computers in Education, 418424.
Price, B., Baecker, R., Small, I. (1998). An Introduction to software visualization. In J. Stasko, J.
Domingue, M. H. Brown, & B. A. Price (Eds.), Software Visualization (pp. 3-27). MIT Press.
Shinkai, J., Sumitani, S. (2007). Development of programming learning support system emphasizing
process. Japan Journal of Educational Technology, 31(Suppl.), 45-48.
Shneiderman, B., Mayer, R. (1979). Syntactic/Semantic interaction in programmer behavior: A modeland experimental results. International Journal of Computer and Information Sciences, 8(3), 219238.
Sorva, J., Karavirta, V., Malmi, L. (2013). A review of generic program visualization systems for
introductory programming education. ACM Transactions on Computing Education (TOCE), 13(4),15.
Watanabe, K., Tomoto, T., Akakura, T. (2015). Development of a learning support system for reading
source code by stepwise abstraction. In S. Yamamoto (Ed.), Human Interface and theManagement of Information. Information and Knowledge in Context. HIMI 2015. Lecture Notes in
Computer Science (Vol. 9173, pp. 387-394). Springer, Cham.
Yamashita, K., Nagao, T., Kogure, S., Noguchi, Y., Konishi, T., Itoh, Y. (2016). Code-reading support
environment visualizing three fields and educational practice to understand nested loops.
Research and Practice in Technology Enhanced Learning, 11(3), 1-22. doi:10.1186/s41039-0160027-3

440

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Automatic Detection of Negotiation in
Collaborative Complex Problem Solving
Interactions
Daevesh SINGHa*, Ulfa KHWAJAa , Sahana Murthya , & Ramkumar RAJENDRANaa
IDP in Educational Technology, Indian Institute of Technology Bombay, INDIA*daeveshsingh@iitb.ac.in
Abstract: When learners collaborate on complex problems and open-ended tasks, themechanism of negotiation plays a crucial role in establishing a common understanding
and achieving a shared goal among them. Research has shown that negotiation
improves problem-solving processes, making it an essential skill to be developed
among learners. In this study, we propose a method for automating the identification ofnegotiation in learners' discourse during collaboration. We leverage language models
like BERT, RoBERTa, and GPT2 along with traditional machine learning models like
logistic regression to detect utterances of negotiation in learners' discourse while theycollaboratively solve engineering estimation problem in an Open-Ended Learning
Environment (OELE) called Modeling Based Estimation Learning Environment(MEttLE). Our findings suggest that our approach can accurately identify negotiationutterances with a high accuracy of 0.924 and 0.781 kappa value with a relatively smaller
training set. Our method is the first step in real-time detection of negotiation, thereby
enabling educators to design scaffolds and environments to help learners engage in
effective negotiations.
Keywords: negotiation, open-ended learning environments, collaboration, complex
problem-solving.

1. Introduction
Collaborative problem-solving (CPS) is a vital skill involving multiple individuals working
together to devise and sustain collective solutions to challenges (Fiore et al., 2017). CPS is
crucial as the modern workforce increasingly demands seamless collaboration within diverse
teams, efficient exchange of expertise, and effective communication across disciplinary
boundaries (Pugh, Rao, Stewart, & D'Mello, 2022). To foster this skill, it is imperative tocreate opportunities for students to engage in productive CPS activities.
Despite the positive impact of CPS on both academic and social educational
outcomes, significant challenges persist when implementing CPS effectively within everyday
classroom settings (Johnson, Johnson, & Smith, 2007). Teachers encounter difficulties instructuring group interactions and monitoring fruitful collaboration (Van Leeuwen, Janssen,
Erkens, & Brekelmans, 2013). Students also face hurdles in CPS, including unevenparticipation in group tasks (Freeman & Greenacre, 2010) and deficiencies incommunication and collaborative skills (Li & Campbell, 2008; Pauli, Mohiyeddini, Bray,
Michie, & Street, 2008).
To address these challenges, it is imperative to pinpoint and assess essential CPS
processes exhibited by learners and offer diagnostic feedback (Fiore et al., 2017). This
paper focuses on automating the identification of a critical CPS process—Negotiation—whilestudents collaboratively tackle complex problem within an Online Environment for Learningand Education (OELE) called Modeling Based Estimation Learning Environment (MEttLE).
This automated analysis will subsequently empower educators to provide valuable feedback
to learners.
The mechanism of negotiation is vital for effective collaboration, particularly incomplex problem-solving domains, enabling collaborators to employ diverse strategies and441

reach agreements through critical evaluation, justification, sensemaking, and co-constructionof solutions (Hesse et al., 2015). Previous research has demonstrated the productive
behaviours exhibited by learners when engaging in negotiation within OELEs during complex
problem-solving tasks (Khwaja & Murthy, 2022). Therefore, fostering negotiation in suchenvironments is crucial for supporting learners' problem-solving processes. However,identifying negotiation utterances in learners' discourse is challenging, particularly when theinteractions are complex and multi-layered. Also, the manual identification of negotiationutterances is time-consuming and impractical for large-scale studies.
To address this gap, researchers have proposed a method for automating theidentification of negotiation and other collaborative mechanisms in learners' discourse during
the process of collaborative problem-solving (Flor et al., 2016; Pugh, Rao, Stewart, andD'Mello, 2022; Hao et al., 2017). We extend this work by leveraging state-of-the-art
language models like BERT, RoBERTa, and GPT2 to automatically detect negotiationutterances in learners' discourse while they solve open-ended problems. Results show that
GPT2 emerged as the best-performing model with an accuracy of 0.924 and a kappa valueof 0.781. The novelty of our research lies in the fact that the models employed here
demonstrate improved performance compared to existing approaches in automating theidentification of negotiation utterances in learners' discourse during collaborative problemsolving. Additionally, the study presents a unique context by focusing on the engineering
estimation problem, which is a frequent practice used by engineers and scientists whenexact data and precise governing equations are unavailable (Mahajan, 2014).

Figure 1. The figure shows a collaboration between two students, and the discourse data iscaptured and converted to transcripts. This transcript is then analysed by the ML models forgenerating discourse analytics for teachers and personalised feedback for students, which is
first delivered to the teacher, who has the agency to modify it before delivering it to students.
The study constitutes a part of a larger research that aims to enhance productive
behaviours in collaborations by providing students with personalised feedback (Fig. 1). Here,the teacher will have access to the discourse analytics affording them the agency to modify
the feedback provided by the models. The overarching goal is to attain insightfulperspectives on the evolution of different and essential collaborative learning mechanisms
during collaboration. This can help educators design scaffolds to enable learners to engage
in productive behaviours.
The paper is structured as follows: Section 2 presents the background and literaturesynthesis, followed by the OELE description in Section 3. We then delineate the researchgoal and procedure in Section 4. Subsequently, we present the results in Section 5, leading
to the discussion and conclusion in Section 6.

442

2. Background and Related WorkIn this section, we first highlight the significance of negotiations in the process ofcollaborative problem-solving. The second sub-section discusses the theoretical CPS
framework that we have employed for the manual coding of the data. Then, we discuss thestate-of-the-art text classification algorithms. Finally, we discuss different studies that aim todetect negotiation in discourse data.

2.1 Negotiation and Problem-solving
Effective collaboration in Computer Supported Collaborative Learning (CSCL) researchnecessitates negotiation when dealing with divergent opinions and ideas from peers.
Negotiation plays a crucial role in complex problems and open domains that require criticalexamination, the convergence of ideas, and contributions from all collaborators to arrive at asuitable solution (Carell & Herrmann, 2009). In this context, it can be defined in various
ways, including resolving conflicts, attempting to agree on goals, and critically examining
different perspectives (Baker, 1994; Fleck et al., 2009). Learners engaged in a negotiationmust present their ideas, defend their positions, and use various strategies to reach an
agreement. This process must ultimately result in a shared understanding of the problem toensure that all collaborators have reasoned and agreed upon a common ground in problemsolving (Beers, 2006). Through negotiation, learners can engage in meaningful discussions
and sensemaking, exchange ideas, learn from their peers, and contribute to productive
outcomes. It is particularly vital in OELEs, where learners need to explore new ideas,
tools and critically examine complex problems. Therefore, it is essential to identify andunderstand negotiation utterances and design scaffolds to foster it among learners to
support their problem-solving abilities.

2.2 Collaboration Learning Mechanisms Framework to identify negotiation
To identify and comprehensively understand instances of negotiation among learners within
MEttLE (Kothiyal & Murthy, 2018), we employed the Collaboration Learning Mechanisms(CLM) framework developed by Fleck et al. (2009). This framework centres on two pivotal
mechanisms for collaborative learning within open problem contexts: coordination anddiscussion. Within these mechanisms, the CLM framework emphasises how specific actionsand verbal interactions represent different aspects of collaboration and how they collectively
contribute to productive outcomes in collaborative problem-solving activities. The
collaborative discussion mechanism entails the exchange of information, ideas, andnegotiation of meaning, while the coordination mechanism underscores joint attention,awareness, and narration. This framework helped us identify and understand the mechanismof negotiation that involved verbal discourse of disagreeing, conflicting, proposing
alternatives, criticising, explaining, and justifying one's idea, and actions such as undoing,
deleting, nodding head to depict disagreement, etc. CLM defines negotiation as aconversation that involves all parties being critically and constructively engaged with eachother’s proposals for joint consideration. We then further added 9 more mechanisms ofcollaboration based on our data. To ensure the reliability and validity of our approach, two
researchers independently calculated the inter-rater reliability for 10% of the dataset,
resulting in a robust kappa score of 0.7, indicative of strong agreement between the
researchers. Among the various collaborative mechanisms, we observed that learners were
engaged in negotiation approximately 31.5% of the time during their problem-solving
activities.
Table 1. The table presents the episode of Negotiation from Group 2, where both thelearners S3 and S4 read the problem statement and immediately get into a discussion on
how to solve it. S3 is seen to propose a solution method that S4 does not easily accept.
Student Utterences
Time
Codes

443

S3
S4

S3

S4
S3

lets just skip the part like there will be somedelays ..assume that the entire time we have the same
velocity like it was 0, then it reaches to its maximum ...
(shakes head in disagreement and then rereads
Problem statement and says ) So the entire track is 50
meters. So you have to consider that it will start from astop position initially
Yes, it will start from stop position but .what I am saying
is it will not gradually increase from the stop position .
For this problem, let us just assume that it will directly
reach the end point ..
Yes but (interrupts S3) But there needs to be someassumption behind this right? What are you basing yourassumption on?
I am just reducing all the functions like..whatever the 3rdparameters

0:24:18

Negotiation

0:24:32

Negotiation

0:24:50

Negotiation

0:25:06

Negotiation

0:25:11

Negotiation

2.3 Text Classification
Text classification, a machine-learning technique used to categorise open-ended text into
predetermined categories, has gained significant attention in research. Its application tocoding discourse data addresses the time and labour-intensive nature of manual coding
processes. While manual coding can consume days of effort, machine learning models canaccomplish the same task within a few minutes, thereby substantially reducing the required
time and resources.
Recent advancements in large language models have presented opportunities toenhance classification accuracy further. Among these models, Bidirectional Encoder
Representations from Transformers (BERT) (Devlin et al., 2018) have emerged as a notabletext representation model specifically designed for natural language processing. BERT
distinguishes itself from previous models by generating dynamic and contextualised word
representations through unique training tasks, such as masked-language modelling andnext-sentence prediction. Leveraging the mechanism of transfer learning, BERT can bereadily adapted to downstream tasks with relative ease.
In addition to BERT, another influential language model is RoBERTa (Liu et al.,2019). RoBERTa, based on transformer architecture, has garnered attention for its
remarkable language generation capabilities. Pretrained on a massive corpus of text data,
RoBERTa exhibits exceptional performance across various natural language processing
tasks, including text classification. By capturing contextual dependencies and semantic
relationships between words and phrases, RoBERTa effectively comprehends andgenerates coherent text.
Another prominent language model worth mentioning is GPT2 (Generative
Pretrained Transformer 2). GPT2 is a transformer-based model known for its impressive
language generation capabilities. It is trained on a massive corpus of text data and has
demonstrated exceptional performance in a wide range of natural language processing
tasks, including text classification. GPT2 captures the contextual dependencies andsemantic relationships between words and phrases, enabling it to understand and generate
coherent text.
To summarise, the availability of powerful language models such as BERT, GPT2, and
RoBERTa offers researchers the opportunity to enhance the accuracy of text classification.
Their proficiency in capturing contextual information, coupled with their transfer learning
capabilities and demonstrated success across various natural language processing tasks,
positions them as promising options for automating the categorisation of discourse data.

2.4 Collaboration Analytics

444

Collaboration Analytics pertains to the methodologies and strategies employed in theautomated or semi-automated capture, analysis, mining, and extraction of data concerninginteractions among collaborators (Schneider et al., 2021). We ground our work in theexisting domain of linguistic modelling of collaborative problem-solving. In this sub-section,
we shall discuss the existing research that makes use of advanced NLP methods to analyse
discourse data in CPS. This discourse data emerge either from the text chats or istranscribed from the speech) and is used to model various skills during CPS, including
negotiation, regulation, argumentation etc.
The paper by Flor et al. (2016) presents a study that investigates the use of
automated techniques for accurately categorising interactions in collaborative problemsolving within simulated science tasks. It applies natural language processing (NLP)
methods to analyse and classify these interactions using Naïve Bayes (NB) and Hidden
Markov Models (HMM). The results reveal the effectiveness of the automated classification
models in accurately identifying and categorising various types of interactions, achieving an
average accuracy of 59.2%.
Pugh, Rao, Stewart, and D'Mello (2022) investigated the feasibility of detecting sevencognitive and social CPS skills (including sharing information, negotiation, etc.) in classroom
and lab settings. The participants were middle school kids who collaboratively solved mathand physics problems. They achieved an AUROC value of 0.78 in the classroom setting and0.83 in the lab setting using BERT.
The work by Hao et al. (2017) focuses on developing an automated annotation
system called CPS-rater. This tool annotates the discourse into various CPS skills likenegotiation, regulation etc. They have used different models, including Random Forest (RF)and NB. Linear chain conditional random field emerged as the best model, achieving an
accuracy of 0.732 and a kappa value of 0.636.
To summarise, several studies have contributed to the field of automated analysis
and classification of interactions in collaborative problem-solving (CPS) activities. Building
upon these previous works, our study aims to extend the existing research by introducing
novel contributions. Firstly, we focus on the specific context of the engineering estimation
problem within collaborative problem-solving. This contextual specificity provides uniqueinsights and findings applicable to the domain of engineering problem-solving. Secondly, ourwork employs advanced machine learning models that have demonstrated improved
performance compared to existing approaches. By utilising these models, we aim toenhance the accuracy and efficiency of automated identification and categorisation ofnegotiation utterances in learners' discourse during collaborative problem-solving. Theseadvancements contribute to the broader understanding and advancement of automatedanalysis techniques in the domain of CPS, addressing the need for more precise andeffective analysis of collaborative interactions.

3. Learning Environment
The Modelling-Based Estimation Learning Environment (MEttLE), as shown in Figure 2, isan open-ended learning environment designed to scaffold novice learners in their estimation
problem-solving. The tool's five sub-goals trigger a model-based estimation process that isessential for solving the estimation problem (Figure 2a). The three-phased model-building
sub-goals, which include functional, qualitative, and quantitative aspects, help learners
create, contextualise, and evaluate models of complex problems, one calculation, and one
evaluation. MEttLE also includes metacognitive prompts (Figure 2b) that encourage learners
to reflect on their models and problem-solving processes, as well as simulators (Figure 2c),
hints, an info centre, guide me, question prompts, and a problem map to facilitate themodelling process. Novice learners have the flexibility to choose any path and revisit any
sub-goals at any time in MEttLE.

445

Figure 2. Overview of MEttLE Interface with sub-goals (2a), Detailed sub-goal with
Metacognitive Prompts and other scaffolds (2b) and Simulator-4(2c).
To solve the estimation problem in this study, the solver is tasked with estimating thepower of a car while comprehending the underlying problem system. To achieve this, theymust analyse how the car behaves under the given operating conditions, determine thedominant parameters for these conditions, and create an equation involving the dominantparameters. Additionally, they must make assumptions and approximations to create anddetermine a simplified model of the problem context and its corresponding mathematicalequation. One noteworthy aspect of MEttLE is its open design, which provides immense
opportunities for collaborative learning wherein learners undertake meaningful discussions
and negotiations to solve this problem.

4. Methods
4.1 Participants
The participants in this study were three dyads comprising two female and four malestudents (age range of 18-21 years) from 3rd or 4th-year undergraduate engineeringcolleges in Mumbai. Students from mechanical or electronics engineering were chosen as
participants, as the problem in MEttLE required knowledge specific to these disciplines. The
students were paired randomly based on their availability based on a pre-study Google form
that collected their demographic details. IRB approval from our institution was sought.

4.2 Procedure and Data Collection and Data Analysis
The study was conducted in a research lab setting where the three engineering
undergraduate dyads participated on different days. The researcher explained the study
goal, the procedure, and the data that would be collected, after which the participants signed
the consent form. The lab setup involved a single computer with MEttLE for bothparticipants, who had to solve the estimation problem in it collaboratively. They were asked
to articulate their views explicitly during the collaboration. On average, the time taken bypairs to interact with MEttLE was 90 minutes. Data was collected from multiple sources, likescreen recording using OBS Studio (OBS, 2021), which ran on a laptop containing MEttLE
as well as audio and video data; however, for this study, we only used the transcription of
their audio data. Learners’ discourse data was then passed to an AI tool called Otter.ai forspeech-to-text transcription. Further, researchers coded the data using the CLM framework
to identify learners’ negotiation instances.

4.3 Feature extraction, model training, and evaluation446

In this study, we employed three traditional machine learning classifiers, namely Logistic
Regression (LR), NB, and RF. We used the features obtained via TF-IDF (Spärck Jones,
1972) and the Word2Vec (Mikolov et al., 2013) approach. The intention of using theseclassifiers was that they are employed in most of the research in the domain of learning
analytics. Additionally, we used three state-of-the-art pre-trained neural networks for natural
language processing that include BERT, RoBERTa, and GPT2. These models were trained
for binary classification problem, i.e. to classify discourse data into two categories: thepresence of negotiation (labelled as 1) and the absence of negotiation (labelled as 0) We
shall first discuss the feature extraction, model training of traditional machine learningmodels before delving into large language models. TF-IDF Feature Extraction:
We employed the TfidfVectorizer from thesklearn.feature_extraction.text module to convert the text data into numerical features. The
vectoriser was configured with parameters such as a minimum document frequency of 5 anda maximum of 10,000 features. This ensured that only frequently occurring words wereconsidered, and the feature dimensionality was controlled.
In addition to TF-IDF, we trained a Word2Vec model using the gensim library. The
Word2Vec model learned word embeddings from the training sentences with specifichyperparameters, such as size = 100, window = 5, min_count = 1, and workers = 4. These
parameters were selected based on prior experimentation and domain knowledge.
The training data was preprocessed and divided into two sets: one for the TF-IDFbased approach and the other for the Word2Vec-based approach. For TF-IDF, the training
data was transformed into TF-IDF features using the fitted vectoriser. For Word2Vec, thetraining data sentences were tokenised and converted into word embeddings using thetrained Word2Vec model. The target variable was created by encoding the labels as
integers.
We trained three classifiers, LR, RF, and NB, on the training data. We evaluated theperformance of the classifiers on the test data using various metrics like classification
accuracy, F1-score, and kappa value ( explained at the end of this section).
For BERT, we tokenised the texts using the Hugging Face Transformers library's tokeniserfunction and converted them to input features with padding and truncation. We split the data
into a training set and a validation set and labelled each instance of discourse as either
negotiation or non-negotiation. We then created training and validation datasets as PyTorch
TensorDatasets, using the input features and labels. We fine-tuned the pre-trained BERT
model for our specific task of classifying negotiation and non-negotiation discourse data. We
defined the training parameters, such as batch size, number of epochs, and learning rate,
and used the AdamW optimiser and the Cross-Entropy Loss function. In our case, we
considered a batch size of 16, and the number of epochs was taken to be four. The learning
rate was set as 2*(10)-5.
We then created data loaders for the training and validation datasets and trained themodel on the training set for a fixed number of epochs using mini-batches. During eachepoch, we optimised the model's weights using backpropagation and stochastic gradientdescent.
To evaluate the performance of the model, we used the validation set, which the model had
not seen during training. The same procedure was applied for RoBERTa and GPT2 as well,and the hyperparameters were also chosen to be the same. For performance evaluation, we
chose the following four metrics:
1. Classification accuracy (CA): the ratio of the number of correct classifications to thetotal number of classifications.
2. F score (F1): is the harmonic mean of recall and precision.
3. Kappa (κ): measures agreement between the actual and the predicted labels byconsidering the by-chance prediction. κ is calculated using Eq. (1), where P0 is theoverall accuracy of the model and Pe is the measure of the agreement between themodel predictions and the actual class values as if happening by chance.
௉బ ି௉೐
ଵି௉బ……… (1)447

5. Result
This section describes the results of the prediction models developed for classification.
Table 2. The evaluation metric of performance of ML models on “negotiation” classification.
Text Features Classifier CA
F1
Kappa
TF-IDFLR
0.818 0.777
0.283
NB
0.794 0.712
0.070
RF
0.903 0.893
0.664
Word2Vec
LR
0.784 0.6890 0NB0.369 0.355
0.076
RF
0.873 0.853
0.530
BERT
0.907 0.906
0.722
RoBERTa
0.889 0.889
0.675
GPT2
0.924 0.924
0.781
The performance of various machine learning and deep learning models was evaluated on
the task of classifying discourse data into negotiation and non-negotiation. The models weretrained on a dataset with 1989 instances and tested on a dataset with 686 instances, andthis data split was done on a group level, i.e., data from two groups was used for training,
and it was tested on the discourse of the remaining group. The dataset was preprocessedusing standard NLP techniques like tokenisation, stop-word removal, and stemming. Twotypes of features were used for training the traditional machine learning models: TF-IDF and
Word2Vec.
The machine learning models used in this study were LR, NB, and RF. The deeplearning models used were BERT, RoBERTa, and GPT2.
The results of the experiment are summarised in Table 2. The performance of the models
was evaluated using classification accuracy, the F1-score, and the kappa metric (explainedin the previous section). The results show that deep learning models outperformed traditional
machine learning models in terms of F1-score and kappa value. GPT2 achieved the best F1score and kappa values of 0.924 and 0.781, respectively.
The NB model exhibited the lowest performance, achieving an F1-score of 0.355
when utilising Word2Vec features. Similarly, LR demonstrated the least favourable kappa
score of 0 when employing the Word2Vec features, suggesting that its classification results
were not significantly different from those that could be attributed to random chance alone.
In addition, the results indicate that the choice of feature representation also plays a crucialrole in the performance of the models. The results demonstrate that the traditional machinelearning models showed better performance with TF-IDF features.
Overall, the results of this study suggest that deep learning models like BERT,RoBERTa, and GPT2 are more effective than traditional machine learning models for thetask of classifying discourse data into negotiation and non-negotiation. The choice of feature
representation also plays a significant role in the performance of the models

6. Discussion and Conclusion
In conclusion, this study proposes an approach for automating the identification ofnegotiation in learners' discourse during collaboration. By leveraging language models like
BERT, RoBERTa, and GPT, we accurately identified negotiation utterances in learners'
discourse while they solved complex engineering estimation problems in an OELE called
MEttLE with high precision and recall.
The practical applications of our research have the potential to impact educationalsettings greatly. We envision the utilisation of language-based collaboration analytics (CA)models in authentic educational environments for targeted interventions to enhance CPS
skills. One key application involves the deployment of automated reports to teachers whooversee multiple groups of students engaged in CPS activities. These reports, accessible448

through a teacher dashboard, could provide insights into each group's engagement withvarious aspects of CPS, such as constructing shared knowledge. This functionality
empowers teachers to identify groups requiring additional support and allocate theirresources effectively. Moreover, teachers can pinpoint individual students' strengths andweaknesses, facilitating the establishment of personalized improvement goals.
Furthermore, the proposed approach isn't limited to teacher-facing analytics; it canalso provide learner-facing feedback to foster CPS skill development. CA models could offer
individual team members insights into their contributions and demonstration of different CPS
skills. This personalized feedback enhances self-awareness, self-reflection, and theevaluation of strengths and weaknesses among learners. Such feedback facilitates tracking
skill improvement across multiple collaborative engagements.
Nonetheless, our study is not without limitations. Our results suggest that ourapproach can accurately identify negotiation utterances in this specific context, but thegeneralizability of our method to other learning environments and contexts needs to befurther investigated. Additionally, the study relied on language models to identify negotiation,
which may not capture nonverbal cues and context-specific language use. Future research
could explore the use of multimodal methods to identify negotiation in discourse.

Acknowledgements
The authors would like to thank the volunteers for their participation in the study. The authors
acknowledge the Research Scholars of the Indian Institute of Technology (IIT) Bombay for
their suggestions and input; and the Indian Ministry of Education for financial support in
conducting this research.

References
Baker, M. (1994). A model for negotiation in teaching-learning dialogues. Journal of Interactive
Learning Research, 5(2), 199.
Baker, T., & Clark, J. (2010). Cooperative learning – a double-edged sword: A cooperative learningmodel for use with diverse student groups. Intercultural Education, 21, 257–268.
Beers, P. J., Boshuizen, H. P., Kirschner, P. A., & Gijselaers, W. H. (2006). Common ground, complex
problems, and decision-making. Group decision and negotiation, 15(6), 529-556.
Carell, A., & Herrmann, T. (2009). Negotiation-Tools in CSCL-Scenarios-Do they have a valid use?
Chan, T. W., Roschelle, J., Hsi, S., Kinshuk, Sharples, M., Brown, T., Patton, C., Cherniavsky, J.,
Pea, R., Norris, C., Soloway, E., Balachefe, N., Scardamalia, M., Dillenbourg, P., Looi, C.-K.,
Milrad, M., & Hoppe, U. (2006). One-to-one technology-enhanced learning: An opportunity for
global research collaboration. Research and Practice in Technology-Enhanced Learning, 1(1), 329.
computer-supported communication of knowledge: cognitive, computational, and educational
perspectives. Springer, New York, 1-10.
Conger, S., & Loch, K. D. (1995). Ethics and computer use. Communications of the ACM, 38(12), 3032.
De Dreu, C. K., Weingart, L. R., & Kwon, S. (2000). Influence of social motives on integrativenegotiation: a meta-analytic review and test of two theories. Journal of personality and social
psychology, 78(5), 889.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectionaltransformers for language understanding. arXiv preprint arXiv:1810.04805.
Emara, M., Rajendran, R., Biswas, G., Okasha, M., & Elbanna, A. A. (2018). Do Students' Learning
Behaviors Differ when they Collaborate in Open-Ended Learning Environments?. Proceedings of
the ACM on HumanComputer Interaction, 2(CSCW), 1-19.
Fiore, S. M., Graesser, A., Greiff, S., Griffin, P., Gong, B., Kyllonen, P., ... & von Davier, A. (2017).
Collaborative problem solving: Considerations for the national assessment of educational
progress.
Fischer, F., Kollar, I., Haake, J. M., & Mandl, H. (2007). Perspectives on collaboration scripts.
Scripting449

Fleck, R., Rogers, Y., Yuill, N., Marshall, P., Carr, A., Rick, J., & Bonnett, V. (2009, November).
Actions speak loudly with words: unpacking collaboration around the table. In Proceedings of the
ACM international conference on interactive tabletops and surfaces (pp. 189-196).
Flor, M., Yoon, S. Y., Hao, J., Liu, L., & von Davier, A. (2016, June). Automated classification ofcollaborative problem solving interactions in simulated science tasks. In Proceedings of the 11th
workshop on innovative use of NLP for building educational applications (pp. 31-41).
Freeman, L., & Greenacre, L. (2010). An examination of socially destructive behaviors in group work.
Journal of Marketing Education, 33, 5–17.
Hao, J., Chen, L., Flor, M., Liu, L., & von Davier, A. A. (2017). CPSǦRater: Automated sequential
annotation for conversations in collaborative problemǦsolving activities. ETS Research Report
Series, 2017(1), 1-9.
Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable
collaborative problem solving skills. Assessment and teaching of 21st century skills: Methods andapproach, 37-56.
Jeong, H., & Hmelo-Silver, C. E. (2016). Seven affordances of computer-supported collaborative
learning: How to support collaborative learning? How can technologies help? Educational
Psychologist, 51(2), 247-265.
Johnson, D. W., Johnson, R. T., & Smith, K. (2007). The state of cooperative learning inpostsecondary and professional settings. Educational Psychology Review, 19, 15–29.
Jonassen, D., Strobel, J., & Lee, C. B. (2006). Everyday problem-solving in engineering: Lessons forengineering educators. Journal of Engineering Education, 95(2), 139-151.
KHWAJA, U., & MURTHY, S. Understanding Learners’ Negotiation Processes During Ill-Structured
Engineering Estimation Problem-Solving.
Kirschner, F., Paas, F., & Kirschner, P. A. (2009). A cognitive load approach to collaborative learning:
United brains for complex tasks. Educational psychology review, 21(1), 31-42.
Kothiyal, A., & Murthy, S. (2020). Disciplinary Model-Based Reasoning and Metacognition Underlies
Good Estimation Performance by Engineering Undergraduates.
Lai, H., Lin, W. J., & Kersten, G. E. (2010). The importance of language familiarity in global businessenegotiation. Electronic Commerce Research and Applications, 9(6), 537-548.
Li, M., & Campbell, J. (2008). Asian students’ perceptions of group work and group assignments in a
New Zealand tertiary institution. Intercultural Education, 19, 203–216.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). Roberta: A robustly
optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Mahajan, S. (2014). The art of insight in science and engineering: mastering complexity (p. 408). The
MIT Press.
Maldonado, H., Klemmer, S., & Pea, R. (2009). When is collaborating with friends a good idea?
Insights from design education. OBS (2021). Open Broadcasting Software. Retrieved fromhttps://www.obsproject.org/
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations ofwords and phrases and their compositionality. Advances in neural information processing
systems, 26.
Pauli, R., Mohiyeddini, C., Bray, D., Michie, F., & Street, B. (2008). Individual differences in negative
group work experiences in collaborative student learning. Educational Psychology, 28, 47–58.
Pugh, S. L., Rao, A., Stewart, A. E., & D'Mello, S. K. (2022, March). Do Speech-Based Collaboration
Analytics Generalize Across Task Contexts?. In LAK22: 12th International Learning Analytics
and Knowledge Conference (pp. 208-218).
Richardson, G. P., & Andersen, D. F. (1995). Teamwork in group model building. System Dynamics
Review, 11(2), 113-137.
Schneider, B., Dowell, N., & Thompson, K. (2021). Collaboration analytics—current state andpotential futures. Journal of Learning Analytics, 8(1), 1-12.
Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval.
Journal of documentation, 28(1), 11-21.
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2013). Teacher interventions in asynchronous, co-located CSCL setting: Analyzing focus, means, and temporality. Computers in
Human Behavior, 29, 1377–1386.
Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction
in computer-supported collaborative learning. Computers & education, 46(1), 71-95

4506KLK -/ HW DO (GV   3URFHHGLQJV RI WKH VW ,QWHUQDWLRQDO &RQIHUHQFH RQ &RPSXWHUV LQ (GXFDWLRQ$VLD3DFLILF 6RFLHW\ IRU &RPSXWHUV LQ (GXFDWLRQ$ SDJH MXPS UHFRPPHQGDWLRQ PRGHO EDVHG
RQ GLJLWDO WH[WERRN FRQWHQWV DQG VWXGHQW
ORJ GDWD:HQKDR :$1* D 1DWVXPL <$0$0272 D )X]KHQJ =+$2 D (WVXNR .80$0272 E
=LFKHQJ .$1* F &KHQJMLX <,1 G
D*UDGXDWH 6FKRRO RI 6\VWHP ,QIRUPDWLFV .REH 8QLYHUVLW\ -DSDQ
E,QIRUPDWLRQ 6FLHQFH DQG 7HFKQRORJ\ &HQWHU .REH 8QLYHUVLW\ -DSDQF)DFXOW\ RI 6\VWHPV 6FLHQFH DQG 7HFKQRORJ\ $NLWD 3UHIHFWXUDO 8QLYHUVLW\ -DSDQ
G
5HVHDUFK ,QVWLWXWH IRU ,QIRUPDWLRQ 7HFKQRORJ\ .\XVKX 8QLYHUVLW\ -DSDQ\LQFKHQJMLX#PN\XVKXXDFMS$EVWUDFW ,Q WKH DQDO\VLV RI VWXGHQW OHDUQLQJ GDWD UHVHDUFKHUV IRXQG WKDW VWXGHQWVRIWHQ VZLWFK SDJHV EXW FDQQRW DFFXUDWHO\ ILQG WKH WDUJHW SDJH 7KHUHIRUH LQ WKLV VWXG\D SDJH UHFRPPHQGDWLRQ PRGHO WR SURYLGH VWXGHQWV ZLWK WKH WDUJHW SDJH IRU MXPSLQJ WRLPSURYH WKH HIILFLHQF\ RI VWXGHQW SDJH MXPSLQJ EHKDYLRU ZDV GHYHORSHG 7KH
UHFRPPHQGDWLRQ PRGHO OLQHDUO\ FRPELQHV WKH FRQWHQWEDVHG UHFRPPHQGDWLRQ PRGHOXVLQJ WHUP IUHTXHQF\±LQYHUVH GRFXPHQW IUHTXHQF\ 7),') DQG WKH UHFRPPHQGDWLRQPRGHO EDVHG RQ VWXGHQW ORJ GDWD DQG GHWHUPLQHV WKH RSWLPDO ZHLJKW RI OLQHDUFRPELQDWLRQ WKURXJK JUDGLHQW GHVFHQW )LQDOO\ WKH DFFXUDF\ RI WKH PRGHO ZDVHYDOXDWHG E\ FRPSDULQJ LW ZLWK H[SHUWJHQHUDWHG UHFRPPHQGDWLRQ RXWFRPHV.H\ZRUGV 3DJH MXPS UHFRPPHQGDWLRQ 7),') /RJ GDWD

 ,QWURGXFWLRQ
2QOLQH OHDUQLQJ KDV EHHQ RQ WKH LQFUHDVH LQ WKH ODVW WZR GHFDGHV 0DUWLQ   $V DQ
LPSRUWDQW SDUW RI RQOLQH OHDUQLQJ HERRNV DQG RWKHU GLJLWDO WHDFKLQJ PDWHULDOV DUH JUDGXDOO\UHSODFLQJ WUDGLWLRQDO SDSHU WH[WERRNV GXH WR WKHLU ORZ FRVW DQG EHWWHU SRUWDELOLW\ :DQJ -
6KLPDGD $   +RZHYHU UHVHDUFK KDV IRXQG WKDW VRPH VWXGHQWV FDQ UHPHPEHU OHVVFRQWHQW ZKHQ UHDGLQJ WH[WERRNV RQ VFUHHQV WKDQ ZKHQ UHDGLQJ SDSHU WH[WERRNV DQG WKHLUXQGHUVWDQGLQJ RI WKH FRQWHQW LV VKDOORZHU WKDQ ZKHQ UHDGLQJ SDSHU WH[WERRNV /DXWHUPDQ 7
$FNHUPDQ 5   $W WKH VDPH WLPH WKH VXGGHQ RXWEUHDN RI &29,' KDV IRUFHG VFKRROVDURXQG WKH ZRUOG WR DGRSW RQOLQH WHDFKLQJ PHWKRGV RQ D ODUJH VFDOH 5HVHDUFK E\ (PPD 'RUQ
DQG %U\DQ +DQFRFN  VKRZV WKDW &29,' KDV FDXVHG VLJQLILFDQW GLVUXSWLRQ WR WKH
HGXFDWLRQ V\VWHP DQG VWXGHQWV¶ DFDGHPLF SHUIRUPDQFH KDV GHFOLQHG WR YDU\LQJ GHJUHHV
5HVHDUFKLQJ DQG LPSURYLQJ RQOLQH OHDUQLQJ KDV EHFRPH DQ LPSRUWDQW IDFWRU LQ LPSURYLQJ
VWXGHQWV¶ RQOLQH OHDUQLQJ RXWFRPHV *DUFtD0RUDOHV   ,Q WKH DQDO\VLV RI RQOLQH OHDUQLQJEHKDYLRU <LQ HW DO  IRXQG WKDW VWXGHQWV ZLWK EHWWHU DFDGHPLF SHUIRUPDQFH KDYH D
VLJQLILFDQWO\ KLJKHU %DFNLQJ 7UDFN 5DWH %75 WKURXJK WKH DQDO\VLV RI OHDUQLQJ EHKDYLRU
SDWWHUQV EDVHG RQ VWXGHQW EHKDYLRU GDWD +DYHUNDPS  DOVR VKRZHG WKURXJK UHVHDUFKWKDW EDFNWUDFNLQJ EHKDYLRU KDV D XQLTXHO\ SRVLWLYH HIIHFW RQ LPSURYLQJ FRQWHQW XQGHUVWDQGLQJZKHQ UHDGLQJ GLJLWDO PDWHULDOV +RZHYHU LQ WKH DQDO\VLV RI OHDUQLQJ GDWD IURP  VWXGHQWVZH QRWHG WKDW PRUH WKDQ  RI EDFNWUDFNLQJ EHKDYLRU UHTXLUHV PRUH WKDQ WZR MXPSV WRQDYLJDWH WR WKH WDUJHW SDJH LQGLFDWLQJ WKDW VWXGHQWV¶ EDFNWUDFNLQJ EHKDYLRU LV LQHIILFLHQW
7R LPSURYH WKH HIILFLHQF\ RI EDFNWUDFNLQJ EHKDYLRU DQG SURYLGH VWXGHQWV ZLWK QDYLJDWLRQIRU SDJH MXPSLQJ WKLV VWXG\ SURSRVHV D SDJH UHFRPPHQGDWLRQ PRGHO WKDW FRPELQHV
FRQWHQWEDVHG DQG XVHU GDWDEDVHG PHWKRGV WR KHOS VWXGHQWV ILQG WKH WDUJHW SDJH IRU MXPSLQJ
DQG LPSURYH OHDUQLQJ HIILFLHQF\ ZKHQ UHDGLQJ HERRNV

451

 /LWHUDWXUH UHYLHZ
 7),') DOJRULWKP7),') LV D FRPPRQO\ XVHG ZHLJKWLQJ WHFKQLTXH IRU LQIRUPDWLRQ UHWULHYDO DQG WH[W PLQLQJZKLFK FDQ UHIOHFW WKH LPSRUWDQFH RI D ZRUG WR D GRFXPHQW RU D FRUSXV7KH 7),') DOJRULWKP ZDV ILUVW SURSRVHG E\ 6SDUFN -RQHV  WR HYDOXDWH WKHSHUIRUPDQFH RI LQIRUPDWLRQ UHWULHYDO V\VWHPV /DWHU LW LV ZLGHO\ XVHG LQ YDULRXV QDWXUDOODQJXDJH SURFHVVLQJ WDVNV )RU H[DPSOH :X HW DO  DSSOLHG 7),') WR UHOHYDQFH
GHFLVLRQ 6KDK]DG 4DLVHU HW DO  XVHG WKH 7),') DOJRULWKP WR H[DPLQH WKH UHOHYDQFH RINH\ZRUGV DQG GRFXPHQWV LQ WKH FRUSXV =KRX HW DO  DSSOLHG 7),') WR FOXVWHU QHZV'XH WR LWV VLPSOH PDWKHPDWLFDO FDOFXODWLRQ IRUPXOD ORZ FRPSXWDWLRQDO FRPSOH[LW\ DQG
UHODWLYHO\ JRRG DFFXUDF\ LQ FODVVLILFDWLRQ :X  WKLV VWXG\ DOVR XVHV WKLV DOJRULWKP LQ WKHUHFRPPHQGDWLRQ PRGHO VWXGHQW ORJ GDWD DQDO\VLV LQ (GXFDWLRQ:LWK WKH GHYHORSPHQW RI ZHEEDVHG HGXFDWLRQ HGXFDWLRQDO GDWD PLQLQJ KDV EHFRPH D
SURPLVLQJ ILHOG 5RPHUR   DQG PDQ\ VWXGLHV KDYH EHHQ FRQGXFWHG RQ PLQLQJ VWXGHQW ORJ
GDWD ,Q PLQLQJ VWXGHQW ORJ GDWD <LQ HW DO  PLQHG OHDUQLQJ EHKDYLRU SDWWHUQV RI GLIIHUHQWW\SHV RI VWXGHQWV IURP GDWD RI VWXGHQWV UHDGLQJ GLJLWDO ERRNV =KDR HW DO  SUHGLFWHG
VWXGHQWV¶ DFDGHPLF SHUIRUPDQFH XVLQJ  SUHGLFWLRQ DOJRULWKPV EDVHG RQ VWXGHQW GDWD IURP DQ
HERRN V\VWHP 5LHVWUD*RQ]iOH] HW DO  SUHGLFWHG VWXGHQW SHUIRUPDQFH LQ WKH HDUO\VWDJHV RI D FRXUVH E\ DQDO\]LQJ ORJ GDWD IURP DQ /06 V\VWHP *REHUW  HYDOXDWHG
VWXGHQWV¶ DELOLWLHV LQ VFLHQFH LQTXLU\ WKURXJK WKHLU ORJ GDWD
,Q WKH UHVHDUFK GLUHFWLRQ RI UHFRPPHQGLQJ VRPHWKLQJ WR VWXGHQWV EDVHG RQ VWXGHQW ORJ
GDWD $KHU  UHFRPPHQGHG RQOLQH FRXUVHV WKURXJK 0RRGOH V\VWHP¶V VWXGHQW FRXUVHEURZVLQJ GDWD 5HGG\  UHFRPPHQGHG WKH OHDUQLQJ VHTXHQFH RI FRXUVHV EDVHG RQ WKHFRXUVHV DQG VWXGHQWV¶ JUDGHV
7KHVH VWXGLHV DOO IRFXV RQ WKH SHUVRQDOL]HG FRXUVH RU OHDUQLQJ SDWK UHFRPPHQGDWLRQVIRU VWXGHQWV ZKLOH WKLV SDSHU IRFXVHV RQ UHFRPPHQGLQJ SDJHV RI GLJLWDO ERRNV DW D PRUH
VSHFLILF OHYHO

 'HVLJQ DQG GHYHORSPHQW RI SDJH MXPS UHFRPPHQGDWLRQ PRGHO,Q WKLV VHFWLRQ WKH VSHFLILF FDOFXODWLRQ PHWKRG RI 7),') LQ FRQWHQWEDVHG UHFRPPHQGDWLRQPRGHOV WKH PHWKRG RI PLQLQJ SDJH UHOHYDQFH IURP VWXGHQW ORJ GDWD DQG WKH FRQVWUXFWLRQPHWKRG RI K\EULG PRGHOV DUH PDLQO\ LQWURGXFHG )LQGLQJ SDJH DVVRFLDWLRQV EDVHG RQ WH[WERRN FRQWHQW
7KLV VWXG\ XVHV 7),') DQG FRVLQH VLPLODULW\ WR HYDOXDWH WKH UHOHYDQFH RI SDJHV 7),') LV DQ
DQDO\WLFDO PHWKRG IRU HYDOXDWLQJ WKH LPSRUWDQFH RI ZRUGV LQ D GRFXPHQW 7) UHSUHVHQWV WKHIUHTXHQF\ RI D ZRUG LQ D GRFXPHQW DQG ,') UHSUHVHQWV WKH XQLTXHQHVV RI D ZRUG LQ DGRFXPHQW 5DPRV -   %\ FRPELQLQJ WKH WZR 7),') WDNHV LQWR DFFRXQW ERWK WKH QXPEHU
RI RFFXUUHQFHV DQG WKH XQLTXH YDOXH RI D ZRUG SURYLGLQJ D PRUH FRPSUHKHQVLYH HYDOXDWLRQ RI
WKH DVVRFLDWLRQ EHWZHHQ D ZRUG DQG D WH[W 7KH 7),') YDOXH LV FDOFXODWHG XVLQJ WKH IROORZLQJIRUPXOD𝑇𝐹 𝐸 𝑡, 𝑝

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑤𝑜𝑟𝑑 𝑡 𝑖𝑛 𝑝𝑎𝑔𝑒 𝑝 𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡 𝐸
𝑇𝑜𝑡𝑎𝑙 𝑤𝑜𝑟𝑑𝑠 𝑜𝑓 𝑝𝑎𝑔𝑒 𝑝 𝑖𝑛 𝐸
𝑝𝑐
𝐼𝐷𝐹 𝐸 𝑡log2
𝑓 𝐸|𝑡
𝑇𝐹𝐼𝐷𝐹 𝐸 𝑡, 𝑝
𝑇𝐹 𝐸 𝑡, 𝑝 ∗ 𝐼𝐷𝐹 𝐸 𝑡

452

7DEOH  )RUPXOD RI 7),')
(
SF
S
W
I (_W

(ERRN QXPEHU
7RWDO QXPEHU RI SDJHV RI ($ SDJH LQ ($ ZRUG
1XPEHU RI SDJHV WKDW W DSSHDU LQ (7KURXJK WKH 7),') YDOXH RI ZRUGV HDFK SDJH RI WKH WH[WERRN FDQ EH VLPSO\ YHFWRUL]HG
DQG WKHQ WKH FRVLQH VLPLODULW\ EHWZHHQ WH[WV FDQ EH FDOFXODWHG 7KH FORVHU WKH FRVLQH VLPLODULW\
EHWZHHQ WZR YHFWRUV LV WR  WKH PRUH VLPLODU WKH WH[WV DUH 5HSHDWLQJ WKH FDOFXODWLRQ SURFHVV
ZH FDQ JHW D VLPLODULW\ PDWUL[ 5Į EHWZHHQ WH[WV 0LQLQJ SDJH UHOHYDQFH EDVHG RQ VWXGHQWV¶ OHDUQLQJ GDWD
$FFRUGLQJ WR WKH UHVHDUFK UHVXOWV RI &KLED   WKH UHDGLQJ VSHHG HYDOXDWLRQ LQGH[ ZRUGVSHU PLQXWH ZSP IRU VWXGHQWV ZKR XVH (QJOLVK DV D VHFRQG ODQJXDJH ZKHQ UHDGLQJ (QJOLVKPDWHULDOV LV  ZSP %DVHG RQ WKLV ZSP WKH WLPH 7U UHTXLUHG IRU D VWXGHQW WR UHDG D SDJH
FDQ EH LQIHUUHG IURP ZSP DQG WKH WRWDO QXPEHU RI ZRUGV RQ WKH SDJH :V E\ WKLV IRUPXOD
𝑇𝑟

𝑊𝑠
𝑤𝑝𝑚
60𝑠

:KHQ D VWXGHQW SHUIRUPV D MXPS SDJH EHKDYLRU D SDJH VHTXHQFH ZLOO EH JHQHUDWHG
&RQVLGHULQJ WKDW WKH VWXGHQW ZLOO QRW UHDG DOO WKH FRQWHQW RQ WKH WDUJHW SDJH FRPSOHWHO\ ZH XVH
 WLPHV WKH UHDGLQJ WLPH 7U RI WKDW SDJH DV WKH VWDQGDUG GXUDWLRQ IRU WKDW SDJH)LJXUH  7KH PHWKRG RI MXGJLQJ SDJH UHOHYDQFH

)LJXUH  7KH WH[W VLPLODULW\ PDWUL[ 5ȕ:KHQ WKH VWXGHQW¶V VWD\ WLPH RQ D FHUWDLQ SDJH LQ WKH SDJH VHTXHQFH LV JUHDWHU WKDQ RU
HTXDO WR WKH VWDQGDUG GXUDWLRQ RI WKDW SDJH LW LV FRQVLGHUHG WKDW WKH VWXGHQW KDV IRXQG WKH
WDUJHW SDJH RI WKLV MXPS WKDW LV WKH SDJH LV UHODWHG WR WKH VWDUWLQJ SDJH RI WKH MXPS EHKDYLRUDQG LW LV UHIOHFWHG LQ WKH WH[W VLPLODULW\ PDWUL[ 5ȕ DV DQ LQFUHDVH RI  LQ WKH FRUUHVSRQGLQJ
HOHPHQW YDOXH 7KHQ HDFK HOHPHQW LV GLYLGHG E\ WKH VXP RI WKH HOHPHQWV LQ LWV URZ DQG
QRUPDOL]HG WR XSGDWH WKH PDWUL[ 5ȕ453 7KH FRQVWUXFWLRQ RI WKH UHFRPPHQGDWLRQ PRGHO
7KLV VWXG\ FRPELQHV WKH FRQWHQWEDVHG WH[W VLPLODULW\ PDWUL[ 5Į DQG WKH XVHU GDWDEDVHG WH[WVLPLODULW\ PDWUL[ 5ȕ OLQHDUO\ DFFRUGLQJ WR WKH IROORZLQJ IRUPXOD WR REWDLQ WKH WH[W VLPLODULW\ PDWUL[
5 Įȕ RI WKH K\EULG PRGHO ZKHUH Į DQG ȕ DUH REWDLQHG E\ JUDGLHQW GHVFHQW 7KHQ VRUWLQJ HDFKURZ RI WKH 5 Įȕ PDWUL[ LQ GHVFHQGLQJ RUGHU ZH FDQ REWDLQ WKH UHFRPPHQGDWLRQ OLVW IRU HDFKSDJH𝑅 𝛼, 𝛽𝛼𝑅𝛼𝛽𝑅𝛽7KLV SDSHU XVHV WKH IROORZLQJ IRUPXOD %DUWHOO  WR ILQG WKH RSWLPDO YDOXHV RI WKHSDUDPHWHUV Į DQG ȕ𝐽𝑖 𝛼, 𝛽∑𝑗≻𝑞 𝑘 𝑅𝑖𝑗 𝛼, 𝛽𝑖

𝑅𝑖𝑘 𝛼, 𝛽𝑅𝑖𝑘 𝛼, 𝛽∑𝑗≻𝑞 𝑘 𝑅𝑖𝑗 𝛼, 𝛽𝑖
1arg min 𝐽
𝐽𝑖 𝛼, 𝛽|𝑄|𝑗 ≻ 𝑞𝑖 𝑘 PHDQV WKDW IRU SDJH L WKH XVHU LV PRUH OLNHO\ WR MXPS WR SDJH M WKDQ WR SDJH N 5LM
LV WKH VLPLODULW\ YDOXH EHWZHHQ SDJH L DQG SDJH M ZKLOH 5LN LV WKH VLPLODULW\ YDOXH EHWZHHQ SDJH
L DQG SDJH N 4 ^TL` LV WKH UHFRPPHQGDWLRQ OLVW IRU WKH WH[WERRN PDGH LQ DGYDQFH E\ H[SHUWVZKHUH TL LV WKH UHFRPPHQGDWLRQ OLVW IRU SDJH L :KHQ WKH RUGHU RI WKH UHFRPPHQGDWLRQ OLVWJHQHUDWHG E\ WKH PRGHO LV FRPSOHWHO\ FRQVLVWHQW ZLWK WKH RUGHU RI WKH H[SHUW UHFRPPHQGDWLRQ
OLVW - %\ FRPSDULQJ WKH RUGHU RI WKH UHFRPPHQGDWLRQ OLVW JHQHUDWHG E\ WKH PRGHO ZLWK WKH
H[SHUWJHQHUDWHG UHFRPPHQGDWLRQ OLVW WKH Į DQG ȕ WKDW PLQLPL]H - DUH WKH RSWLPDO SDUDPHWHUYDOXHV 0RGHO HYDOXDWLRQ DQG UHVXOW
 0RGHO HYDOXDWLRQ PHWKRG,Q  SDUW RI WKH H[SHUWJHQHUDWHG UHFRPPHQGDWLRQ OLVW LV XVHG DV WKH WUDLQLQJ VHW DQG WKHUHFRPPHQGDWLRQ OLVW 5P JHQHUDWHG E\ WKH K\EULG PRGHO LV GHWHUPLQHG $QRWKHU SDUW RI WKH
H[SHUWJHQHUDWHG UHFRPPHQGDWLRQ OLVW LV XVHG DV WKH YDOLGDWLRQ VHW 5H :KHQ RQH RI WKH WRSWKUHH UHFRPPHQGHG SDJHV JLYHQ E\ 5P LV LQFOXGHG LQ WKH WRS WKUHH UHFRPPHQGHG SDJHV RI 5H
WKH PRGHO UHFRPPHQGDWLRQ LV FRQVLGHUHG VXFFHVVIXO 7KH HYDOXDWLRQ LQGH[ DFFXUDF\ LV
REWDLQHG E\ FDOFXODWLQJ WKH TXRWLHQW RI WKH QXPEHU RI VXFFHVVIXO SDJHV DQG WKH WRWDO QXPEHU RISDJHV

 0RGHO HYDOXDWLRQ UHVXOW,Q WKLV VWXG\ ZH FROOHFWHG GDWD IURP  XQLYHUVLW\ VWXGHQWV ZKR XVH (QJOLVK DV D VHFRQGODQJXDJH VWXGLHG WKH ³&RPPHUFLDO /DZ´ GLJLWDO WH[WERRN LQ (QJOLVK DQG PLQHG WKH DVVRFLDWLRQV
EHWZHHQ WH[WV IURP WKLV GDWD)LJXUH  7KH WH[W LPDJH RI ³&RPPHUFLDO /DZ´

454

7DEOH  6DPSOHV RI ORJ GDWD
8VHU ,'
DGPLQ
GHPR
DGPLQ

2SHUDWLRQ
1(;7 LGSDJH
1(;7 LGSDJH
35(9 LGSDJHERRNSDJH'DWH


3DJH 1R

7KURXJK JUDGLHQW GHVFHQW WKH ILQDO GHWHUPLQHG YDOXHV RI 𝛼 DQG 𝛽 DUH DQG  UHVSHFWLYHO\7KH K\EULG PRGHO WKH UHFRPPHQGDWLRQ PRGHO EDVHG RQ XVHU GDWD DQG WKH
FRQWHQWEDVHG UHFRPPHQGDWLRQ PRGHO ZHUH HYDOXDWHG DFFRUGLQJ WR WKH PHWKRG LQ  DQG
WKH UHVXOWV DUH VKRZQ LQ WKH WDEOH
7DEOH  7KH DFFXUDF\ RI PRGHOV&DWHJRU\K\EULG PRGHO LQ WKLV VWXG\
XVHU GDWDEDVHG PRGHO RQO\
FRQWHQWEDVHG PRGHO RQO\$FFXUDF\




7KH DFFXUDF\ RI WKH UHFRPPHQGDWLRQ OLVW JHQHUDWHG XVLQJ RQO\ XVHU GDWD LV  WKH
DFFXUDF\ RI WKH UHFRPPHQGDWLRQ OLVW JHQHUDWHG XVLQJ RQO\ WH[WERRN FRQWHQW LV  DQG WKHDFFXUDF\ RI WKH PRGHO OLQHDUO\ FRPELQHG WKURXJK JUDGLHQW GHVFHQW UHDFKHV  7KH UHVXOWVSURYH WKDW WKH UHFRPPHQGDWLRQ OLVW JHQHUDWHG E\ WKH K\EULG PRGHO LV PRUH DFFXUDWH DQG
HIIHFWLYH WKDQ XVLQJ RQO\ WHDFKLQJ PDWHULDO FRQWHQW RU XVHU GDWD DQDO\VLV DORQH &RQFOXVLRQV
7KLV VWXG\ OLQHDUO\ FRPELQHG WKH UHFRPPHQGDWLRQ PRGHO EDVHG RQ WH[WERRN FRQWHQW DQG WKHUHFRPPHQGDWLRQ PRGHO EDVHG RQ XVHU GDWD DQG GHWHUPLQHG WKH RSWLPDO ZHLJKWV RI WKH WZRWKURXJK JUDGLHQW GHVFHQW UHVXOWLQJ LQ D K\EULG UHFRPPHQGDWLRQ PRGHO IRU WKH SDJH $IWHU
FRPSDULQJ ZLWK H[SHUW GDWD WKH DFFXUDF\ RI WKH K\EULG PRGHO ZDV HYDOXDWHG SURYLQJ WKHDFFXUDF\ RI WKH SDJH UHFRPPHQGDWLRQ PRGHO LV KLJKHU WKDQ WKDW RI D VLQJOH PRGHO
2Q WKH RWKHU KDQG WKH SRRU SHUIRUPDQFH RI WKH UHFRPPHQGDWLRQ PRGHO EDVHG VROHO\ RQ
FRQWHQW LQGLFDWHV WKDW WKHUH LV VWLOO PXFK URRP IRU LPSURYHPHQW LQ WKH SURFHVVLQJ PHWKRGV RI
WKH WH[WERRN FRQWHQW $QG GXH WR WKH ODFN RI H[SHUWJHQHUDWHG UHFRPPHQGDWLRQ OLVWV DQG
VWXGHQW OHDUQLQJ ORJV WKH RYHUDOO SHUIRUPDQFH RI WKH PRGHO LV QRW VDWLVIDFWRU\
7KHUHIRUH PRUH UHVHDUFK LV QHHGHG WR LPSURYH WKH UHFRPPHQGDWLRQ PRGHO DQG WKHHIIHFWLYHQHVV RI WKH UHFRPPHQGDWLRQ V\VWHP VWLOO QHHGV WR EH YHULILHG LQ DFWXDO OHDUQLQJHQYLURQPHQWV$FNQRZOHGJHPHQWV
7KLV UHVHDUFK ZDV SDUWLDOO\ VXSSRUWHG E\ *UDQWVLQ$LG IRU 6FLHQWLILF 5HVHDUFK 1R +
IURP WKH 0LQLVWU\ RI (GXFDWLRQ &XOWXUH 6SRUWV 6FLHQFH DQG 7HFKQRORJ\ 0(;7 LQ -DSDQ

5HIHUHQFHV0DUWLQ ) 6XQ 7 :HVWLQH & '   $ V\VWHPDWLF UHYLHZ RI UHVHDUFK RQ RQOLQH WHDFKLQJ DQG
OHDUQLQJ IURP  WR  &RPSXWHUV HGXFDWLRQ  
:DQJ - 6KLPDGD $ 2L 0 2JDWD + 7DEDWD <   'HYHORSPHQW DQG HYDOXDWLRQ RI D
YLVXDOL]DWLRQ V\VWHP WR VXSSRUW PHDQLQJIXO HERRN OHDUQLQJ ,QWHUDFWLYH /HDUQLQJ(QYLURQPHQWV    

455

/DXWHUPDQ 7 $FNHUPDQ 5   2YHUFRPLQJ VFUHHQ LQIHULRULW\ LQ OHDUQLQJ DQG
FDOLEUDWLRQ &RPSXWHUV LQ +XPDQ %HKDYLRU  
'RUQ ( +DQFRFN % 6DUDNDWVDQQLV - 9LUXOHJ (   &29,' DQG VWXGHQW OHDUQLQJ LQ WKH
8QLWHG 6WDWHV 7KH KXUW FRXOG ODVW D OLIHWLPH 0F.LQVH\ &RPSDQ\  
*DUFtD0RUDOHV 9 - *DUULGR0RUHQR $ 0DUWtQ5RMDV 5   7KH WUDQVIRUPDWLRQ RI KLJKHU
HGXFDWLRQ DIWHU WKH &29,' GLVUXSWLRQ (PHUJLQJ FKDOOHQJHV LQ DQ RQOLQH OHDUQLQJ
VFHQDULR )URQWLHUV LQ SV\FKRORJ\  
<LQ & 5HQ = 3RO\]RX $ :DQJ <   /HDUQLQJ EHKDYLRUDO SDWWHUQ DQDO\VLV EDVHG RQ GLJLWDO
WH[WERRN UHDGLQJ ORJV ,Q 'LVWULEXWHG $PELHQW DQG 3HUYDVLYH ,QWHUDFWLRQV WK ,QWHUQDWLRQDO&RQIHUHQFH '$3,  +HOG DV 3DUW RI WKH VW +&, ,QWHUQDWLRQDO &RQIHUHQFH +&,, 
2UODQGR )/ 86$ -XO\ ±  3URFHHGLQJV  SS   6SULQJHU ,QWHUQDWLRQDO
3XEOLVKLQJ+DYHUNDPS < ( %UnWHQ ,   7KH 5ROH RI 6WUDWHJLF %DFNWUDFNLQJ :KHQ 5HDGLQJ 'LJLWDO,QIRUPDWLRQDO 7H[W IRU 8QGHUVWDQGLQJ /LWHUDF\ 5HVHDUFK DQG ,QVWUXFWLRQ 
6SDUFN -RQHV .   $ VWDWLVWLFDO LQWHUSUHWDWLRQ RI WHUP VSHFLILFLW\ DQG LWV DSSOLFDWLRQ LQ
UHWULHYDO -RXUQDO RI GRFXPHQWDWLRQ    
:X + & /XN 5 : 3 :RQJ . ) .ZRN . /   ,QWHUSUHWLQJ WILGI WHUP ZHLJKWV DV PDNLQJUHOHYDQFH GHFLVLRQV $&0 7UDQVDFWLRQV RQ ,QIRUPDWLRQ 6\VWHPV 72,6     
4DLVHU 6 $OL 5   7H[W PLQLQJ XVH RI 7),') WR H[DPLQH WKH UHOHYDQFH RI ZRUGV WR
GRFXPHQWV ,QWHUQDWLRQDO -RXUQDO RI &RPSXWHU $SSOLFDWLRQV    =KRX = 4LQ - ;LDQJ ; 7DQ < /LX 4 ;LRQJ 1 1   1HZV WH[W WRSLF FOXVWHULQJ RSWLPL]HGPHWKRG EDVHG RQ 7),') DOJRULWKP RQ VSDUN &RPSXWHUV 0DWHULDOV &RQWLQXD    
:X + <XDQ 1  0D\  $Q ,PSURYHG 7),') DOJRULWKP EDVHG RQ ZRUG IUHTXHQF\ GLVWULEXWLRQLQIRUPDWLRQ DQG FDWHJRU\ GLVWULEXWLRQ LQIRUPDWLRQ ,Q 3URFHHGLQJV RI WKH UG ,QWHUQDWLRQDO&RQIHUHQFH RQ ,QWHOOLJHQW ,QIRUPDWLRQ 3URFHVVLQJ SS  
5RPHUR & 9HQWXUD 6   (GXFDWLRQDO GDWD PLQLQJ $ VXUYH\ IURP  WR  ([SHUWV\VWHPV ZLWK DSSOLFDWLRQV    
=KDR ) .XPDPRWR ( <LQ &  -XO\  7KH HIIHFW DQG FRQWULEXWLRQ RI HERRN ORJV WR PRGHOFUHDWLRQ IRU SUHGLFWLQJ VWXGHQWV¶ DFDGHPLF SHUIRUPDQFH ,Q  ,QWHUQDWLRQDO &RQIHUHQFH RQ$GYDQFHG /HDUQLQJ 7HFKQRORJLHV ,&$/7 SS   ,(((5LHVWUD*RQ]iOH] 0 GHO 3XHUWR 3DXOH5Xt] 0 2UWLQ )   0DVVLYH /06 ORJ GDWD DQDO\VLV IRU
WKH HDUO\ SUHGLFWLRQ RI FRXUVHDJQRVWLF VWXGHQW SHUIRUPDQFH &RPSXWHUV (GXFDWLRQ 
*REHUW - ' 6DR 3HGUR 0 5D]LXGGLQ - %DNHU 5 6   )URP ORJ ILOHV WR DVVHVVPHQW PHWULFV0HDVXULQJ VWXGHQWV VFLHQFH LQTXLU\ VNLOOV XVLQJ HGXFDWLRQDO GDWD PLQLQJ -RXUQDO RI WKH /HDUQLQJ
6FLHQFHV    
<LQ & 8RVDNL 1 &KX + & +ZDQJ * - +ZDQJ - - +DWRQR , 7DEDWD <  'HFHPEHU /HDUQLQJ EHKDYLRUDO SDWWHUQ DQDO\VLV EDVHG RQ VWXGHQWV¶ ORJV LQ UHDGLQJ GLJLWDO ERRNV,Q 3URFHHGLQJV RI WKH WK LQWHUQDWLRQDO FRQIHUHQFH RQ FRPSXWHUV LQ HGXFDWLRQ SS  $KHU 6 % /RER / 0 5 -   &RXUVH UHFRPPHQGHU V\VWHP LQ (OHDUQLQJ ,QWHUQDWLRQDO-RXUQDO RI &RPSXWHU 6FLHQFH DQG &RPPXQLFDWLRQ    
5HGG\ 6 /DEXWRY , -RDFKLPV 7  $SULO  /HDUQLQJ VWXGHQW DQG FRQWHQW HPEHGGLQJV IRUSHUVRQDOL]HG OHVVRQ VHTXHQFH UHFRPPHQGDWLRQ ,Q 3URFHHGLQJV RI WKH 7KLUG  $&0&RQIHUHQFH RQ /HDUQLQJ# 6FDOH SS  
5DPRV -  'HFHPEHU  8VLQJ WILGI WR GHWHUPLQH ZRUG UHOHYDQFH LQ GRFXPHQW TXHULHV
,Q 3URFHHGLQJV RI WKH ILUVW LQVWUXFWLRQDO FRQIHUHQFH RQ PDFKLQH OHDUQLQJ 9RO  1R  SS &KLED .DWVXKLUR   7KH HIIHFW RI H[WHQVLYH UHDGLQJ RQ (QJOLVK UHDGLQJ VSHHG %XOOHWLQ RI WKH)DFXOW\ RI ,QWHUQDWLRQDO 6WXGLHV DW %XQN\R 8QLYHUVLW\    
%DUWHOO % 7 &RWWUHOO * : %HOHZ 5 .   $XWRPDWLF FRPELQDWLRQ RI PXOWLSOH UDQNHG UHWULHYDO
V\VWHPV ,Q 6,*,5¶ 3URFHHGLQJV RI WKH 6HYHQWHHQWK $QQXDO ,QWHUQDWLRQDO $&06,*,5&RQIHUHQFH RQ 5HVHDUFK DQG 'HYHORSPHQW LQ ,QIRUPDWLRQ 5HWULHYDO RUJDQLVHG E\ 'XEOLQ &LW\
8QLYHUVLW\ SS   6SULQJHU /RQGRQ

456

" !   !" " "   #" !
#" !"$  #" ! #"           
        )!(-$#$&"(' -&#$&"(''&#'(()(##*&'(-%#)!(-$ $"%)(&##*&'('&+-#$#'&)($$!$*### ##&#&$'"#*&'(-%#L4@4@BGB<A9B ><A74< 46 =C   >>0>>809? :1 70,=90= @9/0=>?,9/492 ,9/ 100/-,.6 -,>0/ :9 ?3,?
,>>0>>809?,=048;:=?,9??:;=:8:?070,=949249,>D9.3=:9:@>:9749070,=9492#1?09
,@?:8,?4. 100/-,.6 -D ?30 70,=9492 09A4=:9809? 4> @>0/ ,> , B,D :1 =0,74E492 ?34>
100/-,.6 :B0A0= @>@,7 ?0>?> ,=0 /4114.@7? ?: 4/09?41D 70,=9492 :-50.?> 0C3,@>?4A07D
:9.0;? 8,;> .,9 ;=:A4/0 , .:8;=0309>4A0 ,9/ /0?,470/ .30.6 :1 , 70,=90=>
@9/0=>?,9/492-@?,=0/4114.@7??:,>>0>>'30=01:=0?34>>?@/D@>0/,=0.:8;:>4?4:9
.:9.0;? 8,; .,770/ 4?@47/ .:9.0;? 8,;  8,;  8,; >D>?08 .,9 ;=:A4/0
>08,9?4. 100/-,.6 ?: ?30 70,=90= :9 /4110=09.0> 49 @9/0=>?,9/492 ,@?:8,?4.,77D *0
@>0/?34>100/-,.6,>,1:=8,?4A0,>>0>>809?1:=?3009?4=070.?@=0(>492?308,;
,> ,9 ,>>429809? 1:= , 1:=8,?4A0 ,>>0>>809? B0 .:9/@.?0/ , 7,=20>.,70
,>D9.3=:9:@>.:@=>0B4?3,.7,>>:1   >?@/09?>1=:8 1,.@7?40>'30=0>@7?>>3:B
?3,? 70,=90=> @9/0=>?,9/492 4> 093,9.0/ -D .:9.0;? 8,;> ,9/ ?3,? 1:=8,?4A0
,>>0>>809?-D80,9>:1.:9.0;?8,;>8,D09.:@=,20=0A4>4:9:170>>:98,?0=4,7>  :=8,?4A0 ,>>0>>809? ,>D9.3=:9:@> :97490 70,=9492 ,>>429809? 64?-@47/.:9.0;?8,;

!&$"'&"!
/<A68 G;8 +1%
 BHG5E84> BA?<A8 ?84EA<A: ;4F FCE847 E4C<7?L 4A7 7<I8E:8AG?L 4A7 G;8
78@4A79BE89986G<I8BA?<A8?84EA<A:;4F586B@84A<A6E84F<A:?L<@CBEG4AG<FFH8 0;8E84E8
GJB GLC8F B9 BA?<A8 ?84EA<A: @8G;B7F A4@8?L FLA6;EBABHF 4A7 4FLA6;EBABHF  %A
4FLA6;EBABHFBA?<A8?84EA<A:G;8G846;8ECE8C4E8FG846;<A:@4G8E<4?F4A7CEBI<78FG;8@GB
?84EA8EFJ;BG;8AFGH7L4GG;8<EBJAC4684A7G<@8 0;84FLA6;EBABHFGLC8<FG;8FH5=86GB9
G;<F FGH7L  0;<F GLC8 B9 BA?<A8 ?84EA<A: 4??BJF ?84EA8EF GB @4A4:8 G;8<E BJA ?84EA<A: 4A7
CEBI<78FC8EFBA4?<M87?84EA<A:BCCBEGHA<G<8F#4EE<FBA *:HL8A  +AG;8BG;8E;4A7FB@8?84EA8EF7BABG;4I8FH99<6<8AGBCCBEGHA<G<8FGB<AG8E46G7HE<A:
?84EA<A:0;4@E<$4F4A8G4?  "BE@4G<I84FF8FF@8AG<F>ABJAGB;8?CG;8F8?84EA8EF
GB 47=HFG G;8<E BJA ?84EA<A: )8HF8A 88>@4A &BBFG8A G8A 4A7 BF;H<M8A   
"BE@4G<I84FF8FF@8AG<F5EB47?L789<A874F9887546>CEBI<7877HE<A:G;8?84EA<A:CEB68FF HAA)H?I8ABA  $BJ8I8E<G<F4?FB>ABJAG;4GG;<F<F7<99<6H?GGBE84?<M8#H?<>8EF
18H:8A4A744EG@4A   
%A946<?<G4G<A:4CCEBCE<4G8?84EA<A:9BE?84EA8EFJ898?GG;4G<GJ4F<@CBEG4AGGB5845?8
GB4FF8FF6B@CE8;8AF<I8?L4A7F8@4AG<64??L4:4<AFGG;86BAG8AG58<A:G4H:;G4A7'@4CF
HF8F6BA68CG@4CFGB8KG8EA4?<F8?84EA8EFHA78EFG4A7<A:4A7CEBI<78F4HGB@4G<6F8@4AG<6
9887546> BA G;8 7<998E8A68 <A HA78EFG4A7<A: J<G; G;8 G846;8E $<E4F;<@4 34@4F4>< 8G 4?   0;8 CBFF<5<?<GL B9 FH6; C8EFBA4?<M87 4HGB 4FF8FF@8AG @84AF G;4G C8EFBA4?<M87
9BE@4G<I84FF8FF@8AG<F4?FBCBFF<5?8<A4FLA6;EBABHFBA?<A8?84EA<A:J;<6;@4LCEB@BG8
?84EA8EF E89?86G<BA BA G;8 G846;<A: @4G8E<4?  0;<F FGH7L 8K4@<A8F G;8 89986G<I8A8FF B9
9BE@4G<I8 4FF8FF@8AG HF<A: ' @4CF <A 4FLA6;EBABHF BA?<A8 ?86GHE8F  0;8 ?86GHE8 <F 4A

457

<AGEB7H6G<BAGB74G4?<G8E46L9BE4??946H?G<8FB94F<A:?8HA<I8EF<GL4A7<F6BA7H6G87BA4?4E:8
F64?8J<G;  FGH78AGF &"$%
"BE@4G<I84FF8FF@8AG<F789<A874F9887546>CEBI<7877HE<A:G;8?84EA<A:CEB68FF8AA8GG "BE@4G<I84FF8FF@8AG<FHF89H?9BE?84EA8EFGBF8?9 E8:H?4G8G;8<E?84EA<A:<G<F7<99<6H?GGB46;<8I84CCEBCE<4G89BE@4G<I84FF8FF@8AG#H?<>8EF18H:8A4A744EG@4A   "BE
4CCEBCE<4G89BE@4G<I84FF8FF@8AG<G<F<@CBEG4AGGBCEBI<789887546>BAHA78EFG4A7<A:B9G;8
?84EA<A:FH5=86G<A49BE@G;4G<F5BG;6B@CE8;8AF<I84A784FL9BE?84EA8EFGBHA78EFG4A7<A
BE78EGBCEB:E8FFG;8<E?84EA<A: 14E<BHF4HGB@4G87FLFG8@F;4I8588ACEBCBF87GB46;<8I8
G;<F4CCEBCE<4G8?L4A74G4?BJ6BFG 0;8L4E8B9G8ACE<@4E<?L4HGB@4G879BEF6BE<A::8A8E4?
G8FGF"85E<4A<57H??4;  $BJ8I8E8I8AJ;8AG8FGE8FH?GF4E8:<I8A4FF6BE8F<G<F
7<99<6H?G 9BE ?84EA8E GB E89?86G BA F ;8F HA78EFG4A7<A:  08FGF 4?FB @4>8 <G 7<99<6H?G GB
6B@CE8;8AF<I8?L4FF8FFJ;4G?84EA8EF;4I8?84EAG BAGE4FG<A:?LJ89B6HF87BA9BE@4G<I8
4FF8FF@8AGF 4A7 F8?9 E8:H?4G87 ?84EA<A: HF<A: 6BA68CG @4CF   6BA68CG @4C 8KG8EA4?<M8F
G;8FGEH6GHE8B94G4E:8G6BA68CG5L6BAA86G<A:6BA68CGF4FAB78F4A7G;8<EE8?4G<BAF;<CF4F
?<A>F*BI4>4O4F  BA68CG@4CF64ACEBI<7846B@CE8;8AF<I84A778G4<?876;86>
B94?84EA8EFHA78EFG4A7<A:4A7CEBI<78F8@4AG<69887546> 0;HF<G<FHF89H?GBHA78EFG4A7
G;8?84EA8ETFFG4G8B9HA78EFG4A7<A:@BE87<E86G?L4A7;4F588AFH::8FG874F4A89986G<I8GBB?
9BE9BE@4G<I84FF8FF@8AG84H7EL2<?FBA$BJ8I8E6BA68CG@4CF4E87<99<6H?GGB4FF8FF4HGB@4G<64??L4A7G;8L<A6HE;<:;6BFGF
9BE@4C4FF8FF@8AG 0;8LB9G8ABC8E4G8FH6;G;4GG846;8EF4A7C88EFI<8JG;8?84EA8EF@4C
4A7<AG8E46GJ<G;<G9BE?84EA<A: %G<F7<99<6H?GGB64EELBHGFH6;46G<I<G<8F<A4FLA6;EBABHFBA?<A8
?84EA<A: 28G;8E89BE8GHEA87GB'@4CF %A6BAGE4FG'@4CF4E8HF87GB4HGB@4G<64??L
4FF8FF?84EA8E@4CF$<E4F;<@434@4F4><8G4?   '@4CF4??BJG;86B@C4E<FBAB9
@4CF 6BAFGEH6G87 5L G;8 G846;8E 4A7 G;8 ?84EA8E GB CEBI<78 4HGB@4G<6 9887546> BA G;8
7<998E8A68F58GJ88A846;@4C4F8EEBEFBE?46>B9HA78EFG4A7<A:B96BA68CGF5LG;8?84EA8E 
0;8 I4?<7<GL B9 G;8 4HGB@4G<6 4FF8FF@8AG B9 ' <A 6B@C4E<FBA J<G; F8I8E4? @4AH4?
@8G;B7F;4F4?E847L588A6BA9<E@872HAA4FE<,4<?4<8G4?   0;8'<G H<?76BA68CG
@4C FLFG8@ ' )4C /LFG8@ ;4F 588A HF87 <A 9468 GB 9468 ?84EA<A: 9BE G;8 9BE@4G<I8
4FF8FF@8AGB9846;?8FFBA4A7;4F588AF;BJAGB8A45?8G846;8EFGBCEBI<78@BE8E8?8I4AG
9887546>GB?84EA8EF,4<?4<2HAA4FE<8G4?   0;8E89BE8J86BAF<78E87G;4G<AG8:E4G<A:' )4CF <AGB 4FLA6;EBABHF BA?<A8 6?4FF8F JBH?7 6BAGE<5HG8 GB 5H<?7<A: 4 6L6?8 <A J;<6;
?84EA8EF6BH?7788C8AG;8<EBJA?84EA<A: & ' "!#&#(%& "$ %(!$"!"'%!! %%%CEB687HE8B9G;89BE@4G<I84A7FH@@4G<I88I4?H4G<BAF54F87BAG;8'@4CHF87<AG;<F
FGH7L<FF;BJA<A"<:HE8 0;8G846;8EE8CE8F8AGFFH@@4E<M876BA68CGFG;4GF ;8J4AGFGB
GE4AF98EGBG;8?84EA8E4F46BA68CG@4CFH@@4E<M87@4C 0;8FH@@4E<M87@4C64A58
6E84G87BA4'@4CFLFG8@4A74HGB@4G<64??L786B@CBF87<AGB4F8GB9?<A>F4A7AB78F5L
4'@4CFLFG8@ 0;8786B@CBF87@4C<FE898EE87GB4F4><GJ8E86E84G879BE846;6?4FF "BE 846; 6?4FF G;8 ?84EA8E 7BJA?B47F 4 ><G 6E84G87 5L G;8 G846;8E 4A7 5H<?7 <G GB
E8CE8F8AG;<F ;8EHA78EFG4A7<A:B9G;8G4H:;G6BAG8AG?84EA8E@4C4F4FF<:A@8AG "887546>
<F4I4<?45?84G4ALG<@89BE846;4FF<:A@8AG 0;8?84EA8E64ACE8FFG;89887546>5HGGBA7HE<A:
@4C5H<?7<A:GB78G8E@<A8J;8G;8EG;8E84E8?<A>FG;4G4E8<@CEBC8E?L6BAA86G87 %AG;<FFGH7L
4FF8FF@8AG64EE<87BHG4GG;<FG<@8<F64??879BE@4G<I84FF8FF@8AG %AG;<F64F8<A6BEE86GBE
HA6BAA86G876BA68CGF4E8;<:;?<:;G87HF<A:HAG4::87?<A>F %A4FLA6;EBABHFBA?<A86?4FF8F?84EA8EF4E89E88GB4GG8A76?4FF8F4G4ALG<@8@4><A:
<G7<99<6H?G9BEG846;8EFGB7<E86G?L<AG8EI8A8<A?84EA8EF?84EA<A: $BJ8I8EG;8')4CFLFG8@
CEBI<78FE84? G<@84A7F8@4AG<69887546>54F87BA6BA68CGF4A7 <GFE8?4G<BAF;<CFFBG;4G
?84EA8EF 64A E86BAF<78E G;8 ?8FFBA 6BAG8AG 466BE7<A:?L ?847<A: GB 4 ;<:;8E DH4?<GL G;8<E
HA78EFG4A7<A:B9846;?86GHE8 9G8E4F8E<8FB9<A7<I<7H4??8FFBAFG;8?84EA8EF6B@C?8G874

458

'@4C 5H<?7<A:G8FG4A749<?? <A G;8 5?4A>G8FGGB6;86>G;8<EHA78EFG4A7<A:B9G;8?8FFBA %AG;<FFGH7LG;84FF8FF@8AGB9G;8F8G8FGF<FE898EE87GB4FFH@@4G<I84FF8FF@8AG 0;8E8FH?GF
B99BE@4G<I84FF8FF@8AG4E8CEBI<787GB?84EA8EF4F9887546>J;8E84FG;BF8B9FH@@4G<I8
4FF8FF@8AG4E8BA?LHF875LG846;8EF 
4F87BAG;845BI8G;89B??BJ<A:4E8G;8E8F84E6;DH8FG<BAF.-9BEG;<FFGH7L4F4A
<A<G<4? 8I4?H4G<BA B9 G;8 9BE@4G<I8 8I4?H4G<BA 9E4@8JBE> HF<A: G;8 ' )4C .-  B8F
9BE@4G<I8 4FF8FF@8AG HF<A: G;8 ' @4CF <@CEBI8 FH@@4G<I8 4FF8FF@8AG C8E9BE@4A68.- B8F9BE@4G<I84FF8FF@8AGHF<A:'@4CF8A6BHE4:8CBF<G<I8?84EA8E58;4I<BE       












 










   





















            































             

















          )&"E4@8JBE>9BE9BE@4G<I84A7FH@@4G<I84FF8FF@8AGHF<A:'@4C  %!"&!! "'$%
  #  "!

0;<FCE46G<64?4CC?<64G<BAJ4FG4E:8G874G4?86GHE864??87Q%AGEB7H6G<BAGB 4G4(<G8E46L R0;8
6?4FF8F7<F6HFF87<AG;<FC4C8E4E8?86GHE8F FGE7GBG;J;<6;J8E878?<I8E87<AG;89BE@4G
78F6E<587 <A 6;4CG8E   0;8 C4EG<6<C4AGF J8E8   9<EFG :E478 HA<I8EF<GL FGH78AGF 9EB@ 
7<998E8AG946H?G<8F  4G49EB@
C4EG<6<C4AGFJ8E84A4?LM87<AG;<FFGH7L 0;8E8J8E8G;E88
E84FBAF9BE47BCG<A:G;8?B:74G4 ,4EG<6<C4AGF4:E887GBHF8G;874G40;89887546>
9EB@G;8')4CFLFG8@<FABG4ABHG?<8E4A76B@C?8G874G4 0;874G4J8E8;4A7?87<A
6B@C?<4A68J<G;G;88G;<64?6B78F8FG45?<F;875LG;89<EFG4HG;BETFHA<I8EF<GL BAG8AGFB9 FGE7GBG;?86GHE8FJ<??8KC?4<A 0;8G4E:8G6?4FF8F4E84FLA6;EBABHF
BA?<A8?86GHE8FJ<G;J88>?L<AFGEH6G<BA4?I<78BF4A7FGH78AGF@4LI<8JG;8I<78BF4GG;8<EBJA
C468 4A7 G<@<A:  $BJ8I8E ?84EA8EF J8E8 E8DH<E87 GB I<8J I<78BF J<G;<A GJB J88>F  9G8E
I<8J<A: G;8 <AFGEH6G<BA4? I<78BF FGH78AGF JBE> BA ' )4C 4FF<:A@8AGF 54F87 BA G;8
<AFGEH6G<BA4? I<78BF J;<6; 4E8 4668CG87 HAG<? GJB J88>F 49G8E G;8 8A7 B9 G;8 G; 6?4FF $BJ8I8E FGH78AGF 4E8 <A9BE@87 G;4G CB<AGF J<?? 58 E87H687 5L   49G8E GJB J88>FPG;8
7847?<A89BEG;8I<78BF 
0;8F86BA7?86GHE8J4F<A4AB@A<5HF6?4FF<AJ;<6;G;8?84EA8EF8?86G87F8I8E4?B9
G;8I<78B@4G8E<4?FGBG4>8G;HFG;8')4C4FF<:A@8AGJ4FABG6BA7H6G87 %AG;8G;6?4FF
4 ' @4C G8FG 4A7 4 9<?? <A G;8 5?4A> G8FG J8E8 6BA7H6G87 G;8 ' @4C G8FG 6BAF<FGF B9 G;8
6BA68CGFHF87<AG;8'@4C4FF<:A@8AGF9EB@6?4FF8F GB 0;89<?? <A G;8 5?4A>G8FG<F
CEBI<787<AG;89BE@B94G8KGHF<A:G;8F86BA68CG 0;89<?? <A G;8 5?4A>G8FG<F4@H?G<C?86;B<68
DH8FG<BA4??4AFJ8EF4E8CE8F8AG874F6;B<68F9BE846;DH8FG<BA %A477<G<BAG;8')4C4A7
9<?? <A G;8 5?4A>G8FGF846;;474 @<AHG8G<@8?<@<G   " !!!"!   !!
6BA68CG@4CJ4F6E84G875L8KGE46G<A:4A76BAA86G<A:6BA68CGF9EB@F?<78FG;4GF8EI874F
6?4FF@4G8E<4?F 0;8')4C7B8FABGFGE<6G?L9B??BJG;8ABG4G<BA789<A87<AG;86BA68CG@4C
5HGG;88@C;4F<F<FBAFGH78AGF4A7G846;8EF4:E88<A:J<G;G;86BA68CGF58<A:E8CE8F8AG87 
0;8F64?8B9G;8')4CFG;4GG;8?84EA8EFJ8E84F>87GB5H<?7J4FFH6;G;4GG;84FF<:A@8AGF;474A4I8E4:8AH@58EB9 AB78F4A7 ?<A>F *BG8G;4GFB@8B9G;8AB78F4A7

459

?<A>FJ8E8CE8 5H<?GFBG;846GH4?AH@58EB9AB78F4I8E4:87 4A7G;8AH@58EB9?<A>F
J4F  %A'@4CG8FGG;8AB78F4A7?<A>FJ8E84A7 E8FC86G<I8?L !K6?H7<A:G;8
CE8 5H<?G@4CG;8E8J8E8AB78F4A7?<A>F 0JB6?4FF6B GHGBEF4CCEBI87G;86BA68CGF
HF87<AG;8@4C (84EA8EF 64A HF8 <@@87<4G8 9887546> J;<?8 5H<?7<A: @4CF GB 788C8A G;8<E
HA78EFG4A7<A: B9 CBBE?L HA78EFGBB7 6BA68CGF 4A7 E8?4G<BAF;<CF  $BJ8I8E 5864HF8 G;<F
9887546><FABGCE8F8AG87HAG<?G;8?<A>FG;4GF;BH?758HF87GB6BAA86GG;8@4E878G8E@<A87
G;8?84EA8E<FE8?84EA87GB6BA9<E@;<F ;8EHA78EFG4A7<A:B9G;8A868FF4EL6BA68CGF4A7G;8<E
E8?4G<BAF;<CF5L9BE8K4@C?8E898EE<A:GBG;86?4FFI<78B %'&%! %'%%"! "#!"
0;874G4HF87<AG;<F4A4?LF<F4E8G;8?B:74G4B9G;8')4CG;8F6BE8FB9G;89<?? <A G;8 5?4A>
G8FG4A7G;8?B:74G4B9I<8J<A:BA?<A8I<78BF 0;8?B:74G4B9G;8')4C4E8G;8@4CF6BE8F
4A79887546>6BHAGF9BEG;8'@4C4FF<:A@8AGF 0;8?B:74G49BEG;8')4CG8FG<FBA?LG;8@4CF6BE8 0;89<?? <A G;8 5?4A>FG8FGF6BE8FJ8E84?FBHF875L6BAI8EG<A:G;8F6BE8FB9
4?? DH8FG<BAF<AGBC8E68AG4:8FB9
"EB@G;8I<8J<A:?B:FB9BA?<A8I<78B@4G8E<4?FG;8
AH@58EB9G<@8FG;8FC887B9G;8I<78B<F6;4A:87G;8AH@58EB9G<@8FG;8I<8J<A:CB<AGB9
G;8I<78B<F6;4A:87F88>54EBC8E4G<BA4A7G;8AH@58EB9G<@8FG;8I<78B<FFGBCC8764A
58B5G4<A87 "BEBHE4A4?LF<FJ87<I<787G;8F874G45LG;8AH@58EB96?4FF8F4A7HF87G;8@
4FG;84I8E4:8AH@58EB9G<@8FC8E6?4FF 0;8@4CF6BE89<?? <A G;8 5?4A>FG8FGF6BE84A7
4FF<:A@8AGF6BE8J8E8F6BE87BA4
CB<AGF64?85HGBG;8EC4E4@8G8EFJ8E8ABG64CC87  !#"!

0;8@84AF4A7FG4A74E778I<4G<BAF9BE846;74G4CB<AG4E8?<FG87<A045?8 0;84I8E4:8
4FF<:A@8AGF6BE8J4FJ<G;4FH5@<FF<BAE4G8B9 0;84I8E4:8AH@58EB9G<@8FG;8
9887546>J4F46G<I8?LHF87J4F 0;8946GG;4G@4ALFGH78AGFJ8E846G<I878FC<G8G;8'
)4CJ4F4A8J4FF<:A@8AGFH::8FGFG;4G<GJ4F4668CG874F4A4FF<:A@8AG )H?G<C?8E8:E8FF<BA4A4?LF<F).J4FHF87GB78G8E@<A8G;8E8?4G<BAF;<C58GJ88A
846;G8FGBE?B: *BG8G;4G9BE4??E8FH?GF<A045?8GBG;8/<:A<9 6B78F4E878ABG874F 
ST
 ST ST  S T  S T  "EB@ G;8 E8FH?G B9 045?8  G;8E8 4CC84E87 GB 58 4
E8?4G<BAF;<C 58GJ88A F6BE8F BA G;8 9<?? <A G;8 5?4A>F G8FG 4A7 G;8 ' @4C G8FG 4F J8?? 4F
F6BE8F BA G;8 G4F>F  $BJ8I8E G;8 @H?G<C?8 . FDH4E87 I4?H8 J4F J;<6; <F ABG ;<:;?L
8KC?4A4GBEL /C84E@4AF6BEE8?4G<BA4A4?LF<FJ4F6BA7H6G8758GJ88AG;8')4C4A79<?? <A
G;8 5?4A>G8FGF6BE8F 0;8E8FH?GFF;BJ874F?<:;G?LCBF<G<I86BEE8?4G<BA&  0;8E8J4F
4?FB?<GG?86BEE8?4G<BA58GJ88A9<?? <A G;8 5?4A>G8FGF6BE8F4A7G;8AH@58EB99887546>FE  
0;8E8FH?GB9045?8F;BJ87G;4G5BG;G;8AH@58EB99887546>F4A7G;84I8E4:8F6BE8
BAG;8')4C4FF<:A@8AG;4I84A89986G $BJ8I8EG;8@H?G<C?8. FDH4E87I4?H8J4F?BJ  0;8E89BE8/C84E@4AF6BEE8?4G<BA4A4?LF<FJ4F6BA7H6G87BAG;8@4CG8FGF6BE8F4A7
9887546>6BHAGF;BJ8I8EG;8E8J4FAB6BEE8?4G<BA&   0;84I8E4:8F6BE8BAG;8')4C4FF<:A@8AGJ4FCBF<G<I8?L6BEE8?4G87J<G;G;84I8E4:8F6BE8BAG;8')4CG8FG&   045?8 ##(#& *($#$  $!!( (I8E4:8F6BE8B9'@4CG8FGI8E4:8F6BE8B99<?? <A G;8 5?4A>G8FGI8E4:8F6BE8B9'@4C4FF<:A@8AGI8E4:8AH@58EB99887546>FI8E4:8AH@58EB9F88>54EBC8E4G<BAFI8E4:8AH@58EB96;4A:<A:I<78BFC887
I8E4:8AH@58EB9FGBCBC8E4G<BAF

460)84AF /                      

0;8E8FH?GFB9045?8<A7<64G87G;4GG;8AH@58EB99887546>F8FF<BAF;474A89986G $BJ8I8EG;8@H?G<C?8. FDH4E87I4?H8J4F?BJ  /C84E@4AF6BEE8?4G<BA6B899<6<8AGJ4F
64?6H?4G874A7AB6BEE8?4G<BAJ4F9BHA7&  0;8E8FH?GFB9045?8F;BJ87G;4G9887546>
6BHAGF J8E8 E8?4G87 GB G;8 AH@58E B9 F88> 54E BC8E4G<BAF FC887 6;4A:8F 4A7 FGBC
BC8E4G<BAF  0;8 @H?G<C?8 . FDH4E87 I4?H8 J4F ?BJ    2;8A G;8 6BEE8?4G<BA 6B899<6<8AGF
J8E878G8E@<A87<GJ4F9BHA7G;4GG;89887546>6BHAG4A7F88>54EBC8E4G<BAFJ8E8DH<G8
CBF<G<I8?L 6BEE8?4G87 J<G; &    0;8 AH@58E B9 FGBCC87 BC8E4G<BAF J4F 4?FB CBF<G<I8?L
6BEE8?4G87&  $BJ8I8EG;8E8J4F?<GG?86BEE8?4G<BAJ<G;G;8FC8876;4A:8&  
045?8    &')!(' +( ( $(* *&! ' ( !!#(!#  ('( '$& # (,%!#($&- *&!' ' (  % ('( '$& #)"& $  ' #  %
''#"#('$&
%AG8E68CG
/6BE8B9')4CG8FG
*H@58EB99887546>F/6BE8B9')4C4FF<:A@8AG

!FG<@4G87/G7  


!EEBE GI4?H8
 
 
    



,ENGN
8 
8 










045?8 &')!('+(($(**&!'(%('('$&#(,%!#($&-*&!''(#)"&$ '#%''#"#('$&
%AG8E68CG
*H@58EB99887546>F/6BE8B9')4C4FF<:A@8AG

!FG<@4G87/G7 
 



!EEBE
 GI4?H8
 
 
 

,ENGN
8 
 8 
8 







045?8 &')!('+(($(**&!'(%''#"#('$&#(,%!#($&- *&!' ' ( #)"& $  ' '  & $%&($#' '% #$%&($#'#'($%$%&($#'
!FG<@4G87/G7 %AG8E68CG *H@58EB99887546>F
*H@58EB9F88>54EBC8E4G<BAF*H@58EB9FC8876;4A:8BC8E4G<BAF*H@58EB9FGBCBC8E4G<BAF!EEBE



 


GI4?H8
 
 




,ENGN8   8  




 



045?8    &')!(' +( ( $(* *&! ' ( #)"& $   # (,%!#($&-*&!''(#)"&$' &'%##'($%$%&($#'
!FG<@4G87/G7  !EEBE%AG8E68CG 
 *H@58EB9F88>54EBC8E4G<BAF
*H@58EB9FC8876;4A:8BC8E4G<BAF
*H@58EB9FGBCBC8E4G<BAF
 


GI4?H8
 
 
 
 

,ENGN8   8   8   !#!! ""
A4?LF<FE8FH?GF<A7<64G8G;4G4788C8EHA78EFG4A7<A:B9G;8')4CG8FG@4L4?FB<@CEBI8
F6BE8FBAG;89<?? <A G;8 5?4A>G8FG 0;8')4CG8FGF6BE8F4E84?FBE8?4G87GBG;8')4C
4FF<:A@8AGF6BE8FJ<G;4788C8EHA78EFG4A7<A:B9')4C4FF<:A@8AGF4?FB<@CEBI<A:G;8
')4CG8FGF6BE8F 0;<F<F4E84FBA45?8E8FH?G5864HF8G;86BA68CGB9')4C4FF<:A@8AG
J4FHF87<AG;8')4CG8FG $BJ8I8EJ;<?8J89BHA7G;4GG;8')4C4FF<:A@8AGF6BE8F
J8E8E8?4G87GBG;8AH@58EB99887546>FJ87<7ABG9<A74AL6BEE8?4G<BA +A8CBFF<5?8E84FBA<FG;4G@BE89E8DH8AG9887546>7B8FABGA868FF4E<?L?847GB@BE889986G<I8?84EA<A: 0;8E89BE8
4?G;BH:;G;8CBFF<5<?<GLB9@88G<A:.- J4F<78AG<9<879HEG;8E4A4?LF8F4E8A868FF4EL 

461

%A477<G<BA4A4?LF<FE8FH?GFFH::8FGG;4G9887546>@4L;4I88A6BHE4:87?84EA8EFGB
8A:4:8 <A 46G<I<G<8F GB E8I<8J G;8 6?4FF I<78B 4:4<A  0;<F 6BA9<E@87 G;8 CBFF<5<?<GL G;4G
9BE@4G<I8 4FF8FF@8AG B9 G;8 ' )4C 4FF<:A@8AG <A G;8 4FLA6;EBABHF BA?<A8 6?4FF HF<A:
9887546>6BH?7CB<AGBHGG;8?84EA8EF?46>B9HA78EFG4A7<A:4A74??BJG;8@GBE86;86>G;8
@4G8E<4? G;8E85L 8A45?<A: G;8@ GB 46DH<E8 6BA68CGF G;4G G;8 G846;8E J4AG87 GB GE4AF98E
G;EBH:;4CCEBCE<4G8GE<4?4A78EEBE 0;8F8E8FH?GF<A7<64G8G;4G.-J4F<78AG<9<87 $BJ8I8E
478G4<?874A4?LF<FB9G;8984GHE8B9?84EA8EF4A7;BJ9887546><FHF87E8@4<AF46;4??8A:8  "!'%"!!'&'$"$%%AG;<FFGH7LJ8CEBCBF8G;8HF8B94')4C4A79E4@8JBE>GBE84?<M889986G<I89BE@4G<I8
4FF8FF@8AG <A 4FLA6;EBABHF BA?<A8 6?4FF8F  0;<F 9BE@4G<I8 4FF8FF@8AG F;BH?7 58
6B@CE8;8AF<I84A745?8GBCB<AGBHGFBHE68FB98EEBE4:4<AFGG;8?84EA<A:6BAG8AG<ABE78E9BE
G;8?84EA8EGBE89?86GBA;<FBE;8EBJA?84EA<A:4A7GB5889986G<I8 0;8E89BE8<AG;<FFGH7L
J8 8I4?H4G8 4FLA6;EBABHF BA?<A8 ?84EA<A: HF<A: G;8 ' )4C /LFG8@ J;<6; 4HGB@4G<64??L
8I4?H4G8F?84EA8EFT6BA68CG@4CF 
BAF8DH8AG?L<GJ4F6BA9<E@87G;4G9BE@4G<I88I4?H4G<BA<AG;8')4CFLFG8@@4L
E8499<E@G;8?84EA<A:@4G8E<4? 28J8E84?FB45?8GB6BA9<E@G;4GG;8C8E9BE@4A68B9G;8')4C5H<?7<A:<A9?H8A687G;89<A4?FH@@4G<I88I4?H4G<BA4A7J858?<8I8G;4GJ8J8E845?8GB
CEBI<784HF89H?6L6?8B94FLA6;EBABHFBA?<A8?84EA<A:GB?84EA8EF $BJ8I8EJ8;4I8ABGL8G
588A 45?8 GB 8K4@<A8 ;BJ 9BE@4G<I8 4FF8FF@8AG J4F HG<?<M87 4A7 ;BJ G;8 89986GF 7<998E87
78C8A7<A:BAG;84GGE<5HG8FB9G;8?84EA8EFG;8E89BE8478G4<?878I4?H4G<BA<FA88787 $!%
0,@/=D*47>:9$:9.0;?8,;;492,9/1:=8,?4A0,>>0>>809?70809?>>@;;:=?492
74?0=,.D,9/70,=9492%&&"&)*) &%&##&)+!-#)%!%,*!%&%'+$''!%;;
 7:-,70990??%
:=8,?4A0,>>0>>809?.=4?4.,7=0A40B ****$%+!%,+!&% ')!%!'#*'&#!.')+!   
@99!@7A09:9&*.=4?4.,7=0A40B:1=0>0,=.3:91:=8,?4A0,>>0>>809?>'30
7484?0/ >.409?414. 0A4/09.0 :1 ?30 48;,.? :1 1:=8,?4A0 ,>>0>>809?> 49 0/@.,?4:9 )+!#****$%+*) % -#,+!&% 
0-=4,94-/@77,3!
>D>?08,?4.=0A40B:11:=8,?4A0,>>0>>809??::7>49?30-709/0/
70,=949209A4=:9809?%+)%+!&%#&,)%#& %!%)!% %&#&.   ,==4>:9  %  :294?4A0 ;=0>09.0 1:= 0110.?4A0 ,>D9.3=:9:@> :97490 70,=9492 '30 =:70 :1
=0170.?4A049<@4=D>071/4=0.?4:9,9/80?,.:294?4:9 #$%+*&(,#!+.&%#!%,+!&% )+!%!)+!&% 
4=,>348,'+,8,>,64@6@/,@9,:4
=,80B:=6:164?-@47/.:9.0;?8,;1:=
,@?:8,?4. /4,29:>4> ,9/ 4?> ;=074849,=D @>0 *)  % )+! !%  %&#&. % %
)%!%   
!0@>090068,9   ::>?09?09 =4960   :>3@4E09  $  110.?> :1 1:=8,?4A0
,>>0>>809?>?:/0A07:;>071=02@7,?4:9,8:92>4C?32=,/0>?@/09?>%0>@7?>1=:8,=,9/:84E0/
.:9?=:770/49?0=A09?4:9+,!*!% ,+!&%# -#,+!&%   ":A,6,F,>
'30:=4249>:1?30.:9.0;?8,;;492?::7,9/?30.:9?49@4920A:7@?4:9
:1?30?::7%&)$+!&%-!*,#!/+!&%   
$,47,4*@99,>=4*+:>34/,,D,>34+4=,>348,'
'30;=,.?4.,7@>0:14?@47/
.:9.0;?8,;:91:=8,?4A0,>>0>>809?*) %)+!!% %&#&. % %)%!%   
'3,8=4',>,9%49,",94!!4=,9/,!
/A,9?,20>,9//4>,/A,9?,20>
:1 :97490 0,=9492 /@=492 ?30 #)  ;,9/084. '30 ;0=.0;?4:9> :1 >?@/09?> ,? @92 ,??,
(94A0=>4?D% &!#!%*  3??;> /:4:=2   6>>A4  *@99,>=4*$,47,4,D,>34+4=,>348,'),74/4?D:164?-@47/80?3:/1:=,>>0>>809?
:170,=90=-@47/8,;-D.:8;,=492B4?38,9@,780?3:/>    &%%&)$+!&%
%.*+$*    

462" !   !" " "   #" !
#" !"$  #" ! #"               $)  )")&,)" $& $'#($$  &$ ('*) %%"%$%'#)!(,%)%$!+'(!),&$#! $)'%' %#&*)!$$!)*!(,%)%$!+'(!),&$
?;E=EB1;?E;9 =CD;I?D?E13:@  '6323;/<24=@/A674B4@=;7<BC7B7=</<23F>3@73<130/A32B=3D723<13
0/A32 32C1/B7=< 6/A 033< 5@=E7<5  ;/8=@ 16/::3<53 7< @3/:7H7<5 B67A B@/<A7B7=< 7A
3FB@/1B7<5 3D723<13 4@=; @3/:E=@:2 32C1/B7=</: 2/B/ =<D3<B7=</::G B67A 2/B/
3FB@/1B7=< >@=13AA 7A >3@4=@;32 0G ;/<C/::G 16==A7<5 1:/AA3A 4=@ 1=;>/@7A=<
=E3D3@ B63 A3:31B7=< @3?C7@3A 3F>3@B 9<=E:3253 =4 E6716 1:/AA3A A6=C:2 03
1=;>/@32 E7B6 E6716 7<271/B=@A < B67A ABC2G E3 >@=>=A3 B63 CA3 =4 233> :3/@<7<5/:5=@7B6;A B= C<1=D3@ 7<63@3<B 1/CA/: @3:/B7=<A67>A E7B67< D/AB /;=C<BA =4 3F7AB7<5
2/B/&>317471/::GE33;>:=GAB@C1BC@/:/5<=AB71;=23:7<5/1/CA/:A3/@16/:5=@7B6;
9<=E<4=@7BA3F13>B7=</:>3@4=@;/<13=<@3/:E=@:22/B/B=3FB@/1B1=;>:3F1/CA/:
1/<272/B3A7<32C1/B7=<'67A/>>@=/166/A033<@343@@32B=/AL=0A3@D/B7=</:1/CA/:
27A1=D3@GM*33D/:C/B3B6334431B7D3<3AA=4B67A;3B6=2CA7<5@3/:E=@:232C1/B7=</:
2/B/ /<2 1=;>/@3 7BA /2D/<B/53A E7B6 B6=A3 =4 1=<D3<B7=</: /CB=;/B71 1=;>/@7A=<;3B6=2A%3AC:BA23;=<AB@/B3B6/BB63>@=>=A32;3B6=21/<723<B74GD/@7=CA1/CA/:
1/<272/B3A 7<1:C27<5 B6=A3 B6/B /@3 274471C:B 4=@ 6C;/<A B= 27A13@< /<2 3D3< B6=A3
E7B6=CB1/CA/:@3:/B7=<A67>A7443@3<B4@=;3F7AB7<5;3B6=2AB63>@=>=A32/>>@=/16
2=3A <=B @3?C7@3 A3:31B32 =@ 47F32 1=;>/@/B7D3 7<2713A B6CA >=B3<B7/::G C<1=D3@7<5
1=;>/@/B7D37<2713AB6/B3:C236C;/<1=;>@363<A7=<*3/<B717>/B3B6/BB67A@3A3/@16
E7::3</0:3B631=::31B7=<=4AC0AB/<B7/:3D723<134@=;@3/:E=@:232C1/B7=</:2/B//<2>@=;=B33D723<130/A3232C1/B7=<  D723<130/A32 32C1/B7=< @3/:E=@:2 3D723<13 &! 1/CA/: A3/@16
1/CA/:/</:GA7A  

F945>3521C5454E31D9?>1F95C 9C5H@53D54D?@B?F94554E31D9?>D81D9C>?D21C54
?> 9>DE9D9?> ?B 5H@5B95>35 +!$ 9C 75>5B1<<I 5HDB13D54 6B?= CICD5=1D93 B5F95GC 1>4
B1>4?=9J54 3?>DB?<<54 DB91<C (*C <D8?E78 D85C5 =5D8?4C @B?F945 8978<5F5< 5F945>35
3?<<53D9>7<1B751=?E>DC?65F945>359C496693E<D2531EC54?9>7C?9C?6D5>3?CD<IE>5D8931<?B
E>651C92<5)<145&B9>C<??
*85B56?B5D853?>35@D?6B51<G?B<45F945>3581C255>
@B?@?C549>D85=54931<695<4#181:1>
1>4C5F5B1<1DD5=@DC81F5255>=145D?1@@<I
9DD?D8554E31D9?>1<695<41CG5<<(51<G?B<45F945>359C5F945>355HDB13D546B?=F1B9?ECB51<G?B<441D11>41<D8?E789D9C<5CCB5<912<5D81>D81D5HDB13D546B?=CICD5=1D93B5F95GC1>4
(*C9D9C5H@53D54D?25F?<E=9>?EC%EB @B5F9?EC =5D8?4 3?=@1B5C 1 3?>DB?< 7B?E@ G9D8 1 3?BB5C@?>49>7 D1B75D 7B?E@ D?
F5B96I D85 56653DC ?6 1> 9>D5BF5>D9?> $1;1>9C89 *89C =5D8?4 9C 14F1>D175?EC 1C 9D
1<<?GCD8556653DC?61>9>D5BF5>D9?>D?25F5B969542IC5DD9>7D1B75D1>43?>DB?<7B?E@CD85B52I
5<9=9>1D9>7D85>5546?B5H@5B9=5>DC?G5F5B9DB5AE9B5C=1>E1<<IC5DD9>7D853<1CC5CD?25
3?=@1B541>49DCEC59>1ED?=1D935HDB13D9?>6B?==1>I31C5C9C496693E<D*?144B5CCD89C9CCE5G545F5<?@541=5D8?4D81D31>1ED?=1D931<<IC51B386?B3<1CC5C
G9D8C9=9<1B3?>D5HDE1<9>6?B=1D9?>CE381C7B145<5F5<1C13?>DB?<7B?E@%;E=EB15D1<?G5F5BD89C=5D8?41<<?GC?><I?>5613D?BD?253?>C945B541>43?=@1B54*?6EBD85B
E@41D5 D89C =5D8?4 G5 14?@D54 1 =5D8?4 31<<54 K?2C5BF1D9?>1< 31EC1< 49C3?F5BIL

463

%2C5BF1D9?>1< 31EC1< 49C3?F5BI 9C D85 @B?35CC ?6 8I@?D85C9J9>7 31EC1< B5<1D9?>C89@C 6B?=
?2C5BF1D9?>1<41D1*89C=5D8?481C1DDB13D541DD5>D9?>6B?=D85=1389>5<51B>9>73?==E>9DI
D8ECB56<53D9>7D85496693E<DI?63?>4E3D9>7(*CG89381B5D857?<4CD1>41B46?B5CD12<9C89>7
31EC1< B5<1D9?>C89@C "?@5J&1J 5D 1< #??9: 5D 1< &5D5BC 5D 1<%2C5BF1D9?>1<31EC1<49C3?F5BI81C255>1@@<954D?=1>I4?=19>CCE381C9>53?>?=93CD?
E>45BCD1>41>4=?45<D859=@13D?6=?>5D1BI@?<93I1>49>29?9>6?B=1D93CD?9>65B>5DG?B;
CDBE3DEB5C6B?=75>55H@B5CC9?>41D11>4@B9?B9D9J55H@<?B1D?BI5H@5B9=5>DC!1<19>1D81>5D
1<

 ?G5F5B G5 81F5 I5D D? 69>4 =1>I 1@@<931D9?>C ?6 ?2C5BF1D9?>1< 31EC1<
49C3?F5BI9>54E31D9?>1<41D1*85B56?B5G5B19C5D856?<<?G9>7DG?B5C51B38AE5CD9?>C('C (' -81D4?G5?2D19>6B?=D85?2C5BF1D9?>1<31EC1<49C3?F5BI=5D8?4141@D54D?
B51<G?B<454E31D9?>1<41D1 ('  ?G 49665B5>D 1B5 D85 B5CE<DC ?6 D85 3?>F5>D9?>1< =5D8?4 1>4 ?2C5BF1D9?>1<
31EC1<49C3?F5BI
*?144B5CCD8512?F5('CG51DD5=@DD?1@@<ID85?2C5BF1D9?>1<31EC1<49C3?F5BI=5D8?4
G893881C1<B514I255>1@@<9549>?D85B695<4CD?B51<G?B<454E31D9?>1<41D1
?B ('  G5 1@@<I D85 31EC1< 1>1<IC9C 1<7?B9D8= 31<<54 CDBE3DEB1< 17>?CD93 =?45<9>7)# G8938 81C C8?G> 7B51D @5B6?B=1>35 9> B51<G?B<4 29?<?7931< 41D1 D? B51<G?B<4
54E31D9?>1<41D11>4F5B96I9DC56653D9F5>5CC?B(' G53?=@1B5?EB@B5F9?EC<I@B?@?C54
=5D8?4G9D8D85=5D8?4EC9>7)#1>43<1B96ID859B381B13D5B9CD93C1>449665B5>35C*8B?E78
D85C5 ('C G5 19= D? @B5C5>D 1 >5G @1D8 6?B =?B5 56653D9F5 5F945>35 5HDB13D9?> 6B?=
54E31D9?>1<41D1







 +'+!,*85 =5D8?4?<?7I EC54 9> D89C CDE4I 3?>C9CDC ?6 DG? CD5@C *85 41D1C5DC EC54 1B5 <9CD54 9>
*12<5  *85 69BCD CD5@ 9C 6?B ('  1>4 9D 9>F?<F5C 5H1=9>9>7 G85D85B )# 9C 1@@<9312<5 D?
54E31D9?>1<41D11>4G85D85B9D@B?F945C56653D9F5B5CE<DC)@5396931<<IG5C5<53D41D1C5D/6?B
1>1<IC9C1>41@@<I)#D?9D*85C53?>4CD5@9C6?B(' 1>49D9>F?<F5C3?=@1B9>7D85)#
21C54=5D8?4G9D8?EB@B5F9?EC<I@B?@?C54=5D8?4?BD89CCD5@G51@@<I)#D?41D1C5D
0G89389C6B?=41D1C5D.6?BD85C1=513D9F9DI1CD81D9>D85@B5F9?ECCDE4I2I%;E=EB15D
1<

> 1@1>?>5D12<5D?B1C9=9<1B45F9359C49CDB92ED54D?5138CDE45>DE>45BD85)38??<&B?:53D*89C@B?:53DEC5C1<51B>9>71>1<ID93C@<1D6?B=31<<54D85"CICD5=%71D1
1>43?<<53DCD85<?741D1133?B49>7<I>D853EBB5>DCDE4IG5ED9<9J5D85<?741D16B?=
D85"CICD5=
*851>1<IC9C9C3?>4E3D541C6?<<?GC97EB5 9BCDD85<?741D13?>D19>9>79>6?B=1D9?>
?>D85CDE45>DCM13D9F9D95C9>D85@<1D6?B=9>3<E49>7G81DD8513D9F9D95C1B51>4G85>D85I1B5
3?=@<5D54 9C 3?<<53D54 )53?>4 D89C 9>6?B=1D9?> 9C CE==1B9J54 D? C8?G D85 >E=25B ?6
?@5B1D9?>C G9D8 D85 4979D1< D5HD2??;C 9> 5138 3<1CC *89B4 1 31EC1< C51B38 9C 3?>4E3D54 D?
45D5B=9>5G85D85B131EC1<B5<1D9?>C89@5H9CDC?EBD8D85B5CE<DC1B549C@<1I541C1D12<59>
D856?B=?6 C1>4 C133?B49>7D?1C5DD8B5C8?<49>1<<ID85B5CE<DC1B5?ED@ED1C149B53D54
13I3<937B1@8
464

*12<5  %#&'!(%$%*' &&'% $ %$+$)!%$")*!(('(#!+''($

#1>E1<?=@1B9C?>

)().

ED?=1D54?=@1B9C?>

)()/)()0('$)&
1:E>9?B8978
C38??<
1:E>9?B8978
C38??<

ED?=1D541EC1<H@<?B1D9?>
)#ED?=1D541EC1<H@<?B1D9?>)#)!($#$1;1>9C89

%;E=EB15D1<

1E>9F5BC9DI*89CCDE4I

1:E>9?B8978
C38??<*89CCDE4I

!*' #5D8?4%F5BF95G
)()'%#$!+'(!), )()-5 EC54 D85 41D1 3?<<53D54 6B?= 1> 54E31D9?>1< 41D1 1>1<IC9C 3?>D5CD 3?>4E3D54 2I D85
?E>39<6?BF945>35B9F5>4E31D9?>(5C51B38

)@5396931<<I G5 ED9<9J54 D85 54E31D9?>1< 41D1 ?2D19>54 G9D8 D85 " CICD5= 1D 1
E>9F5BC9DI*85C5 41D19>3<E4544979D1<54E31D9?>1<=1D5B91<2B?GC9>72581F9?B9>6?EB3?EBC5C
?665B541C!IE 41D19>D85
1>4
13145=93I51BC*12<5 *85<?7CG5B5?B71>9J54
133?B49>7D? 3<1CCD9=51>43?>D5>D C1>441D1G5B5@B?4E3546?B51389>4931D?B6?B 5138
3<1CC*85 9>4931D?BC1B5<9CD54 9>*12<5 *12<5  *##',% )()$)&'







&"
)@B9>7)@B9>7)@B9>7)@B9>7

()#(





465

$ $)#(    *12<5 $!)%'('%# )()
&*($#$*"&)+#%&,%&)+#**&"&
,"&
)+##(!,#(!$+##***%$"$

'&%($#?E>D?6"53DEB5&5B9?4
)E=?6%@5B1D9?>CF5B175%@5B1D9?>C
)E=?6DD5>49>7)DE45>DC$E=25B?6&175C?F5B549>"53DEB5&5B9?4
$E=25B?65F935C+C549>"53DEB5&5B9?4
)E=?6#1B;5BCF5B175#1B;5BC
$E=25B?6*51389>7#1D5B91<C+C549>"53DEB5&5B9?4
F5B1759>1<H1=)3?B5?6DD5>49>7)DE45>DC
-85D85BD85?EBC5G1C%><9>5?B$?D)()'%#*$!%'!  %%" )()

?B1D1C5D0G51>1<IJ54D85<?7C?6D8541ICG85>13D9F5B5149>713D9F9D95CG5B53?>4E3D54
1DD85C1=5:E>9?B8978C38??<9>C38??<I51BC
1>4
3D9F5B5149>79CK3?>C945B54
25CDCE9D546?BD5HD2??;B5149>785<@9>7<51B>5BCE>45BCD1>4E>61=9<91B9>6?B=1D9?>9>D5HDC
1>4@B?F949>71CDBE3DEB541@@B?138D?<51B>9>7L*?I?;1G15D1<@ 3D9F5B5149>7
5>12<5C?>5D?B514AE93;<I1>4E>45BCD1>49=@?BD1>D@?9>DC1>49D9>F?<F5CD85EC5?6=E<D9@<5<51B>9>7 D??<C CE38 1C 52??;C 1>4 <51B>9>7 1>1<ID93C D??<C *85 D1B75D 3<1CC 3?>C9CD54 ?6
D8B55C5CC9?>C>1=5<I1I 1I 1>41I 2??;=5=?C=1B;5BC1>4D9=5BCG5B5
EC549>D8513D9F9D95C>1449D9?>AE9JJ5CG5B514=9>9CD5B541>4B5149>7C@554G?B4C@5B
=9>ED5-&#G1C=51CEB54256?B51>416D5BD853<1CC@B51>4@?CD3<1CC*859>4931D?BC
3?=@ED546B?=D85<?741D1?6D85C513D9F9D95C1B5<9CD549>*12<5 *12<5 $!)%'(% )()&*($#( ( ( #(! #(! ## ##$
## ## ( ($
( ( *# *# *# '+0&('+0&)*-&#&(-&#&)*)+#)#('&%($#3D9F5B5149>741C82?1B4EC5?>1I 
3D9F5B5149>741C82?1B4EC5?>1I 
3D9F5B5149>741C82?1B4EC5?>1I #1B;5BEC5?>1I #1B;5BEC5?>1I #5=?EC5?>1I #5=?EC5?>141IG9D8>?3<1CC#5=?EC5?>1I #5=?EC5?>1I (5149>7?@5B1D9?>C?>1I (5149>7?@5B1D9?>C?>141IG9D8>?3<1CC
(5149>7?@5B1D9?>C?>1I (5149>7?@5B1D9?>C?>1I *9=5BEC5?>1I *9=5BEC5?>1I *9=5BEC5?>1I )3?B59>D85@B5AE9J
)3?B59>D85@?CDAE9J
-?B4C@5B=9>ED5256?B5D8513D9F5B5149>713D9F9DI
-?B4C@5B=9>ED516D5BD8513D9F5B5149>713D9F9DI
)3?B5?6CE==1BI16D5B13D9F5B5149>73<1CC


466



&&"! "%'!) #  )# !1<19>1D81> 5D 1< ED9<9J5C 1 DI@5 ?6 455@ <51B>9>7 31<<54 75>5B1D9F5
14F5BC1B91<>5DG?B;$$9C;>?G>6?B9DC129<9DID?75>5B1D561;5?2:53DCD81D1B53<?C5
D? B51< ?2:53DC 2I 81F9>7 DG? >5DG?B;C >1=5<I 1 75>5B1D9F5 >5DG?B; 1>4 1 49C3B9=9>1D9F5
>5DG?B;3?=@5D5)#9C1>1@@<931D9?>?6D85$3?>35@DD?31EC1<1>1<IC9C1>49CEC54
D?45D5B=9>531EC1<B5<1D9?>C89@C1=?>7B51<G?B<454E31D9?>1<41D197EB5 )@5396931<<I)# AE1>D9695C D85 <9;5<98??4 ?6 31EC1< B5<1D9?>C89@C 25DG55> 9>4931D?BC 1>4 5H1=9>5C
G85D85B31EC1<B5<1D9?>C89@C5H9CD*858I@5B@1B1=5D5BC1B5D85C1=51CD8?C5EC549>D85
?B979>1<
CDE4I

!*'<7?B9D8=?6)DBE3DEB1<7>?CD93#?45<9>7)#
4?@D546B?=             ) % , %)!$ '%# )  %('+)!%$" *(" !(%+'- #) %
&))%'",%'"*)!%$")  )()
*?1>CG5B(' G55H1=9>54G85D85B)#9C1@@<9312<5D?54E31D9?>1<41D11>4G85D85B9D
@B?F945C56653D9F5B5CE<DC)#G1C1@@<954D?41D1C5D/6?B1>1<IC9C97EB5 C8?GCD85B5CE<D?6)#6?B41D1C5D/
!*'(5CE<D?6)#6?B1D1C5D/

467#()(*!+)#&'(#!&')!(*851>1<IC9CB5F51<54131EC1<B5<1D9?>C89@25DG55>D85>E=25B?6D51389>7=1D5B91<CEC54
4EB9>73<1CC#,"&1>4D851F5B175>E=25B?6?@5B1D9?>C,%&97EB5 1BB?G
*89CB5CE<D9C9>DE9D9F5<IE>45BCD1>412<5B5CE<DD85=?B5DI@5C?6D51389>7=1D5B91<C1F19<12<5D857B51D5BD85>E=25B?6?@5B1D9?>C $)#(&#()(*&')!('?G5F5B131EC1<B5<1D9?>C89@G1C6?E>425DG55>D85>E=25B?645F935DI@5CEC549>3<1CC,"&1>4D85C5=5CD5B7B145@?9>D1F5B175?6D85CDE45>D1DD5>455C**97EB5 
1BB?G *89CB5CE<D3?E>D5BCD859>DE9D9?>1C=?B5DI@5C?645F935C<514D?89785B7B145C6
D89CB5<1D9?>C89@9C13DE1<<I131EC1<?>5D85>9D9C1>5G54E31D9?>1<D538>9AE5G?BD8DBI9>7
*8531EC1<B5<1D9?>C89@25DG55>G85D85BD853?EBC5G1C?665B54?><9>5%$"$1>45>4
?6C5=5CD5B7B145C**97EB5 1BB?G G1C1<C?3?E>D5B9>DE9D9F5*85>?D9?>?6
13145=93 7B145C 9>3B51C9>7 G9D8 D85 1F19<129<9DI ?6 ?><9>5 3<1CC5C 9C 496693E<D D? 5>F9C9?>
?G5F5B96CE38B5<1D9?>9CDBE5D85>>5G54E31D9?>1<@?CC929<9D95CG9<<253?=51@@1B5>D*&'&($#)'!&!($#'%
31EC1<B5<1D9?>C89@G1C6?E>425DG55>D85D?D1<>E=25B?6?@5B1D9?>C)+#%&1>4D85
>E=25B ?6 @1BD939@1D9>7 CDE45>DC )+# ** 97EB5  1BB?G  ?G5F5B D89C 31EC1<
B5<1D9?>C89@G1CB5F5BC54*85>E=25B?6?@5B1D9?>C?63?EBC5494>?D9>3B51C5D85>E=25B
?6CDE45>DC>CD514D857B51D5BD85>E=25B?6CDE45>DCD85=?B5?@5B1D9?>C9>F?<F54D89C
31C5C8?GC1=?B5>1DEB1<31EC1<B5<1D9?>C89@  %,!'$)') '(*")(%) %$+$)!%$"#) %$%('+)!%$"
*("!(%+'-  )()
*?1>CG5B(' G51@@<954)#D?41D1C5D0G8938G1C6B?=D85C1=513D9F9DI1CD81D9>D85
@B5F9?ECCDE4I41D1C5D.97EB5 C8?GCD85B5CE<D?6)#6?B41D1C5D0

!*' (5CE<D?6)#6?B1D1C5D0







468



 ('$#(&*#($#'
*851>1<IC9CB5F51<54D8556653DC?6D859>D5BF5>D9?>CG519=54D?F1<941D5?B5H1=@<5G5
6?E>4 D81D 13D9F5 B5149>7 13D9F9D95C ?> 1I  )+# ** 16653D54 D85 AE1<9DI ?6 =5=?
CE==1B95C)+#)#(97EB5 1BB?G 3*&'&($#)'!&!($#'%
> 1449D9?> G5 >?D54 C?=5 B5F5BC1<C ?6 D85 D9=5 1H9C G8938 CE775CD54 D85 @?CC929<9DI ?6
31EC1<9DI?B5H1=@<5G5?2C5BF541B5<1D9?>C89@25DG55>-&#9>D85@?CD3<1CC13D9F9DI
-&#&)*1>4=5=?EC5?>1I ## 97EB5 1BB?G 9F5>D85?B45B?6D85
13D9F9D95CD8549B53D9?>?6D8531EC1<B5<1D9?>C89@=ECD25B5F5BC54?G5F5B96D8549B53D9?>
G1C14:ECD54D8531EC1<B5<1D9?>C89@25DG55>=5=?EC51>4-&#31>>?D2551C9<I6?E>4
D8B?E78 8E=1> ?2C5BF1D9?> 5>35  D81D 31> @B?35CC <1B75 1=?E>DC ?6 41D1 9C 1D 1>
14F1>D175 1C 9D 9C 12<5 D? CE775CD D85 5H9CD5>35 ?6 B5<1D9?>C89@C D81D 1B5 >?D 3?>C945B54
B5<5F1>D2I8E=1>C '#$)'!&!($#'%
*85B5CE<DC1<C?CE775CD>?D?><ID85K@?CC929<9DID81D131EC1<B5<1D9?>C89@5H9CDCL2ED1<C?D85
K@?CC929<9DI D81D >? 31EC1< B5<1D9?>C89@ 5H9CDCL ?B 5H1=@<5 G5 494 >?D ?2C5BF5 1> 5475
25DG55>13D9F5B5149>741C82?1B4EC5?>1IC 1>4 ( 1>4( *89CB5CE<D
9>4931D54 D81D D85 13D9F9D95C G9D8 D85 41C82?1B4 ?> 1IC  1>4  G5B5 >?D 8978<I 56653D9F5
G85B51CD8?C5?>1I =978D81F5255>F5BI56653D9F5>6EDEB545C97>C6?B13D9F5B5149>7
3<1CC5C1>5=@81C9C?>D8513D9F9D95C3?>4E3D54?>1I 3?E<4253?>C945B54656653D9F5
CE3845C97>CG9<<3?>DB92ED5D?9=@B?F9>7D85<51B>9>756653D9F5>5CC?6<51B>5BCG89<5B54E39>7
D852EB45>?>D51385BC






  (*")(%( )%,%)!$'%#) %('+)!%$"*("!(%+'-#) %&))%
'",%'"*)!%$")  )()

B?= D85C5 B5CE<DC G5 3?>69B=54 D85 @?CC929<9DI ?6 1@@<I9>7 )# D? 54E31D9?>1< 41D1 1>4
75>5B1D9>7EC56E<B5CE<DC?G5F5B>?D1<<31EC1<B5<1D9?>C89@CC8?G>9>D85G5B5DBE5
D85B56?B531B5=ECD25D1;5>G85>9>D5B@B5D9>7D85=>14F1>D175?6D89C=5D8?49CD81D9D
31>@B?35CC<1B751=?E>DC?641D11>4CE775CD31EC1<B5<1D9?>C89@CD81D8E=1>C=1I>?D25
12<5D?1CCE=5%EBB5CE<DC1<C?9=@<954D85@?CC929<9DI?6C5<53D9>7D8531EC1<B5<1D9?>C89@C
D81DC8?E<4255H1=9>54=?B5455@<I>1449D9?>D85B5CE<DC9>4931D54D81D)#31>25EC56E<
6?B 54E31D9?>1< 41D1 1>1<IC9C ?G5F5B =1H9=9J9>7 D89C =5D8?4 B5AE9B5C D85 @B?@5B
9>D5B@B5D1D9?>1>4F5B96931D9?>?6D8531EC1<B5<1D9?>C89@C9>D85?ED@ED %, !'$) ' )  '(*")( % )  %$+$)!%$" #) % $ %('+)!%$"
*("!(%+'-  )()

-5 3?=@1B54 D85 1ED?=1D93 31EC1< C51B38 =5D8?4 )# G9D8 D85 @B5F9?EC 1ED?=1D93
3?=@1B9C?> =5D8?4 1>4 6?E>4 C?=5 49665B5>35C )@5396931<<I )# 4?5C >?D B5AE9B5 D859>4931D?BCD?2569H54G85B51CD85@B5F9?EC=5D8?4B5AE9B5CD859>4931D?BC256?B53?=@1B9C?>
D?2569H54>D89CG1I)#=1I5H@1>4D855H@<?B1D9?>?6613D?B31>4941D5C1>4<514D?D85
@?CC929<9DI?669>49>73?=@1B1D9F59>4935CD81D8E=1>CG?E<4>?D81F5255>12<5D?3?>C945B
*89C651DEB59C1>14F1>D175?6D851ED?=1D9331EC1<C51B38=5D8?41>49>4931D5C9DC@?D5>D91<
D??@5>D854??B6?B>5GB5C51B38



469



 !#!))!%$(44B5CC9>7 ('  9>F?<F54 5HDB13D9>7 1>4 1>1<IJ9>7 D85 41D1 G9D8 213;7B?E>4 9>6?B=1D9?>
@B?F9454 2I 1 B5C51B385B G8? 3?>4E3D54 1> 13D9F5 B5149>7 3<1CC CDE4I EC9>7 D85 "@<1D6?B=>6E<<I1ED?=1D9>7D85=9>9>7@B?35CCG9D8?EDD85>5546?B1>I=1>E1<<I9>@EDD54
9>6?B=1D9?>D85213;7B?E>49>6?B=1D9?>6B?=G8938D8541D1G5B5?2D19>541>4D81D9CD954D?
D8541D19>C?=5G1I=ECD25CD?B54D?1ED?=1D5D855HDB13D9?>?65F945>35-5=ECD>?D5
D81D  1CCE=5C D81D 54E31D9?> 9C 13I3<93 1<D8?E78 F1B9?EC 1>4 3?=@<5H 31EC1<
B5<1D9?>C89@C31>258I@?D85C9J54)#@B?4E35C31>4941D531EC1<B5<1D9?>C89@CD81D81F5
D85 D9=5 1H9C B5F5BC54 1>4 D8EC >554C 9=@B?F5=5>D C? D81D 9D 4?5C >?D @B?4E35 CE38
B5<1D9?>C89@C #&"!)!%$(+><9;5D853?>F5>D9?>1<=5D8?4D85=5D8?49>D89CCDE4IG1C>?D<9=9D54D?1C9>7<59>4931D?B
6?B3?=@1B9C?>5>35D85=5D8?4@B?@?C5485B59>9C12<5D?69>43?=@1B1D9F59>4935CD81D
8E=1>C31>>?D1CCE=5CE381CD85B5<1D9?>C89@25DG55>=5=?EC51>4-&#9>)53D9?>
  
CC8?G>9>)53D9?>   ;>?G9>7C@5396931<<IG893841IMC13D9F9D95C1B556653D9F5=1I
B54E35D852EB45>?>D51385BC1>49=@B?F5<51B>9>756653D9F5>5CC





  

*89CCDE4I1DD5=@D54D?1@@<I)#D?D851ED?=1D545HDB13D9?>?65F945>356B?=B51<G?B<4
54E31D9?>1< 41D1 1>4 5H1=9>54 9DC 56653D9F5>5CC *85 B5CE<DC C8?G54 D81D D85 31EC1<
B5<1D9?>C89@C?ED@ED2I)#4?>?D>535CC1B9<I9>4931D5DBE531EC1<B5<1D9?>C89@C?G5F5B
CE38B5CE<DC1B5>?D>535CC1B9<I=51>9>7<5CC1>4D85129<9DID?1ED?=1D931<<I@B?35CC<1B75
1=?E>DC ?6 41D1 @B5C5>DC D85 @?CC929<9DI ?6 F1B9?EC 31EC1< B5<1D9?>C89@C D81D 1B5 496693E<D D?
945>D96I21C54?>8E=1>1CCE=@D9?>C>1449D9?>D8B?E78D85@B?@?C54=5D8?4G5C8?G54D85@?CC929<9DI?6@B?35549>7G9D8
D85 1>1<IC9C 5F5> G9D8?ED 69H9>7 D85 9>4931D?BC 6?B 3?=@1B9C?> *89C 651DEB5 5>12<5C D85
945>D96931D9?>?63?=@1B1D9F59>4935CD81D8E=1>C31>>?D1CCE=5D9C1<C?1>14F1>D175?6
EC9>71>43?=@ED5BC1C9D31>@B?F945>5G@5BC@53D9F5CD81D7?25I?>4D8?C545B9F542I
8E=1>CEDEB5 CDE495C C8?E<4 3?>4E3D 31EC1< 9>65B5>35C 9> 1449D9?> D? 31EC1< C51B38 1>4
5F1<E1D5D85=17>9DE45?631EC1<56653DC>1449D9?>D85I=1I3?=@1B5?EB=5D8?4G9D8?D85B
1<7?B9D8=C6?B31EC1<C51B38%>5?6D857?1<C?6D853EBB5>DB5C51B38G1CD?9=@<5=5>DD85
@B?@?C54=5D8?41C1CICD5=6?BD51385BC1>4@B?F945B51<G?B<45F945>35?6<51B>9>7>
13895F9>7D89C7?1<D85=5D8?4=ECD259=@B?F541>445F5<?@54G89<5?2D19>9>76554213;
6B?= D51385BC 21C54 ?> D85 1CCE=@D9?> D81D D85 D??< G9<< 25 EC54 9> 1> 13DE1< 54E31D9?>1<
5>F9B?>=5>D!

*89CCDE4IG1CCE@@?BD542I$% &$&

1>4 )&)!!$ &



 



/D73A$ *6/B7A3D723<130/A3232C1/B7=<.'0'/&,1.+),$"1!0',+)01"'#/  =C<17:4=@D723<13@7D3<2C1/B7=<%3A3/@16
&3>B3;03@2C1/B7=</:2/B//</:GA7A
1=<B3AB%3B@73D324@=;6BB>AA7B3A5==5:31=;D73E3232/B/16/::3<53 

470

/:/7</B6/<  =C23B # CG=<  =>3H$/H   &30/5 !  &B@C1BC@/: /5<=AB71
;=23:7<52D3@A/@7/::3/@<7<5=41/CA/:5@/>6A.'2-.#-.'+0.'2   %3B@73D324@=;
6BB>A/@F7D=@5/0A    
/:/7</B6/<  =C23B # CG=<  =>3H$/H   &30/5 !  &B@C1BC@/: /5<=AB71
;=23:7<52D3@A/@7/::3/@<7<5=41/CA/:5@/>6A,1.+),$!&'+##.+'+%#/#.!&  =>3H$/H!C/<23B&16J:9=>4'=:AB7967<
C<3'=E/@2A/:3/@<7<5B63=@G=4
1/CA334431B7<43@3<13<+0#.+0',+),+$#.#+!#,+!&'+##.+'+%>>    $! %
!/6/8/< %  %3/: E=@:2 2/B/ /227B7=</: A=C@13 4=@ ;/97<5 1:7<71/: 2317A7=<A +0#.+0',+)
,1.+),$--)'#"+"/'!#"'!)#/#.!&   !==78!$3B3@A/<H7<5,A1637A16:3@&16J:9=>4
7AB7<5C7A67<51/CA34@=;
34431B CA7<5 =0A3@D/B7=</: 2/B/ !3B6=2A /<2 03<16;/@9A &# ,1.+) ,$ !&'+# #.+'+%
#/#.!&  "/9/<7A67 '  40.!0'+% #2'"#+!# ,+ )#//,+ "#/'%+ 1/'+% .#) 3,.)" #"1!0',+) "0 -+-1 )'/&#"*/0#.5/0&#/'/.G=B=(<7D3@A7BG+-+#/#
#5/B/  !/8C;2/@ % 9I/>N</@  /A<7<3 ! "  :/</5/<   3G=<2 :3/@<7<5
/</:GB71A @/;3E=@9 4=@ B316<=:=5G3<6/<132 3D723<130/A32 32C1/B7=< /<2 :3/@<7<5 <  
+0#.+0',+),+$#.#+!#,+,*-10#./'+"1!0',+,.(/&,-.,!##"'+%/>>   A7/
$/17471&=173BG4=@=;>CB3@A7<2C1/B7=<$&
#9C;C@/=@79=A67#5/B/
3D3:=>;3<B=4/L1=<B@=:5@=C>MA3/@16;3B6=24=@
/CB=;/B323D723<133FB@/1B7=<4@=;@3/:E=@:232C1/B7=<2/B/#!&+'!)#-,.0)=:
  "= >> />/<+-+#/#
$3B3@A/<H7<5&16J:9=>4
)#*#+0/,$!1/)'+$#.#+!#,1+"0',+/+")#.+'+%
)%,.'0&*/> '63!'$@3AA
&:/23&$@7<A:==$ 3/@<7<5/</:GB71AB671/:7AAC3A/<227:3;;/A*#.'!+#&2',.)
!'#+0'/0      '=G=9/E/+!/8C;2/@%=<2='=@79=A67#5/B/
1B7D3@3/27<52/A60=/@2
7</:3/@<7<5/</:GB71A3<6/<132:/<5C/53:3/@<7<53<D7@=<;3<B4431BA=<:3/@<7<5036/D7=@/<2>3@4=@;/<13,1.+),$,*-10#./'+"1!0',+  

471

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

What does process mining of feedbackbehavior reveal about problem-solving inchemistry undergraduates?
Anveshna SRIVASTAVAa*, & Chandan DASGUPTAa
IDP in Educational Technology, Indian Institute of Technology Bombay, India*anveshna.sriv@gmail.com

aAbstract: Feedback is known to play a powerful role in the process of learning andproblem-solving. In a problem-solving context where feedback is expected to positively
influence the process of achieving the target goal, feedback information can be elicitedthrough feedback-seeking questions. In this study, we analyze different purposes forwhich feedback is both sought and given by chemistry undergraduate students (n=3)
as they solved representational problems concerning a medicinal drug. The students
were left unsupervised and, with their consent, were video-recorded to capture theirproblem-solving process. We used the transcript of the video data and coded students’interactions to identify feedback events concerning ‘the task’, ‘process’, and ‘regulation’of the problem-solving activity. These events were then used to generate processmodels of feedback behavior of students, individually and collaboratively, throughProM- a process mining tool. Our models suggest that feedback questions on tasks
and processes dominate the beginning of the problem-solving activity, while feedback
responses on processes and regulation dominate the end phase. Our work has
implications for designing activity problems. We suggest designing well-thought-out
focus questions at the beginning of problem-solving exercises to both scaffold andfacilitate students’ learning because it aligns with the natural unsupervised flow ofproblem-solving, as is observed in this study.
Keywords: Feedback, feedback-seeking, questions, responses, process mining

1. Introduction
We seek feedback because, as humans, we are inherently curious and want to know how weare doing and how we have done (Cutumisu, & Lou, 2020). The process of seeking feedback
and the corresponding gathered response positively influences learning (Rattan et al., 2015).
It helps learners to verify their position, gather insights about strategies that can be used,
and/or provide appropriate metacognitive support (Wang & Wu, 2008). In their feedback
model, Hattie & Timperley (2007) identified that feedback questions were concerned withtasks, processes, self-regulation, and self, while feedback responses were concerned withanswering questions about goal-directed action, overall progression (say, about a concept),and future possibilities. In this study, we draw upon this feedback model to map the trajectories
of individual students as they are involved in feedback behavior during a problem-solving task.
We use ProM (Günther & Van Der Aalst, 2007), an open-source process mining tool
to analyze and visualize the process of interaction between individual student’s feedbackseeking questions and responses. It uses multiple algorithms to shed light on the sequence inwhich a process occurs. Here, we use the Directly Follows Graph, Extended Causal Graph
Miner algorithm, and Heuristics Miner algorithm (Weijters, Van Der Aalst, & De Medeiros,
2006) to visualize the trajectories of interaction among different feedback behaviors.

2. Research Questions
This study addresses the following research questionszWith no specific instructions on collaboration, do students engage in feedback behavior
during a problem-solving activity?

472

z
z

If yes, could we map the sequence of flow in which feedback behavior is manifested?
What does the sequence reveal about the trajectory of individual students’ feedback
behavior in a problem-solving activity?

3. Methods
3.1 Participants, Data Sources & Task
Three male post-graduate chemistry students, who had completed an introductory course on
stereochemistry, volunteered for a study based on tasks about representations of a medicinal
drug. Students were seated at a round table and were individually given task-specificworksheets. They were free to work individually or collaboratively and the activity was videorecorded with minimal presence of the researcher. The video data formed the primary data
source for this study. The video (1.5 hours) was transcribed using a combination of ‘Otter’, an
online transcribing interface, and manual transcription. The study was approved by the IRB of
the institute and students’ consent were received before the conduct of the study.

3.2 Data Coding
Following Hattie & Timperley (2007), we used codes to map feedback questions andresponses pertaining to- a) form and structure of tasks (FQT & FRT- Feedback-seeking
Question about the Task and Feedback Response on the Task), b) processes involved (FQP
& FRP: Feedback-seeking Question about the Processing and Feedback Response about theProcessing of the task), and c) monitoring (FQR & FRR: Feedback-seeking Question about
Regulation and Feedback Response about Regulation of the task). We use codes S1, S2, &S3 for the three students.

4. Analysis and Findings
4.1 Students engage in feedback behavior on their own and PRoM algorithms mapped
the sequence of the flow of feedback questions and responses

a)
b)
c)
d)
Figure 1. a) A feedback-seeking episode (Adapted from Swamy et al., 2022), b) a
‘Directly Follows Graph’ (DFG) representing the path for all the participants, (c) an
Extended Causal Activity Graph, d) Heuristic Net Graphs for each student
Figure 1 a) depicts two students engaging in feedback behavior during the activity even in theabsence of no specific instructions on collaboration. Further, a Directly Follows Graph (‘DFG’;
Figure 1 b)) and an Extended Causal Graph (Figure 1 c)) mapped the overall flow of feedback
events moving from the start (green circle in Figure 1 b) & ‘start’ box in 1 c) to the end (redcircle in 1 b) & ‘end’ box in 1 c) of the problem-solving task for the participants. The nodes andthe lines in the two graphs represent feedback behavior (questions or responses) and thetransitions in these behaviors respectively. The numbers within the DFG nodes give the countof the events. Nodes with the greatest number of events are depicted in darker shades andvice-versa. The graphs reveal that the maximum number of feedback-seeking questions wereasked about the task (FQT-21) and the maximum number of feedback responses were given473

on the process (FRP-52), suggestive of a potential emphasis on process strategies to tackle
questions on task. Further, from both graphs, we find that FQT (21 events) & FQP (10 events)
behavior dominate the beginning phase and FRP (52) & FRR (16) behavior dominate the end
phase of the problem-solving activity, suggesting that

4.2 Individual differences in interaction trajectories
Heuristic Net Graphs (Fig. 1 d)) mapped the trajectory of all students’ interaction between
feedback questions and responses from the start (green circle) to the end (orange circle) of
the problem-solving task. We find that S3 had only 34 events of feedback exchange, while S1& S2 had more than 60 events for the process. We also find that S3 asked extremely few
questions (7) overall. Thus, we can conclude that S1 and S2 were more actively involved in
feedback behavior than S1. Thus, the graph gives an insight into the dynamics of interaction
among the participants during the problem-solving process.

5. Discussion
In summary, we found that the participants naturally drifted toward collaborative problemsolving. The use of PRoM tools revealed that feedback-seeking questions on ‘task’ and‘process’ dominated the start of the activity, while feedback responses on ‘process’ and‘regulation’ dominated the end of the activity. Based on this, we suggest designing wellthought-out focus questions at the beginning of problem-solving exercises to both scaffold andfacilitate students’ learning. The Heuristics Net graph revealed answers to broader questions
like- ‘Who was more/least engaged in the feedback event?’ to finer questions like- ‘What formof feedback event was prominent in an individual’s interaction with the other groupmembers?’ Our work contributes to the field by visualizing the process of feedback behavior
and mapping the sequence of feedback events at both individual and aggregate levels. It has
implications for both pedagogy and research, where such insights could be used to design
scaffolds to promote scientific discourse (Seng & Hill, 2014), infer design principles forfacilitating collaborations, and/or use/design tools to better visualize problem-solving behavior.

Acknowledgements
We would like to thank all the participants of the study. We also extend our gratitude toNarasimha Swamy for his help in providing access to the data.

References
Cutumisu, M., & Lou, N. M. (2020). The moderating effect of mindset on the relationship between
university students’ critical feedback-seeking and learning. Computers in Human
Behavior, 112, 106445.
Günther, C.W., Van Der Aalst, W.M. (2007): Fuzzy mining–adaptive process simplification based on
multi-perspective metrics. In: International Conference on Business Process Management. pp.
328–343. Springer.
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1),81–112. https://doi.org/10/bf4d36
Seng, M., & Hill, M. (2014). Using a dialogical approach to examine peer feedback during chemistry
investigative task discussion. Research in Science Education, 44(5), 727-749.
Rattan, Savani, Chugh, & Dweck (2015). Leveraging mindsets to promote academic achievement:
Policy recommendations. Perspectives on Psychological Science, 10(6), 721-726.
Swamy, N., Srivastava, A., & Dasgupta, C. (2022). Investigating the Role of Students' FeedbackSeeking Actions and Adapting to the Distributed Competence. In Iyer, S. et al. (Eds.).
Proceedings of the 30th International Conference on Computers in Education (ICCE). APSCE.
Wang, S. L., & Wu, P. Y. (2008). The role of feedback and self-efficacy on web-based learning: The
social cognitive perspective. Computers & Education, 51(4), 1589-1598.
Weijters, A. J. M. M., van Der Aalst, W. M., & De Medeiros, A. A. (2006). Process mining with theheuristics miner-algorithm. Technische Universiteit Eindhoven, Tech. Rep. WP, 166, 1–34474" !   !" " "   #" !
#" !"$  #" ! #" " %$ #  !%$"" "  "$   (
$"$  %"##  " !%$"  "# # $%(   %
#"&"#$('              !""#" !  "" $ !"&%
 "" ""! "" $ !"&%
 "" #"  "  "" $ !"&%
5K:7479 @5G 76G FI   %23< <=>.B 38?/<=31+=/< 898$ 7+49; <=>./8=< :/;09;7+8-/ 38 -97:>=/;
:;91;+77381 +8. 809;7+=398 %/-289691B38=/1;+=/. % -9>;</< += + %+3@+8/</;/</+;-2 >83?/;<3=B  !98$ <=>./8=< <=;>116/ 38 38=;9.>-=9;B :;91;+77381 .>/ =9
<B8=+A+8.6913-+6=23853816373=+=398<;/<>6=3813869@/;1;+./<-97:+;/.=9+.?+8-/.
:;91;+77381 $3736+;6B%-9>;</1;+./<+;/69@/;.>/=9<>,4/-=<:/-303-./7+8.< /8./; +8. /8=;B -2+88/6< 37:+-= 9>=-97/< @3=2 0/7+6/< /A-/66381 38 3809;7+6
6/+;8381 +?9;+,6/;/<>6=<+;/<//83838.3?3.>+6+::63-+=398<+8. >6=3<=+;";94/-=< 2+66/81/< 38-6>./ .300/;/8= 6/+;8381 :+-/< +8. ->6=>;+6 +.4><=7/8=<  #/1;/<<398
+8+6B<3< <29@< '@FDA6G5F;A@ FA !A?BGF7D -DA9D3??;@9 '!-  6H3@576
!A?BGF7D -DA9D3??;@9 !- +8. 1/8./; <318303-+8=6B +00/-= % :/;09;7+8-/
/A:6+38381 903=<?+;3+8-/ #/-977/8.+=398<38-6>./.3?/;</=/+-23817/=29.<:;9,6/7<96?381 1>3.+8-/ :;+-=3-+6 :;91;+77381 -966+,9;+=398 +8. :;94/-=
:+;=3-3:+=398=9/82+8-/<5366<   97:>=/; :;91;+77381 809;7+=398 =/-289691B38=/1;+=/. -9>;</<
6/+;83819>=-97/</8=;+8-/-2+88/6<69813=>.38+6<=>.B   

0:7 '@8AD?3F;A@#6G53F;A@ >G7BD;@F3;?76FA7@:3@57677B>73D@;@93@66;9;F3>
5;F;L7@E:;BF:DAG9:'0FAA>E 3>;9@;@9I;F:03;I3@E7?B:3E;EA@5A?BGF3F;A@3>F:;@=;@98AD
BDA4>7? EA>H;@9)3>7>;A9>G7F3>  0:7&;9:7D76G53F;A@EBDAGFBDA<75F35:;7H76
B3DF;5;B3F;A@4K(G@7 ;@6;53F;@9EG557EE8G>@A@ !/BDA9D3??;@95AGDE7
;?B>7?7@F3F;A@*;@;EFDKA8#6G53F;A@  &AI7H7D @A@ !/EFG67@FE83575:3>>7@97E;@
BDA9D3??;@96G7FA>A9;5 EK@F3J 3@634EFD35FF:;@=;@927@97F3>  0:7D78AD7 F:7
D7E73D5:CG7EF;A@E8ADF:;EEFG6K3D73E8A>>AIE• D7F:7D79D3676;887D7@57E3?A@9@A@ !/EFG67@FE;@'0'5AGDE7E ;@5>G6;@9BDA9D3??;@9
E=;>>E;@AF:7DEG4<75FE
• "A97@67D3@67@FD3@575:3@@7>E;@8>G7@57F:7>73D@;@9AGF5A?7EA8@A@ !/EFG67@FE;@'0'3@6BDA9D3??;@95AGDE7E
• 'E ;F BAEE;4>7 FA BD76;5F @A@ !/ EFG67@FE B7D8AD?3@57 ;@ '0' 5AGDE7E GE;@9 F:7;DBDA9D3??;@95AGDE79D367E3@6D7>7H3@F435=9DAG@6835FADE 
 "$!$#

!%$" " " !$$ 475%"##  "$ 0:7B3DF;5;B3@FE;@F:;EEFG6K3D7@A@ ;@8AD?3F;A@8;7>6BDA87EE;A@3>EI:A:3H7EG557EE8G>>K
5A?B>7F76F:D77FKB7EA85AGDE7E'!- !- 3@6'0'5AGDE7E 0:7FAF3>@G?47DA8B3DF;5;B3@FE
;E   ?A@9 F:7E7 EFG67@FE   I7D7 ?3>7 3@6  I7D7 87?3>7  #@FD3@57 5:3@@7>E
;@5>G676;@6;H;6G3>3BB>;53F;A@E *G>F; EF3DBDA<75FE 7J3?E 3@6EB75;3>
BDA9D3?E FD3@E87D ;@F7D@3F;A@3> 36?;EE;A@E   0:;E D78>75FE F:7 *;@;EFDK A8 #6G53F;A@E
36?;EE;A@BA>;5;7E3@6:;9:>;9:FE78875F;H7D75DG;F?7@F3@6E7>75F;A@EFD3F79;7E 55AD6;@9FAF:75GDD;5G>G?67E;9@A8F:7;@6;H;6G3>E5:AA> F:75AGDE7;E6;H;676;@FAF:7
8A>>AI;@9F:D77B3DFE  '!-!AGDE7/FG67@FEI;>>>73D@43E;55A@57BFEA85A?BGF7DBDA9D3??;@9 ;@5>G6;@9
EK@F3J 8G@63?7@F3>>A9;5 3@6BDA9D3??;@9E=;>>E   !-!AGDE7/FG67@FEI;>>67>H7;@FA?AD75A?B>7JBDA9D3??;@9F75:@;CG7E3@6
EFD;H7FA35:;7H7EB75;8;5BDA9D3??;@99A3>E   '0' !AGDE7 0:7 '0' 5AGDE7 ;@F79D3F7E EG4<75F =@AI>7697 I;F: BDA9D3? 3@3>KE;E ;@5>G6;@9 FAB;5E EG5: 3E EF3F;EF;5E  0:;E I;>> 7@34>7 EFG67@FE FA 3BB>K 5A?BGF7D
BDA9D3??;@9E=;>>EFAEB75;8;56;E5;B>;@3DK3D73E3@6EA>H7D7>7H3@FBDA4>7?E %D367E 73D@76 4K EFG67@FE ?3K 47 A4F3;@76 D7B73F76>K ;@ F:7E7 5>3EE7E  3@6 F:7E7
9D367EI;>>47D7BD7E7@F76F:DAG9:F:753>5G>3F;A@A83H7D397E7?7EF7D9D367E -7D8AD?3@57
6;887D7@57EI7D73@3>KL76GE;@9I7;9:F7636<GEF?7@FE'!- !- 3@6'0'
A8!A?BGF7D3@6075:@A>A9K!0%D367E  $$#$(###$AD 63F3 3@3>KE;E  67E5D;BF;H7 EF3F;EF;5E 5A?B3D7 9D367 6;887D7@57E  @ ;@67B7@67@F F F7EF
3EE7EE7E97@67D6;EB3D;F;7E;@!0 8A>>AI764K?73@5A?B3D;EA@E ,@7 I3K+,17JB>AD7E
6;H7DE7 @A@ !/ ?3<AD 435=9DAG@6E  I;F: BAEF :A5 F7EFE  '!- 3@6 !- BD76;5F '0' 9D367E (/- 5A@6G5FE3>>3@3>KE7E   (## 
 "#

!%$" " "   %"# " "    

034>7  6;EB>3KE 3 B7D8AD?3@57 5A?B3D;EA@ A8 +A@ !/ EFG67@FE 43E76 A@ 9D367E ;@ '!- !- 3@6'0'5AGDE7EGE;@9D7B73F76?73EGD7E+,1B AD$ H3>G7  .7EG>FE
:;9:>;9:F 3 E;9@;8;53@F B7D8AD?3@57 6;887D7@57 3?A@9 F:7E7 5AGDE7 53F79AD;7E  -AEF :A53@3>KE;E;@6;53F7EEGB7D;ADB7D8AD?3@57;@!-5A?B3D76FA'!-3@6'0'5AGDE7E *73@I:;>7 '!-3@6'0'5AGDE79D367EI7D75A?B3D34>7 EG997EF;@9EGBBADF;E@776768ADEFG67@FE;@
5A?BD7:7@6;@9G@83?;>;3DEG4<75FEI;F:;@F:7;DD7EB75F;H76;E5;B>;@7E 034>7  !# !  #$ !
!AGDE7E%7@67D#@FD3@57
5:3@@7>E$ 
0
 
$

-
 
- 
-

-AEF:A5
!-'!-'0'9D367
-AEF:A5
$7?3>7*3>7
'@6;H;6G3>BB>;53F;A@!:3@@7>,H7DE730D3@E87D  

*G>F; EF3D-DA<75F#J3? 43E765:3@@7>,H7DE730D3@E87D    !" $ " $&##  $  "#   #$%$# '$"$" %&"#&;EFAD;53>>K F:7D77J;EF763B7D57BF;A@A897@67D 43E76;@5>;@3F;A@EFAI3D6>;F7D3FGD73@63DF 
BAE;@95:3>>7@97E;@/0#*8;7>6E8AD87?3>7E2D;9>7K E3@F7 !7F3>  %7@67DE
;?B35FA@!09D367EI3E7J3?;@76GE;@93B3;DE3?B>7F F7EFF H3>G7   0:7
D7EG>FE67?A@EFD3F76F:3F87?3>7@A@ !/EFG67@FEAGFB7D8AD?76?3>7E 3EBD7E7@F76;@
034>7 476

#@FD3@575:3@@7>E;@8>G7@57A@35367?;5B7D8AD?3@57I3E;@H7EF;93F76F:DAG9:A@7 I3K+,1$H3>G7  B  -AEF :A53@3>KE;ED7H73>76:;9:7D9D367E8AD;@6;H;6G3>
3BB>;53F;A@E3@6?G>F; EF3DBDA<75FEFG67@FE5A?B3D76FAAH7DE73E3@6FD3@E87DEFG67@FE 
*G>F; EF3DBDA<75F7@FD3@FE3>EAAGFB7D8AD?767J3? 43E767@FD3@FE 3EBD7E7@F76;@034>7 +AF34>K ;@6;H;6G3>3BB>;53F;A@3@6?G>F; EF3DBDA<75FEFG67@FE67?A@EFD3F76EF34>7'0
3@635367?;5B7D8AD?3@57 I:;>7AH7DE73E3@6FD3@E87DEFG67@FE?367366;F;A@3>788ADFEFA
D735:E;?;>3D>7H7>E  $ #!$'$%$"#   "*G>F;B>7D79D7EE;A@3@3>KE7EE:AI76F:3F'!-9D367E !-9D367E 3@697@67D5A>>75F;H7>K
7JB>3;@  A8 F:7 H3D;3@57 ;@ '0' 9D367E  I:;5: ;E EF3F;EF;53>>K E;9@;8;53@F  -AEF :A5
5A?B3D;EA@E D7H73>76 E;9@;8;53@F 5A@FD;4GF;A@E 8DA? '!- 9D367E F   B   !-9D367EF  B  3@697@67DF   B FAF:77JB>3@3FADKBAI7D '!-9D367E
:36 F:7 :;9:7EF ;@8>G7@57 M   8A>>AI76 4K !- M   3@6 F:7@ 97@67D M  #@FD3@575:3@@7>6;6@FE;9@;8;53@F>K;?B35FF:7?A67> *G>F;5A>>;@73D;FKI3E3EE7EE76GE;@9
1'$ I;F:H3>G7EI7>>47>AI'!-1'$  !-1'$  %7@67D1'$  ;@6;53F;@9@A?G>F;5A>>;@73D;FK;EEG7E 3EBD7E7@F76;@034>7 
0:7D79D7EE;A@7CG3F;A@;E'0'9D367   '!-9D367  !-  %7@67D %7@67D;E36G??KH3D;34>7I:7@ ;FD7BD7E7@FE87?3>7EFG67@FE;@8>G7@57 
3@6I:7@ ;FD7BD7E7@FE?3>7EFG67@FE $7?3>7EFG67@FEAGFB7D8AD??3>7EFG67@FE4K3@
3H7D397A8 BA;@FE;@'0'9D367E 034>7     !! 

!A@EF
'!-
!-%7@67D?3>7
#@FD3@575:3@@7>E!A78
 
 
 
 
 

/F67DD
 
 
 
 
 

0
 
 
 
 
 

B H3>G7    
 

1'$

 
 
 
 


   
+A@ !/EFG67@FEEFDG99>7;@'!-3@6'0'5AGDE7E5A?B3D76FAF:AE7E=;>>76;@!-6G7
FA G@83?;>;3D EK@F3J 3@6 ;@;F;3> >35= A8 >A9;53> F:;@=;@9 FD3;@;@9  $7?3>7E 7J57>  5:3>>7@9;@9EF7D7AFKB7E  '@6;H;6G3> 3BB>;53F;A@E K;7>6 47FF7D AGF5A?7E F:3@ FD3@E87DE AD AH7DE73E
D75DG;F?7@F  *G>F; EF3D -DA<75FE AGFB7D8AD? 7J3?E 3@6 AH7DE73E D75DG;F?7@F  E:AI53E;@9
6;H7DE77JB7D;7@57E .79D7EE;A@3@3>KE;E7JB>3;@EA8'0'9D367H3D;3@57'!-9D367 !-
9D367  3@6 97@67D ;@8>G7@57 '0' 9D367  .75A??7@63F;A@E 7@5A?B3EE 6;H7DE7 F735:;@9?7F:A6E  BDA?AF;@9 BD35F;53> BDA9D3??;@9  8AEF7D;@9 5A>>34AD3F;A@  3@6 7@5AGD39;@9
B3DF;5;B3F;A@ ;@ ;@8AD?3F;A@ 5A?B7F;F;A@E  '@F7D35F;H7 >73D@;@9 7@H;DA@?7@FE 3@6 BDAH;6;@9
EGBBADF8ADAH7DE73EEFG67@FE3D77EE7@F;3>FA7@:3@57F:7>73D@;@97JB7D;7@57 
+6/63916> C6,+2+;) >5>6'  0;+7/@9;509;-97:>=+=398+6=2385381,+</.98+
<B<=/7+=3-;/</+;-2;/?3/@  #*!&+(%#&&(%&$'+*!%  
383<=;B90.>-+=398   &&##)*+%*) ,*"%&+())(#**&'(&($$!%* !%!)*(-&+*!&%)(&$&*!&%&%!,()!*-(&($$!% !%(&($ ))%
()+#*)6/-=;983-8/@<:+:/;90 383<=;B90.>-+=398 
2==:< /:+:/; /.> =@ :;38= +<:A:;38=*=B:/=9:3-+6:;38=*<8 :;38=*8>7 (/81 <3+  (+81 )  &<3816/19;9,9=3-<38:;91;+76/+;8381 %*(%*!&%#&+(%#&%!!*#(%!% %&#&-    (;316/B<+8=/ -5+2  ;37:981   /8./;.300/;/8-/<38+-+./73-
:/;09;7+8-/90<=>./8=<<=>.B381$-3/8-/%/-289691B8138//;381+8. +=2/7+=3-<$% <>,4/-=<+==2/&83?/;<3=B902+8+ &!#!%)   

477

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

DLOT: An open-source application toassist human observers
Ashwin T Sa*, Danish Shafi SHAIKHa, & Ramkumar RAJENDRANaa
Indian Institute of Technology, Bombay*ashwindixit9@gmail.com
Abstract: Adaptive intelligent educational systems are gaining popularity, offering
personalized learning experiences to students based on their individual needs andstyles. One crucial feature of such systems is real-time personalized feedback.
However, identifying real-time learning processes impacting student performanceremains challenging due to data volume constraints. Current research often relies onlabor-intensive human observation, which is time-consuming and not scalable. To
efficiently collect real-time data, an observation tool is essential. Qualitative/Mixed
Method research explores participant experiences in education, social science, andhealthcare, utilizing methods like focus groups and observations. However, thesemethods can be labor-intensive, particularly in maintaining observation time intervals.
Existing tools lack comprehensive support for education-focused focus groups andobservations. To address these issues, this paper introduces the Data Logging andOrganizational Tool (DLOT), a flexible tool designed for qualitative studies with humanobservers. DLOT offers customizable time intervals, cross-platform compatibility, anddata saving and sharing options. The tool empowers observers to log timestamped
data and is available on GitHub. The DLOT was validated through two studies. The firststudy predicted students' affective states using real-time annotations collected via
DLOT, observing 30 students in each class. The second study created multimodal
datasets in a computer-enabled learning environment, observing 38 students
individually. A successful usability test was conducted, offering a potential solution tochallenges in real-time learning process identification and labor-intensive qualitative
research observation.
Keywords: Human Observation, Annotation Tool, Application, Data Logging,Computer Assisted Direct Observation, Minimal Attention User Interfaces

1. Introduction
Adaptive intelligent educational systems are becoming increasingly popular, particularly in
distance learning and online education. These systems are designed to provide personalized
learning experiences to students, catering to their individual learning needs and styles. Oneof the key features of such systems is the provision of personalized feedback to students
based on their learning processes. To be effective, this feedback needs to be provided in realtime as the student interacts with the learning environment (Mousavinasab et al., 2021, Ashwin& Guddeti, 2020, Taub et al. 2021).
Identifying the various learning processes that contribute to a student's real-time performanceis a challenging task. These processes include self-regulated learning, cognitive engagement,
metacognition, affective states, behavioral and interactional dynamics with peers,collaborative learning, uncertainty, feedback seeking, empathy, and group dynamics. To train
machine learning models to detect these processes in real-time, a large amount of annotateddata is necessary to identify, classify, or predict these constructs (Sinatra, Heddy, & Lombardi,
2015, Praharaj et al. 2021, Ashwin & Guddeti, 2019, Munshi et al. 2018).
Current research in this area involves human observation, where video recordings areanalyzed after the study has concluded. However, this method is time-consuming, laborintensive, and not scalable. Automated machine learning models also face challenges in realtime identification of these constructs due to insufficient data (Wu et al. 2022, TS & Guddeti,2020). Hence, there is a need for a reliable observation tool that can collect data quickly andefficiently in real-time. Such a tool could revolutionize the field of adaptive intelligent478

educational systems, providing personalized feedback to students based on their real-timelearning processes, leading to improved learning outcomes.
On the other hand, Qualitative/Mixed Method research is an approach to research that
focuses on exploring the experiences, attitudes, and perspectives of participants in a study. It
is often used in education, social science, and healthcare research, where understanding
human behavior is a key objective. The research involves collecting data through a variety ofmethods, such as questionnaires, interviews, focus groups, tests, observations, andsecondary data (Johnson, Burke, & Turner, 2003). While some methods like questionnaires
and tests, can be conducted without human observation, others, such as focus groups andobservations, require human observers to gather data. These methods are crucial in providing
insights into human behavior, emotions, and experiences. However, the process of humanobservation can be labor-intensive, especially when there are a large number of subjects or
the observation period is lengthy. In addition, maintaining the exact time interval between
observations can be challenging for human observers. An application or tool that automaticallysets the timer and provides a prompt for the observer to log the data would be incredibly usefulin such cases.
To facilitate the collection of data with specific labels, we have developed this app,drawing inspiration from the Human Affect Recording Tool (HART). Similar to various tools
commonly employed in qualitative and mixed-method research, such as Teamscope, Open
Data Kit (ODK), and KoboToolbox, which find utility in questionnaires, tests, and secondary
data collection, our app serves a comparable purpose. It's worth noting that while theseestablished tools are often used for activities like questionnaires, tests, and secondary datacollection, they may not be ideally suited for tasks like focus groups and observations.
Additionally, it's important to emphasize that not all these tools are available as free and opensource options. Furthermore, the adaptation of new technology might involve a learning curvefor both participants and researchers, potentially influencing the efficiency and accuracy of thedata collection process. In the context of educational research, the Human Affect Recording
Tool (HART) has been utilized for Quantitative Field Observations (QFOs), primarily focusing
on monitoring students' affect and behavior (Ocumpaugh et al. 2015). However, in responseto this limitation, our innovation aims to empower human observers by providing a mechanism
to log data along with timestamps.
Hence, this paper contributes to a DLOT (Data Logging and Organizational Tool)developed to aid qualitative and also quantitative studies involving human observers. DLOTis flexible and can be customized to set time intervals, works with different operating systems,
works for the individual, team (or group), different sets of codes/labels, and offers various
options for saving and sharing data.
Moreover, it's pertinent to emphasize that the central focus of this article is to present
a concise and comprehensive overview of the Data Logging and Organizational Tool (DLOT).
Rather than a typical research study, the article is tailored to provide an in-depth descriptionof DLOT's features and functionality. The rest of the paper is organized as follows: Section 2
provides details of DLOT, Section 3 discusses the validation, and Section 4 concludes thepaper with future directions.

2. DLOTThe DLOT tool was developed using Figma and React Native. Data export was handled using
the docx and xlsx packages. State management was implemented using useState hooks, anddynamic runtime changes were handled with the useEffect hook. The app development andtesting were done using Expo. Additional details about the DLOT tool development are
mentioned in Appendix A1. The complete details, including the readme information, can befound in the GitHub repository.
The DLOT tool offers a wide range of flexible options that cater to various researchrequirements:
Class Labels or Categories: Researchers using DLOT can create custom labels or
categories tailored to their study objectives. For example, Figure 1a illustrates class labels
1

https://github.com/danishsshaikh/DLOT/Appendix A

479

such as "engagement" to "on-task," which can be modified to suit different studies. For
instance, in a metacognition study, labels can be defined as "planning," "monitoring,""evaluating," and more, based on the specific research context. Prior to using the DLOT tool,
researchers need to establish a set of categories or class labels that will be used throughout
the study. This step involves defining the specific labels that will be assigned to observations.
Once set, these categories remain consistent and cannot be changed during the course of thestudy.
Time Flexibility: DLOT offers researchers the flexibility to set the observation timer
according to their study requirements. By default, the timer is set to 10 seconds, providing asuitable time frame for observations. The timer duration is conveniently displayed atop Figure
1a.
Selection of Categories: DLOT empowers researchers to design observationinterfaces that include both radio buttons and checklists, as demonstrated in Figure 1a. Thisfeature enables researchers to make multiple selections based on their study's specific needs,
enhancing the richness and versatility of data collection.

Figure 1. There are three screenshots attached that illustrate the data logging andorganizational tool. The first screenshot (Figure a) displays the interface where thestudent’s name, class labels, timer, and a log button can be found. Once the logging
process is completed, the data can be downloaded in either xlsx or docx format. After
selecting the desired format, the data can be stored locally on the mobile device or in
any other preferred location, as depicted in Figure b). In Figure c), a sample of the logged
data is displayed in a particular format. In this format, the student's name or ID acts as
the primary key, and the data is organized based on the timestamp.
Adjusting Observations (Number of Students and Group Size): DLOT allows
researchers to customize the number of students observed, accommodating studies involving
individuals, small teams, or larger groups like classrooms. Researchers can adapt the tool totheir specific research setting.
Platform Compatibility and Data Storage: DLOT is compatible with both Android and
iOS devices, ensuring broad accessibility for researchers using different operating systems.
Additionally, the tool supports multiple data storage formats, including text and xlsx. Thisversatility facilitates easier data analysis and sharing, streamlining the research workflow.

480File Sharing Options: Once data logging is complete, users can choose to store thefinal data locally or share it through various options such as Google Drive, WhatsApp,
Telegram, Bluetooth, LINE, etc. Figure 1b illustrates the convenient file sharing functionalityof DLOT.
Open-Source Availability: DLOT is available as an open-source application with its
GitHub code2 and official webpage guidelines and read-me information3. Sample screenshotsof DLOT are shown in Figure 1. The tool also includes a discussion forum on GitHub where
researchers can ask questions about using the app or modifying default values, among othertopics.
Training Observers for Data Annotation: To utilize the DLOT tool effectively, humanobservers need to undergo training in annotating or classifying data based on standard
definitions or study guidelines. For instance, in a basic emotion detection study, multiple
observers should observe the same student(s) and provide labels following establishedguidelines, such as a facial action coding system. Observers can choose various annotation
methods, including annotating together or independently and later discussing any
disagreements. Importantly, observers must adhere to the specific purpose of the observation.
For example, suppose the annotations aim to train a classification model like convolutional
neural networks. In that case, guidelines should emphasize labeling based on the specificinstance observed rather than relying on prior frames or external sources of knowledge.
The following steps should be followed to use the DLOT:
1. Train the human observers on the required class labels or categories and establish
inter-rater reliability.
2. Install the application.
3. Check if the required parameters are present in the application.
4. If yes, go to step 5; Else, modify the open-source code according to the requirements.
5. Set the parameters and collect data.
6. Save the data. (More details are in Appendix B4)
By embracing flexibility, customization, and compatibility, DLOT empowers
researchers to efficiently collect and analyze qualitative data in a manner that suits their
specific study objectives and research settings.

3. Validation
The Data Logging and Organizational Tool (DLOT) was validated using two studies. The firststudy focused on affective state prediction in a classroom setting, where real-time annotations
were collected using DLOT. A total of 30 students were observed in each class, and theiraffective states, including engaged, boredom, confusion, frustration, and neutral, werepredicted. The timer interval was set to 5 seconds as per the study requirement. The second
study incorporated DLOT as a component in generating multimodal datasets (interaction logs,
screen recordings, human observations and facial expressions), which involved 38
participants in a computer-enabled learning environment. This study involved observing asingle student at a time, with a time interval of 10 seconds throughout the study. Threeobservers used the tool in the first study, while five used it in the second. The validation
process encompassed an evaluation of system usability, conducted via the System Usability
Scale (SUS) analysis, yielding an average score of 93. SUS, a widely utilized questionnaire,
employs a 5-point Likert scale and subsequently transforms the scores to a range of 0 to 100(Brooke, 1995). The attained score of 93 reflects the collective assessment by participants
and underscores the commendable usability of the system. Observers reported that the tool
was handy and saved them significant time. The user interface was deemed easy to navigate,
and the installation process was straightforward.
DLOT was tested on different operating systems, including Android and iOS, andperformed well without technical glitches or issues. The open-source code of DLOT was made2

https://github.com/danishsshaikh/DLOT
https://danishsshaikh.github.io/DLOT/
4
https://github.com/danishsshaikh/DLOT/Appendix B
3481

available for observers. One observer from the study one used it and found it easy to modify
and make changes. He also mentioned the information provided in the open-source code wasclear and concise, and the accompanying discussion forum proved helpful in addressing
queries and concerns.
During the studies, DLOT was used continuously for extended periods while observing
30 students, and no technical issues such as hanging or data loss were experienced. Overall,the validation process and user feedback demonstrate the reliability, usability, andeffectiveness of the DLOT tool in diverse educational contexts.
Since there are not many manual note-taking tools in the education domain, it isrelevant to compare DLOT with HART, the only existing tool that is similar to DLOT in termsof observing students and noting down their states with a time stamp. The comparison andadvantages of DLOT over HART are as follows: Fixed Class Labels: HART has predefinedclass labels or categories based on Baker Rodrigo Ocumpaugh monitoring protocol (BROMP)(Ocumpaugh, 2015), limiting the customization options for researchers. Time Limitations:
HART imposes a fixed countdown clock of 3 minutes for each observation, which restricts thecoding time to 20 seconds for BROMP observers. Single Selection: HART only allows theselection of one class label or category per student within a time interval, using radio buttons.
Platform Compatibility: HART is designed exclusively for the Android platform, limiting its
accessibility to researchers. Data Storage Formats: HART stores data only in the txt format,
which may limit the options for data analysis and sharing. File Sharing Options: HART data
files are stored locally on Android devices and can be shared via email or transferred tocomputers using USB cables. As explained in the previous section, DLOT offers advantages
compared to HART in terms of flexibility in labeling, time management, selection options,platform compatibility, data storage formats, and file sharing options.
Limitations: In this study, there are several limitations of the Data Logging andOrganizational Tool (DLOT). While DLOT demonstrates potential for real-time qualitative
studies, a comparative usability analysis with existing tools was not conducted. Additionally,
the presence of observers using DLOT could introduce bias, demanding caution. Security andprivacy concerns inherent to digital platforms must also be considered. Furthermore, DLOT's
reliance on smartphone technology and internet access might restrict its use in certain
contexts.
In addition to its use in the education sector, this tool has the potential to be utilized in
multiple other domains. For instance, in research studies, researchers can use the app to log
data regarding participant behaviors, reactions to stimuli, or experiment results. Clinicalpsychologists can also use the app to record data concerning patient sessions, including
symptoms, behaviors, or progress over time. Moreover, coaches or athletes can utilize thetool to log data related to training sessions or competitions, such as times, distances, or
personal bests.

4. Conclusion
The Data Logging and Organizational Tool (DLOT) is a flexible and user-friendly application
developed for data collection and analysis in educational research. It offers researchers theability to create custom class labels or categories tailored to their study objectives, adjust
observation timers, and choose from multiple selection options for data collection. DLOT is
compatible with Android and iOS platforms, ensuring accessibility for researchers using
different operating systems. It supports various data storage formats, facilitating easy analysis
and sharing of data. The tool also provides convenient file sharing options, allowing users tostore data locally or share it through different platforms. Being an open-source application,
DLOT empowers researchers to modify and customize the tool according to their specificneeds. The GitHub repository provides comprehensive information and a discussion forum foruser support. Validation studies conducted on affective state prediction and multimodal
dataset creation have shown the reliability and usability of DLOT. Observers reported that thetool was highly useful and time-saving, with an intuitive interface and straightforward
installation process. DLOT performed well on different operating systems, ensuring its
effectiveness across platforms.

482

Several future directions for the DLOT include adding data visualization capabilities,
user authentication, cloud storage integration, customization in different sets of options,
integration with other educational apps, and audio and video capabilities for recording andanalyzing speech for unstructured data.

References
Ashwin, T. S., & Guddeti, R. M. R. (2019). Unobtrusive behavioral analysis of students in classroom
environment using non-verbal cues. IEEE Access, 7, 150693-150709.
Ashwin, T. S., & Guddeti, R. M. R. (2020). Impact of inquiry interventions on students in e-learning andclassroom environments using affective computing framework. User Modeling and User-AdaptedInteraction, 30(5), 759-801.
Brooke, John. (1995). SUS: A quick and dirty usability scale. Usability Eval. Ind.. 189.
Gale M. Sinatra, Benjamin C. Heddy & Doug Lombardi (2015) The Challenges of Defining and
Measuring Student Engagement in Science, Educational Psychologist, 50:1, 1-13, DOI:
10.1080/00461520.2014.1002924
Jaclyn Ocumpaugh, Ryan S. Baker, Ma Mercedes Rodrigo, Aatish Salvi, Martin van Velsen, Ani
Aghababyan, and Taylor Martin. 2015. HART: the human affect recording tool. In Proceedings ofthe 33rd Annual International Conference on the Design of Communication (SIGDOC '15).
Association for Computing Machinery, New York, NY, USA, Article 24, 1–6.
Johnson, Burke, and Lisa A. Turner. "Data collection strategies in mixed methods research." Handbookof mixed methods in social and behavioral research 10.2 (2003): 297-319.
Mousavinasab, E., Zarifsanaiey, N., R. Niakan Kalhori, S., Rakhshan, M., Keikha, L., & Ghazi Saeedi,
M. (2021). Intelligent tutoring systems: a systematic review of characteristics, applications, andevaluation methods. Interactive Learning Environments, 29(1), 142-163.
Munshi, A., Rajendran, R., Ocumpaugh, J., Biswas, G., Baker, R. S., & Paquette, L. (2018, July).
Modeling learners' cognitive and affective states to scaffold SRL in open-ended learning
environments. In Proceedings of the 26th conference on user modeling, adaptation andpersonalization (pp. 131-138).
Ocumpaugh, J. (2015). Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical andtraining manual. New York, NY and Manila, Philippines: Teachers College, Columbia University
and Ateneo Laboratory for the Learning Sciences, 60.
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-locatedcollaboration modeling using multimodal learning analytics—can we go the whole nine yards?.
IEEE Transactions on Learning Technologies, 14(3), 367-385.
Taub, M., Azevedo, R., Rajendran, R., Cloude, E. B., Biswas, G., & Price, M. J. (2021). How are
students’ emotions related to the accuracy of cognitive and metacognitive processes during
learning with an intelligent tutoring system?. Learning and Instruction, 72, 101200.
TS, A., & Guddeti, R. M. R. (2020). Automatic detection of students’ affective states in classroom
environment using hybrid convolutional neural networks. Education and information technologies,
25(2), 1387-1415.
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022). A survey of human-in-the-loop for machinelearning. Future Generation Computer Systems,135(1), 364-381.

483

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Supporting Learning Through AffordanceBased Design: A Comparative Analysis of"BioVARse" and a Standard Textbook
Companion Application in Biology
Education
Devanshu SAINDANEa, Sunny Prakash PRAJAPATIa & Syaamantak DASa*a
IDP in Educational Technology, Indian Institute of Technology Bombay, India*syaamantak.das@iitb.ac.in
Abstract: Textbooks as an educational artefact has evolved immensely withtechnology, especially through mobile applications. When used in conjunction with
textbooks, mobile applications have the potential to provide a unique set of affordances
to support learning. This paper presents the design of “BioVARse”, a textbook
companion app for Biology subjects that provides these affordances to the learners. A
comparative analysis of these learning affordances with a standard textbook
companion application is presented, and a pilot study for assessing the usability is also
performed. The preliminary results indicate a significantly higher usability of BioVARse,when compared to the standard application. Additionally, the paper also discusses thepossibility of improving the learning experience through an affordance-based design ofa textbook companion app.
Keywords: Multimedia Educational Content, Textbook companion application,
Augmented Reality, Affordance-based Design, Usability

1. Introduction and BackgroundTextbooks are the primary learning materials that are used for studying a subject by learners.
Appropriate technological enrichment can provide access to static and dynamic content, suchas in Phygital textbooks (Prajapati & Das, 2023), which can impact learning outcomes of anindividual, such as their reasoning behaviour (Karnam et al., 2020). Currently, textbooks areenriched with dynamic content through the quick response (QR) codes or through Augmented
Reality (AR) markers on the printed textbook page (Danaei et al., 2020; Ghaem Sigarchian etal., 2018). This content is then accessed through digital devices, such as mobile phones,
which can be termed as the textbook companion applications.
In Science, topics in Biology (and similar subjects) require visualisations for learning ofvarious concepts, as inappropriate visualisations may result in development of misconceptions(Yusof et al., 2020). AR can be used in this situation to effectively facilitate immersive
visualisations using dynamic content, and it can be provided to learners through a mobile
application that can be used in different learning scenarios (Susilo et al., 2021). Companion
apps that are available to the learners primarily focus on the content accessibility, failing toaddress the issue of their integration into learning activities.
Consideration of affordances that are crucial in learning can be an important aspect thatshould be considered for such applications (Hammond, 2010). In human-computer interaction,"affordance" refers to the perceived or actual action possibilities an object offers a user.
Though lacking a universal definition, this work defines it as the set of interactions facilitatedby a tool that supports learning. This study details the design of a textbook companion app(BioVARse), based on affordance analysis and comparison with a standard app, andelucidates a usability study of these application.

484

2. Design Space for BioVARse
A markerless AR app, 'BioVARse,' was developed using Unity 3D, for 10th and 11thgrade students studying Biology. It covers five NCERT curriculum topics, with textbook
diagrams coded as markers for 3D models. These topics were: “Body fluids and circulation”,“Breathing and exchange of gases”, “Digestion and absorption”, “Excretory products and theirelimination”, and “Structure of human eye”. The name BioVARse combines concepts ofmetaverse, virtual reality, and augmented reality. The system architecture, detailed in Figure
1, elucidates the integration of AR, Quiz, and App modules. The prototype is accessible on
Github 1 , and the topics were selected from federal textbooks based on open-source AR
content availability, and resources availability on DIKSHA . The key affordances of BioVARse
applications specifically designed to support learning of the topic are: Note-taking, Voice-overnarration of the text, Access to 3D AR content, Formative assessment for the topic, Access tostatic content, and Sharing of information. The affordances due to mobile devices were samefor both applications, i.e, DIKSHA and BioVARse, which are the ability to interact with learning
content through touch, scroll, text input, etc. (Berthelsen & Tannert, 2020). The affordances(listed above) were facilitated to the user through three primary components of the app, which
are: a) Learn (for static and dynamic content accessibility), b) Explore (for engagement with
content), and c) Test (for formative assessments and feedbacks). These affordances areelaborated in Figure 2 below, which were used as a comparison metric for both theapplications.

Figure 1. System architecture of BioVARse application.

3. Study Design and Results
The usability of the BioVARse app was assessed using the System Usability Scale (SUS),
chosen for its reliability and ease of use. Eleven participants (chosen through convenient
sampling) interacted with the app on the topic 'Human Heart (Body fluids and circulation),' andtheir SUS survey responses were recorded online. Instructions were provided, and users wereencouraged to engage with AR models, multimedia content, and assessments. The SUS
survey was administered for both the BioVARse and NCERT DIKSHA apps, and usability
scores were compared using the Mann-Whitney U test, revealing a significant difference (U =13.5, p < 0.005). The results indicate acceptable usability for BioVARse and poor usability forthe standard DIKSHA application.

1

BioVARse: https://github.com/SPARTA-Research-Group-ET-IITB/BioVARse

485

Figure 2: Key learning affordances and how they are facilitated in the two mobile
applications

References
Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles,
Policy & Practice, 18(1), 5–25. https://doi.org/10.1080/0969594X.2010.513678
Berthelsen, U. D., & Tannert, M. (2020). Utilizing the affordances of digital learning materials. L1
Educational Studies in Language and Literature, 20, Running Issue(Running Issue), 1–23.
https://doi.org/10.17239/L1ESLL-2020.20.02.03
Chen, P.-H. (2021). In-class and after-class lecture note-taking strategies. Active Learning in Higher
Education, 22(3), 245–260. https://doi.org/10.1177/1469787419893490
Danaei, D., Jamali, H. R., Mansourian, Y., & Rastegarpour, H. (2020). Comparing reading
comprehension between children reading augmented reality and print storybooks. Computers &Education, 153, 103900. https://doi.org/10.1016/j.compedu.2020.103900
Ghaem Sigarchian, H., Logghe, S., Verborgh, R., De Neve, W., Salliau, F., Mannens, E., Van deWalle, R., & Schuurman, D. (2018). Hybrid e-TextBooks as comprehensive interactive learning
environments. Interactive Learning Environments, 26(4), 486–505.
Hammond, M. (2010). What is an affordance and can it help us understand the use of ICT in
education? Education and Information Technologies, 15(3), 205–217.
https://doi.org/10.1007/s10639-009-9106-z
Karnam, D., Mashood, K., & Sule, A. (2020). Do student difficulties with vectors emerge partly fromthe limitations of static textbook media? European Journal of Physics, 41(3), 035703.
Norman, D. A. (1999). Affordance, conventions, and design. Interactions, 6(3), 38–43.
Prajapati, S. P., & Das, S. (2023). Designing a Phygital Interface for Textbooks to Support Active
Collaborative Learning. Companion Proceedings of the 28th International Conference onIntelligent User Interfaces, 1–4. https://doi.org/10.1145/3581754.3584117
Pyörälä, E., Mäenpää, S., Heinonen, L., Folger, D., Masalin, T., & Hervonen, H. (2019). The art ofnote taking with mobile devices in medical education. BMC Medical Education, 19(1), 96.
https://doi.org/10.1186/s12909-019-1529-7
Susilo, A., Hardyanto, W., Martuti, N. K. T., & Purwinarko, A. (2021). Mobile learning developmentusing augmented reality as a biology learning media. Journal of Physics: Conference Series,
1918(4), 042013. https://doi.org/10.1088/1742-6596/1918/4/042013
Swamy K L, N., Chavan, P. S., & Murthy, S. (2018). StereoChem: Augmented Reality 3D Molecular
Model Visualization App for Teaching and Learning Stereochemistry. 2018 IEEE 18th
International Conference on Advanced Learning Technologies (ICALT), 252–256.
https://doi.org/10.1109/ICALT.2018.00065
Yusof, C. S., Amri, N. L. S., & Ismail, A. W. (2020). Bio-WTiP: Biology lesson in handheld augmented
reality application using tangible interaction. IOP Conference Series: Materials Science andEngineering, 979(1), 012002. https://doi.org/10.1088/1757-899X/979/1/012002

486

" !   !" " "   #" !
#" !"$  #" ! #"        
    
      
!  
!  !  
53-C-859A -5@ 7EA?4AA -/ 6<  (:12=>?.:16:4>?@12:?>0;16:4/25.A6;=>6>0=@06.83;=<=;A616:4?.=42?21
>@<<;=? 6: <=;4=.996:4 21@0.?6;:  @?;9.?60 .:.8D>6> ;3 0;16:4 /25.A6;=> @>6:4
9.056:2>0.:.11=2>>?528696?.?6;:>;39.:@.89;:6?;=6:4 $=2A6;@>>?@162>3;0@>21;:0;16:4/25.A6;==2<=2>2:?.?6;:>B6?5;@?0;:>612=6:416332=2:02>=28.?6A2?;.9;128
.:>B2= *2<=;<;>229/2116:4A20?;=>?5.?0.<?@=2?52>216332=2:02>2:./86:4?52
16>?6:0?6;:/2?B22:>69<82.:10;9<82C0;12>;8@?6;:> A.8@.?6:4?52>2A20?;=>/D<=2160?6:4 .>>64:92:? >0;=2> B2 .0562A21 ;A2=  56452= .00@=.0D 0;9<.=21 ?;
0;:A2:?6;:.8 92?5;1>  '56> .<<=;.05 5.> ?52 <;?2:?6.8 ?; 2:5.:02 ?2.052=>
@:12=>?.:16:4 ;3 >?@12:?> 0;16:4 /25.A6;=> .:1 69<=;A2 >@<<;=? 6: <=;4=.996:4
21@0.?6;:   $=;4=.996:4 21@0.?6;: 21@0.?6;:.8 1.?. .:.8D>6> ;12%'
<=;4=.996:4<2=3;=9.:02<=2160?6;:  ! 
*:01>?@-:05:3?@A01:@/;05:3.14-B5;>?5:<>;3>-995:3/;A>?1?;221>??1B1>-8-0B-:@-31?
5:/8A05:3 501:@52E5:3 ?@>A3385:3 ?@A01:@? -:0 <5:<;5:@5:3 ->1-? ;2 05225/A8@E  )45? 1:-.81?
@1-/41>? @; <>;B501 @->31@10 ?A<<;>@ -:0 1221/@5B1 2110.-/7 1:4-:/5:3 ?@A01:@?
A:01>?@-:05:3 ;2 <>;3>-995:3 "15:;:1: 1@ -8   ;C1B1> 05>1/@8E 9;:5@;>5:3 -88?@A01:@?<>;3>-995:3-/@5B5@51?5?:;@21-?5.812;>@1-/41>?0A1@;8->31/8-???5F1?-:08595@10
>1?;A>/1?  )41>12;>1 @41 A?1 ;2 9-/45:1.-?10 -A@;9-@5/ -:-8E?5? ;2 ?@A01:@ /;05:3
.14-B5;>?.1/;91?1??1:@5-8 &>1B5;A??@A051?4-B1/;:0A/@10@41-A@;9-@5;:;2<>;3>-995:38;3-:-8E?5?#-;1@
-8 
 ;C1B1>9-:E;2@4190;:;@/;:?501>@4105221>1:/1.1@C11:?@A01:@/;01?-:0- 9;018 -:?C1> C41: /-8/A8-@5:3 /;05:3 .14-B5;> >1<>1?1:@-@5;:?  E /;:?501>5:3 @41?1
05221>1:/1?5@.1/;91?21-?5.81@;05?@5:3A5?4.1@C11:?@A01:@?C4;4-B1/;>>1/@8E?;8B10-
<>;.819 C5@4 ?59<81> /;01 /8;?1> @; @41 9;018 -:?C1> -:0 @4;?1 C4; 4-B1 C>5@@1: 9;>1
/;9<81D/;01 );-00>1??@45?C1<>;<;?1@4131:1>-@5;:;2/;05:3.14-B5;>>1<>1?1:@-@5;:?
@4-@ /;:?501> @41 01B5-@5;: 2>;9 @41 9;018 -:?C1>  %A> ;.61/@5B1 5? @; A@585F1 @41?1
>1<>1?1:@-@5;:? 01>5B10 2>;9 @41 /;05:3 -/@5B5@E 8;3? ;2 ?@A01:@? 0A>5:3 <>;3>-995:3
-??53:91:@? @; <>105/@ @415> <1>2;>9-:/1 -//A>-@18E  E -/451B5:3 -//A>-@1 <>105/@5;:? ;2
-??53:91:@ ?/;>1? .-?10 ;: /;05:3 .14-B5;> >1<>1?1:@-@5;:? C1 019;:?@>-@1 @4-@ @41?1
>1<>1?1:@-@5;:?1221/@5B18E/-<@A>12-/@;>?@4-@-221/@?@415><>;3>-995:3<1>2;>9-:/1?    );/>1-@119.1005:3B1/@;>?@4-@/-<@A>1?@A01:@?/;05:3.14-B5;>?5:/8A05:3@415>-.585@E@;
C>5@1 /;01 ?A//1??2A88E -:0 ?9;;@48E -? C188 -? @415> @1:01:/E @; 1D<8;>1 05221>1:@-<<>;-/41?@4>;A34@>5-8-:01>>;>C1<>;<;?1?;9119.1005:391@4;0? 487

5>?@C1A@585F1@41<>1@>-5:10;01')9;0181:31@-8 
C5@4;A@25:1@A:5:3
@;/;:B1>@?@A01:@?/;015:@;19.1005:3? ;01')5?-<>1@>-5:10.59;0-89;0182;>&"&
C5@4)>-:?2;>91>.-?10:1A>-8->/45@1/@A>1 $1D@@;/;9<A@1-:19.1005:3B1/@;>2;>1-/4
?@A01:@? /;05:3 .14-B5;> C1 A@585F1 @41 -:?C1> /;01 <>;B5010 2;> - 35B1: <>;3>-995:3
-??53:91:@ E/;:?501>5:3@41-:?C1>/;01-?->121>1:/1C1=A-:@52E@41/4->-/@1>5?@5/?
;2-?@A01:@?/;05:3.14-B5;>>18-@5B1@;@45?>121>1:/1/;01 );-//;9<85?4@45?C15:@>;0A/1
@41/;:/1<@;2-:?C1>05>1/@10B1/@;>?+?C45/4->1B1/@;>?<;5:@5:32>;91-/4/;01
C>5@@1:.E-?@A01:@@;@41-:?C1>/;01 
2@1>/-8/A8-@5:3@41+2;>1-/4?@A01:@?/;01C131:1>-@1@C;B1/@;>?@4191-:
;2@41+?+#1-:-:0@41?A9;2@41+?+(A9 ,11D<1/@@4-@?@>A3385:3
?@A01:@?C5884-B1-8->31>+(A90A1@;@415>@>5-8-:01>>;>-<<>;-/4-:0?53:525/-:@/;01
05221>1:/1? )41+(A9/-:.1/-8/A8-@108571.18;C    ,41>1-?  5:05/-@1?@4119.1005:3B1/@;>;2@41-:?C1>/;01-:0 5:05/-@1?
@4119.1005:3B1/@;>;2@41@4/;01C>5@@1:.E-?@A01:@@;?;8B1@41-??53:91:@ )41>12;>1
C1/-:/-8/A8-@1@41+#1-:.E05B505:3+(A9.E :@45??@A0EC11B-8A-@1@41+#1-:-:0+(A9-?>1<>1?1:@-@5B1B1/@;>?2;>
?@A01:@ /;05:3 .14-B5;>? 1D-95:5:3 @415> 1221/@5B1:1?? 5: /-<@A>5:3 -:0 >1<>1?1:@5:3 @41:A-:/1?;2?@A01:@?-<<>;-/41?@;<>;3>-995:3-??53:91:@? 





#      ); -??1?? @41 1221/@5B1:1?? ;2 @41 31:1>-@10 B1/@;>? 5: >1<>1?1:@5:3 ?@A01:@? /;05:3
.14-B5;>?C1/;:0A/@10-:1D<1>591:@@;<>105/@?@A01:@?<>;3>-995:3-??53:91:@?/;>1? ,1A@585F10,+")-:53A/451@-8 
-:;:85:1/;05:3-<<85/-@5;:A?105:<>;3>-995:3
/;A>?1? -@ ;A> A:5B1>?5@E @; -:-8EF1 @41 <>;3>-995:3 8;3? /;881/@10 0A>5:3 - &E@4;:
<>;3>-995:3 1D1>/5?1 /;A>?1 5:
 )4>;A34;A@ @41 /;A>?1 ?@A01:@? >1/15B10 C1178E
<>;3>-995:3-??53:91:@?@;.1/;9<81@10C5@45:-C117 ,1 /-@13;>5F10 ?@A01:@? 5:@; @C; 3>;A<? .-?10 ;: @415> ?/;>1? ;9<  ?@A01:@?
C4; ;.@-5:10 2A88 ?/;>1? -:0 :/;9<  ?@A01:@? C4; 050 :;@  )41: C1 @1?@10 @41<>105/@-.585@E ;2 @41 B1/@;>? .E <>105/@5:3 C45/4 ?@A01:@? C1>1 5: :/;9<  A?5:3 9-/45:1
81->:5:3 ;>.5:->E/8-??525/-@5;:C119<8;E10-'-:0;9;>1?@C5@4-9-D59A901<@4;2 ,1@41:<1>2;>910-2;80/>;??B-850-@5;:C5@4 -:0/-8/A8-@10@41-B1>-31?/;>1?;2
@412;A>@E<1?;2<>105/@5;:<1>2;>9-:/191@>5/?>1/-88<>1/5?5;:-//A>-/E-:0291-?A>1 : @45? 1D<1>591:@ C1 1D-95:10 @41 /;:/-@ -??53:91:@ 35B1: 0A>5:3 @41 950081
?@-31?;2@41/;A>?1 )41:A9.1>;2<>;3>-995:38;3?C-?
9;:3@41 ?@A01:@?
C4;1:3-310C5@4@41/;:/-@-??53:91:@ ?@A01:@?.18;:310@;@41:/;9< 3>;A<C4581
@41>19-5:5:3 ?@A01:@?.18;:310@;@41;9< 3>;A<    )-.81 ?4;C?@41<>105/@5;:>1?A8@?;21-/4B1/@;>5:-<>;3>-995:3-??53:91:@35B1:5:@41
950081?@-31;2@41/;A>?1 ;>/;9<->5?;:C1A?10105@05?@-:/1-:019.1005:305?@-:/1
@;/-8/A8-@14;C@41?@A01:@/;01C-?05221>1:@2>;9@41-:?C1>/;01 5>?@C1/-:?11@4-@@41B1/@;>31:1>-@10.E?A995:3A<+1D45.5@?@4145341?@?/;>1?5:-88@41<1>2;>9-:/1
91@>5/? (1/;:0C1/-:?11@4-@@41?A991@4;0;A@<1>2;>9?@4191-:91@4;05:<>105/@5;:
?/;>1?5:-89;?@-88@4191@>5/? 488)-.81                    
$#$  #$ 





 
%
 
%
 
%

"#!           

      


%"&         #%"
         !

";;75:3-@@41)-.81 C1/-:?11@4-@@41?A991@4;0/;:?5?@1:@8E;A@<1>2;>910@4191-:
91@4;0 -/>;?? -89;?@ -88 91@4;0?  ;:?501>5:3 @41 >1-?;: C4E @45? 4-<<1:10 - <;??5.81
1D<8-:-@5;:5?@4-@?@A01:@?C4;->181??<>;25/51:@5:<>;3>-995:3-:0?@>A3381C5@4/;05:3
9-E >1=A5>1 >1<1-@10 @>5-8 -:0 1>>;> @; ->>5B1 -@ - ?;8A@5;:  )45? 5:/>1-?1 5: @41 :A9.1> ;2
-@@19<@?>1?A8@?5:-3>1-@1>?A9 2C1/;:?501>05?@-:/1;>B1/@;>?5F1-?<1:-8@51?C1/-:
-//A9A8-@1 @419 .E 19<8;E5:3 @41 ?A9 91@4;0 1:-.85:3 @41 501:@525/-@5;: ;2 ?@A01:@?
?@>A3385:3 C5@4 /;05:3  :0110 .;@4 05@ 5?@-:/1 (A9 -:0 9.1005:3 5?@-:/1 (A9 ->1<>;<;>@5;:-8 @; @41 :A9.1> ;2 -@@19<@? .A@ @415> ?/;>1? 050 :;@ 5:/>1-?1 -? 9A/4 -? +(A9 )45?5:05/-@1?@4-@@4105>1/@5;:;2@41+-?C188-?5@??5F1/;:@>5.A@1?@;<>105/@5;: :?A99->E;A>25:05:3??A331?@@4-@@41+(A9>1<>1?1:@?2-/@;>?@4-@?53:525/-:@8E-221/@
?@A01:@<1>2;>9-:/1 





!


)45?<-<1><>1?1:@1091@4;0?A?5:3;01')@;/;:B1>@?@A01:@?/;05:3.14-B5;>?5:@;
0591:?5;:-8B1/@;>? )41+(A991@4;05:/;><;>-@5:305221>1:/1?.1@C11:?@A01:@-:0 -:?C1> /;01  ;01').-?10 >1<>1?1:@-@5;:? ;A@<1>2;>910 105@ 05?@-:/1 <>;B505:3
B-8A-.81 5:?534@? 2;> 10A/-@;>? @; A:01>?@-:0 -:0 ?A<<;>@ ?@A01:@? 81->:5:3 :110? 5:
<>;3>-995:3 10A/-@5;:  :4-:/10 2110.-/7 .-?10 ;: @41?1 5:?534@? /-: 59<>;B1 ?@A01:@
81->:5:3;A@/;91? " )45?C;>7C-??A<<;>@10.E ()'()>-:@$A9.1> &# '$A9.1> &    & !  -:0 &    -<-:   (&(!!$>-:@
2:4- @; '.:4 @.:" 2:4+ ;:4!    ;?52=> ;12%'<=2?=.6:21
9;1283;=<=;4=.996:4.:1:.?@=.88.:4@.42> .=+6A<=2<=6:?.=+6A  26:;:2:     2D>?=;72 1.?. 6: <=;4=.996:4 0;@=>2> ;0?;=.8 16>>2=?.?6;: (:6A2=>6?D ;3
28>6:76 %2?=62A213=;95??< 518 5.:182 :2?
   !.;, &56, !.=B.:& $=602' * .=:2>' 56!  :;B6:4B52:.:1B52=2
'29<;=.8&'"" 3;= >?@12:? 82.=:6:4 <=;4=2>>6;: 6: :;A602 <=;4=.996:4 ?.>7>  :?2=:.?6;:.81@0.?6;:.8.?.!6:6:4&;062?D '.:64@056, !6:29.?>@' #7@/; &569.1.  )6>@.86E6:4>;@=020;122A;8@?6;:3;=
@:12=>?.:16:408.>>B612<=;4=.996:4<=;02>>2> &@>?.6:./686?D    489

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31stInternational Conference on Computers in
Education.Asia-Pacific Society for Computers in Education

Unveiling Learners’ Interaction Behavior in
Virtual Reality Learning Environment
Antony PRAKASH*, Ramkumar RAJENDRANInterdisciplinary Programme in Educational Technology, Indian Institute of Technology
Bombay, India*antony80004@gmail.com
Abstract: VR is increasingly being utilized in various domains, including education, due
to its unique characteristics. Research in this area often relies on physiological sensors,
eye-trackers, VR device orientation, human observers, and pre-test and post-test tocollect data for quantitative studies and on questionnaires, surveys, and interviews tocollect data for qualitative studies in VR Learning. However, there is a dearth of reliabledata sources for studying learner behavior in VRLE, and minimal efforts have been
made to automatically collect behavioral data in this context. Furthermore, there is alack of studies that investigate learning processes through the lens of learners' dynamic
interaction behavior in VRLE. To address these gaps, we have developed a real-timedata collection mechanism that automatically logs learners' interaction behavior in
VRLE, including timestamps. This mechanism was deployed in a room-scale VRLE
called MaroonVR, and a study was conducted involving undergraduate engineering
students. The main objectives of this paper are to identify differences in interaction
behavior between high and low performers and to develop an optimal predictor model topredict the learning outcome using learners' interaction behavior in VRLE. Furthermore,
we propose that the study's findings can be utilized to model learners' behavior in VR
and to provide scaffolding and adaptive personalized VR learning content.
Keywords: Interaction Behavior, VR Learning Environment, Predictor Model, Adaptive
Personalized Learning Content

1. Introduction
Virtual Reality (VR) immerses users in a simulated environment, allowing interaction with
virtual objects and creating a realistic experience (Wade, Zhang, Bian, Fan, Swanson,
Weitlauf,& Sarkar, 2016). Its distinctive qualities of immersion, interaction, and imagination
have led to its application in various domains, including automotive, military, healthcare,sports, and education. In education, VR is utilized to teach concepts that are not visible to thenaked eye, such as DNA strands (Sharma, Jin, Prabhakaran,& Gans, 2018) and the human
circulatory system (Pathan, Rajendran,& Murthy, 2020). It also enables the exploration of
inaccessible places like outer space and ancient civilizations. Additionally, VR provides a safe
environment for experiencing hazardous scenarios, such as firefighting, welding, and oil
refinery operations, which would be dangerous in real life. VR technology has transformed
education by offering immersive and engaging learning experiences that enhance
understanding and provide practical training opportunities in a wide range of subjects.
The qualities of VR have led to a significant increase in research examining its use ineducation. Most of the studies were conducted to measure the impact of VR on learning bycollecting data from pre-test and post-test. Moreover, the educational researchers used thedata collected from self-reported questionnaires, interviews, and surveys (Radianti,
Majchrzak, Fromm, & Wohlgenannt, 2020) to measure the user experience, engagement, andusability of the VR systems and to compare VR-aided and VR-non-aided learning systems(Albus, Vogt,& Seufert, 2021). The researchers also used the data collected from the devices
such as 1) physiological sensors to assess the affective state of the learners while performing
the tasks (Feng, González, Amor, Lovreglio,& Cabrera-Guerrero, 2018), 2) eye trackers toassess the learners' intended area of interest (Wade, Zhang, Bian, Fan, Swanson, Weitlauf,&490

Sarkar, 2016) 3) body trackers to adapt the size of the virtual objects with respect to the size ofthe users, and 4) orientation of the head-mounted displays (HMD) and handheld controllers(HHC) to assess the response time. In addition to the physical devices, human observers werealso used to collect data in VRLEs related to the affective state (Feng, González, Amor,
Lovreglio,& Cabrera-Guerrero, 2018) and the procedural performance (Santamaría-Bonfil,
Ibáñez,Pérez-Ramírez, Arroyo-Figueroa, & Martínez-Álvarez, 2020) of the learners. However,the data provided by human observers are biased due to cognitive, social, and communicativecauses. Moreover, the data provided by human observers also need to satisfy inter-rater
reliability tests in order to become valid (Olmos-Raya, Ferreira-Cavalcanti, Contero,
Castellanos, Giglioli, & Alcañiz, 2018).
All the existing studies reported that using VR technology in the education domain has
resulted in a) better learning compared to other learning environments like simulation orComputer Based Learning Environments (CBLE) for procedure learning content(Santamaría-Bonfil, Ibáñez,Pérez-Ramírez, Arroyo-Figueroa, & Martínez-Álvarez, 2020), b)
positive impacts such as improvement in learning gain, experience closer to reality, intrinsic
motivation, level of interest, skills, and memory retention (Chavez,& Bayona, 2018). Although
the existing studies measured the learning gain, the knowledge of how learners interact with
VRLE and how they learn from the VR environment is not explored. That is, the existingstudies have analyzed the impact of VR intervention on the learning outcomes, but the impact
of interaction behavior in VRLE on the learning outcomes is still in its infancy. Hence, theinteraction behavior of the learners leading to variation in the performance for different
learners is not known. This is mainly due to the non-existence of a data collection mechanismthat can log the learners' interaction behavior in VRLE. To address this gap, we developed amechanism that is able to log all the interaction behavior of the learners in VRLE in real timealong with the time stamp (Prakash,& Rajendran, 2022).
The interaction behavioral data collection mechanism we developed was deployed in
MaroonVR (Pirker, Holly, Lesjak, Kopf, & Gütl, 2019), a virtual reality learning environment(VRLE) utilized for teaching the physics concept of electromagnetic induction. A study was
conducted involving 14 undergraduate students from non-electrical engineering backgrounds,
and their interaction behavioral data (IBD) was logged. The participants' interaction with the
VRLE resulted in a positive learning gain (Prakash, Shaikh, & Rajendran, 2023). In this paper,
we present the extraction of features, such as frequency and duration of action events, toevaluate their impact on the learning outcomes. Furthermore, we discuss the development ofan optimal regression predictor model using the features extracted from the data logged in the
IBD logger to predict the learning outcome. The predictor model thus developed can be used
to early detect the performance of the learners. The knowledge of the early detection of thelearners’ performance can be used by the designers to design VR learning content that is able
to provide the required scaffolding in the form of hints, and feedback to the learners in order tomaximize the learning outcome.
The paper is structured as follows. Section 2 discusses the impacts, the learning
outcomes, and data collected in the education domain using VR. The research questions
addressed by this paper are also presented in section 2. The research methodology is briefed
in Section 3 along with various analyses. The results of the analyses are presented in Section
4. The inferences made from the results are discussed in Section 5 along with the conclusion
describing the limitations and the guidelines for future work.

2. Literature Review and BackgroundIn this section, we first describe the works related to the impacts of VR on learning. Then we
discuss the learning outcomes and the data collected to measure VR impacts and the learning
outcomes in the existing studies. We also give a brief overview of the IBD logger and the datalogged in it.

491

2.1 Impacts of VR on Learning
According to Radianti, Majchrzak, Fromm, and Wohlgenannt, (2020), the results obtained
from employing immersive VR technology in various educational domains indicate an increasein engagement, time dedicated to learning tasks, and the development of cognitive,
psychomotor, and affective skills. VR's ability to offer interaction, immersion (Hamilton,
McKechnie, Edgerton, Wilson, 2021), and a first-person perspective (Mikropoulos, & Bellou,
2010) play a significant role in enhancing learning outcomes by providing realistic
experiences, fostering intrinsic motivation, and increasing interest in learning. Chavez, andBayona, (2018) asserted that no literature reported negative effects of using VR in the learning
process. Nevertheless, the comprehensive literature review conducted by Hamilton,
McKechnie, Edgerton, and Wilson in 2021 revealed a reduction in learning improvement with
VR-based learning when contrasted with both desktop learning and conventional classroom
learning. Despite the negative learning outcome observed in some studies, potentiallyattributed to the use of low-end mobile VR tools and the inclusion of factual learning content in
VRLE, learners have demonstrated increased motivation and interest in learning whenutilizing VR as opposed to traditional approaches and computer-based learning (Makransky,
Terkildsen, & Mayer, 2019). The existing studies suggest that VR has a positive impact onlearning, particularly in VRLE involving procedural learning content. However, there is a lackof literature that examines the impact of VR learning in relation to the dynamic behavior of thelearners in the VRLE.

2.2 Learning Outcomes Measured in VR Learning Environment
Researchers have examined the effects of VR on learning by evaluating various learning
outcomes, including cognitive, procedural, and affective skills (Hamilton, McKechnie,
Edgerton, Wilson, 2021). Cognitive skills pertain to acquiring declarative knowledge,
procedural skills involve psychomotor abilities, and affective skills are related to emotions andattitudes (Hamilton, McKechnie, Edgerton, Wilson, 2021). Among these skills, cognitive skills
have received the most attention in VR studies. The evaluation of cognitive skills typicallyinvolves assessing knowledge acquisition, retention, and transfer. In existing studies
conducted within VR learning environments (VRLE), the assessment of knowledge acquisition
is typically done through pre-tests and post-tests, while knowledge retention is evaluatedthrough delayed post-tests. Procedural skills, on the other hand, are assessed by measuringtask completion time and the sequential order of accessing intermediate steps to accomplish atask (Feng, González, Amor, Lovreglio,& Cabrera-Guerrero, 2018). Affective skills areevaluated using questionnaires and physiological devices such as electrodermal activity
sensors, photoplethysmography sensors, and multichannel physiological sensors (Radianti,
Majchrzak, Fromm, & Wohlgenannt, 2020; Feng, González, Amor, Lovreglio,&
Cabrera-Guerrero, 2018).

2.3 Data Collection in VR Learning Environment
In order to assess the impact of VR on learning and learning outcomes, data is collected
through various methods such as body tracking, performance metrics, physiologicalresponses, questionnaires, and interviews. Body tracking involves analyzing the shape andsize of the user's body, and the VR system responds accordingly by adapting the VR
environment and the size of objects within it (Olade,Fleming,& Liang, 2020). This may also
include VR avatars imitating the body postures and gestures of the users. Performance metricdata includes scores from pre-tests, post-tests, and delayed post-tests, as well as data related
to task completion time and the number of attempts taken. Physiological sensors are used tocollect data on the user's affective state, such as measuring skin conductance levels toevaluate fear and anxiety, heart rate to assess stress, and blood volume pulse amplitude togauge sympathetic arousal (Feng, González, Amor, Lovreglio,& Cabrera-Guerrero, 2018).

492

Questionnaires and interviews provide data for quantitative analysis, addressing researchquestions related to VR usability, user experience, comparisons between VR-aided andnon-VR-aided learning, and technology solutions (Hamilton, McKechnie, Edgerton, Wilson,2021). However, despite the availability of various instruments to collect multimodal data forassessing learning outcomes and system usability, the literature analysis reveals neglect ofdata related to the learners' interaction behavior.
In non-immersive computer-based learning environments (CBLE), to examine thelearners’ learning behavior, the data that is logged has the attributes as mouse-wheel,mouse-wheel click, mouse click left and right, key-stroke, and the mouse movements inaddition to the exercise, activity, and timestamp (Rajendran, Munshi, Emara, & Biswas, 2018).
Similarly, we developed a mechanism that is able to log all the interaction behavior of thelearners happening through HHCs in the VRLE along with timestamps in real-time (Prakash, &Rajendran, 2022; Prakash, Shaikh, & Rajendran, 2023). The designed data collection
mechanism is suitable for integration with an immersive VR system where the HMD isconnected to a desktop computer. The development of the IBD collection mechanism is
discussed in Prakash, & Rajendran, (2022) and Prakash, Shaikh, & Rajendran, (2023). Theinteraction behaviors are logged in a .csv file in the memory of the computer to which the VR
HMD is tethered. The IBD logger contains information such as the HHC used (left or right),
buttons used (grip, trigger, control buttons, and thumbstick), button actions (clicked, unclicked,pressed, released, touched, and untouched), button pressure (a value between 0 and 1
indicating the pressure applied), thumbstick axis (x and y co-ordinates), thumbstick angle (avalue between 0 and 360 ), object interacted and timestamps. An excerpt of the IBD logger isshown in Figure 1. The IBD collection mechanism is deployed in MaroonVR (Pirker, Holly,
Lesjak, Kopf, & Gütl, 2019), a VRLE used to learn the concepts of electromagnetic induction.
Electromagnetic induction is the phenomenon of inducing electromotive force (emf) by moving
a magnet in and around the close proximity of a coil. We used three scenes of Maroon VR: 1)
Faraday’s law experiment (the magnet is grabbed and dragged inside the coil to generateemf), 2) the Falling coil experiment (the magnet and an iron bar are allowed to fall freely inside
the coil to observe the emf), and 3) the Perspective scene (learners take the perspective of themagnet and generate emf through their walking). The interactions happening when the coilturns (2, 4, and 6 turns), coil diameter (2, 4, and 6 units), and magnetic field strength arevaried using virtual interfaces, and the magnet is grabbed, dragged, and dropped and thewalking of the learners in the perspective scene are logged in the IBD logger.
00

Figure 1.An Excerpt of Interaction Behavioral Data Logger

2.4 Research Questions
We measured the impact of VR intervention in learning the concept of electromagnetic
induction in the VRLE, MaroonVR as shown in Table 1. In this paper, we have used theinteraction behavioral data collected to answer the following research questions.
1. Is there a difference in the interaction behavior between the high and low performers?
2. Is it possible to predict learners' learning outcome based on the actions extracted fromtheir interaction behavior?

493

3. Research Methodology
3.1 Study Design
The VR system utilized in the study (Meta's Oculus Quest 2) includes a precautionary noticeindicating that it is not suitable for individuals under the age of 13. Consequently, we refrained
from involving school students in our experimentation. Furthermore, since the learningmaterial is already familiar to electrical engineering students, we excluded them from ourstudy. Instead, we opted to conduct our research with a group of fourteen undergraduate
engineering students randomly selected from the computer science engineering department,
all of whom possess a non-electrical engineering background. After collecting the details
related to demography, and familiarity with VR technology from the study participants, theparticipants’ prior knowledge on the topic of electromagnetic induction was collected using a
pre-test. The participants were allowed to play ‘First Steps’, a VR game for approximately 15
minutes to get familiarized with the controllers of the VR system. Then they experienced the
MaroonVR VRLE for approximately 30 minutes. The IBD was collected non-intrusively. After
the VR intervention, a post-test was conducted to assess the impact of VRLE on the learning
outcome. The experiment was conducted with 1 participant at a time and the total study time
for a single participant was approximately 1 hour and 15 minutes.

3.2 Analyses
We measured the impact of VR intervention on the learning outcome in our previous
publication (Prakash, Shaikh, & Rajendran, 2023). The evaluated result is shown in Table 1.
Table 1. Pre-to-post learning gains - all students (n=14)Pre-test Score
Mean (SD)

Post-test Score
Mean (SD)

Normalized Gain Effect Size(SD)
Cohen’s dPaired t-test(p-value)

5.86 (1.75)

7.64 (2.06)

0.42 (0.68)

3.2 (0.04)

0.81

As we intend to explore the impact of interaction behavior in VRLE on the performance,
we bifurcated the participants into high and low performers. The participants scoring morethan the mean in the post-test are considered high performers and the others as low
performers. Accordingly, there are 8 high performers and 6 low performers.
The action events are extracted from the columns of ‘Controller Index’, ‘Button’, ‘Button
Action’, and ‘Object’ of the IBD logger to evaluate the difference in the interaction behavior of
the high performers and low performers to answer the first research question. The different
action events identified from the logged IBD are shown in the Table 2.
Table 2. Action Events extracted from the IBD logger and their description
Action Events

Description

INFO
NAVIGATE
INTERACT_REL
PERS_WALKSET_COIL_REL
SET_MAG_REL

Reading instruction
Teleport, Scene switching
Handling virtual objects such as magnet and iron bar
Walking in the perspective scene taking the perspective of magnet
Setting the turns in the coil between 2 turns, 4 turns and 6 turns
Varying the magnetic field strength using VR slider interface

In addition to the interactive actions shown in Table 2, the participants also perform
actions which are not logged by the IBD collection mechanism such as walking and turning infalling coil scene and faraday scene, learners’ utterances, and learners’ seeing action. These
actions are coined as non-interactive actions.

494

As mentioned by Prakash, & Rajendran, (2022), the VRLE can be interacted with using
specific buttons present in the HHCs such as the Grip button to interact with VR objects andthe Trigger button to interact with virtual interfaces. No interactions happen in the VRLE due tothe use of irrelevant buttons other than the specified ones and hence they are also kept under
the umbrella of non-interactive actions along with the other non-interactive actions such aslearners’ utterances and seeing.
A comparative analysis is performed between high performers and low performers to
examine the duration and frequency of action events. Several tests are conducted todetermine if there are significant differences in prior knowledge, knowledge gained after VRintervention, and the extracted features of action events from the IBD logger. Pearson's
correlation analysis is utilized to identify action events that exhibit a significant relationshipwith the learning outcome. Following the correlation analysis, multiple linear regression
analysis is conducted using the selected action events to answer the second researchquestion. To determine the optimal predictor model, various information criteria, including the
Akaike information criterion (AIC), Bayesian information criterion (BIC), and Hannan-Quinn
information criterion (HQIC) scores, are evaluated. This predictor model enables theprediction of performance and the provision of personalized feedback, hints, and learning
content to enhance the learning outcome.

4. Results and Discussion
The effect of VR intervention on the learning outcome is evaluated from the data collectedusing the pre-test and post-test (Prakash, Shaikh, & Rajendran, 2023). We measured thenormalized learning gain and effect size to understand the impact of the effect produced by VRintervention. We also found using a paired t-test that the post-test score was significantly
higher than the pre-test score with t(13)=3.2, p=0.04, and identified that the significant learning
outcome resulted was due to VR intervention (see Table 1).

4.1 Research Question 1: Is there a difference in the interaction behavior between thehigh and low performers?
Participants are divided into high and low performers to examine how VRLE interaction
behavior influences the learning outcome. The homogeneity between the high performers andlow performers before VR intervention is assessed using Levene's test. The results ofLevene's test indicate that there is no significant difference in homogeneity between the highperformers and low performers (p < .05) prior to VR intervention (F = 0.545, p-value = .474).
Therefore, the requirement for homogeneity is satisfied.
The significant differences between the high performers and low performers on various
parameters were evaluated using a series of Mann-Whiteney U-tests. The results aretabulated in Table 3.
Table 3. Significance test results for the difference between the high and low performers
Pre-test Score Mean(SD)
Low Performers
5.33 (1.51)
High Performers
6.25 (1.91)
Mann-Whiteney U
U=16.5, andTest Score
p-value=.368*-significant at p-value < .05

VR Intervention
Duration Mean (SD)
2256.17 (649.17)
1743.25 (294.5)
U=13, andp-value=.174

Post-test Score Mean
(SD)
5.67 (1.21)
9.13 (0.99)
U=0,andp-value= .002*

Despite no significant difference in prior knowledge (Mann-Whitney test on pre-testscores), the duration of VR intervention (Mann-Whitney test on VR intervention duration), andthe homogeneity of pre-test scores between high and low performers (Levene's test), there is495

a significant difference in the learning outcome (Mann-Whitney test on post-test scores). Thissuggests that the variation in learning outcomes among participants is attributed to differencesin their interaction behavior within the VRLE. Consequently, various action events, along with
their duration and frequency, were extracted from the IBD logger to investigate the impact ofparticipants' interaction behavior on the learning outcome.
Although there is no significant difference in the duration spent in the VRLE between
high performers and low performers, we noticed that high performers exhibited a higher
number of relevant interactive actions and spent more time engaging in those actions
compared to low performers. Relevant interactive actions are the action events defined in்௢௧௔௟஽௨௥௔௧௜௢௡௢௙௔௖௧௜௢௡௘௩௘௡௧௦
Table 2. We calculated ܴ݈݁݁‫݊݋݅ݐܽݎݑܦ݊݋݅ݐܿܽݎ݁ݐ݊ܫݐ݊ܽݒ‬Ψ ൌ and்௢௧௔௟஽௨௥௔௧௜௢௡௢௙௏ோூ௡௧௘௥௩௘௡௧௜௢௡
ܴ݈݁݁‫ݕܿ݊݁ݑݍ݁ݎܨ݊݋݅ݐܿܽݎ݁ݐ݊ܫݐ݊ܽݒ‬Ψ ൌ ௌ௨௠௢௙௙௥௘௤௨௘௡௖௬௢௙௔௖௧௜௢௡௘௩௘௡௧௦
ௌ௨௠௢௙௙௥௘௤௨௘௡௖௬௢௙ሺ௔௖௧௜௢௡௘௩௘௡௧௦ା௡௢௡ି௜௡௧௘௥௔௖௧௜௩௘௔௖௧௜௢௡௦ሻ

.

The one-tailed Mann-Whitney U test conducted on the Relevant Interaction Duration %revealed a significant difference at p<.05 between high performers and low performers, U = 9,
p = 0.031. However, the one-tailed Mann-Whitney U test conducted on the RelevantInteraction Frequency % showed no significant difference at p<.05 between high performers
and low performers, U = 24, p = 0.476. Thus it is found that there is a significant difference inthe duration of relevant interaction between the high and low performers. Whereas, nosignificant difference is observed in the number of interactions between the high and low
performers. The descriptive statistics of the Relevant Interaction Duration % and the RelevantInteraction frequency % for the high performers and low performers are shown in the Figure 2
as a box plot.

Figure 2. Difference in the interaction behavior between the high and low performers

4.2 Research Question 2: Is it possible to predict learners' learning outcome based on
the actions extracted from their interaction behavior?
4.2.1 Correlation Analysis
The difference in the learning outcome between high performers and low performers can be
attributed to the interaction behavior of the learners within the VRLE, as indicated by the496results of the Mann-Whitney test and Levene's test. To further explore this relationship, we
conducted a Pearson correlation analysis between the action events identified from the IBD
logger and the post-test score. The results of the correlation analysis are presented in Table 4.
Table 4. Correlation Analysis Results (Bolded Variables Indicate Significant Correlation with
Post-Test Score)
Learning Outcome
Variable

Post-test score

Action events

Pearson’s r

p-value

Total VR Intervention Duration-0.695.006

INFO_DUR

0.196

.503

NAVIGATE_REL_DUR-0.144.623

INTERACT_REL_DUR-0.068.82

PERS_WALK_DUR-0.116.693

Total Number of Interactions-0.302.296

NAVIGATE_REL_FREQ-0.626.017

SET_COIL_REL_FREQ-0.487.078

SET_MAG_REL_FREQ-0.662.01

INTERACT_REL_FREQ-0.344.228

PERS_WALK_FREQ-0.125.673

Non-Interactive Action Duration-0.654.011

From the Table 4, the variables Total VR Intervention Duration,
NAVIGATE_REL_FREQ, and SET_MAG_REL_FREQ show a significant negative correlation
with the post-test score, while the variable Non-Interactive Action Duration exhibits asignificant positive correlation. An increase in the values of the negatively correlated variables
is associated with a decrease in the post-test score, whereas an increase in the value of thepositively correlated variable is linked to an increase in the post-test score.
4.2.2 Regression Analysis
The regression analysis was conducted using the variables having a higher correlation withthe post-test score. The forward feature selection algorithm was used to develop multiple
linear regression models.
Table 5. Scores of various scales to choose optimum predictor model to predict learning
outcome using action events of interaction behavior
Regression Model

No. ofPredictors

AIC

BIC

HQIC

R2

RMSE

Non-Interactive Action
Duration,
3
NAVIGATE_REL_FREQ,SET_MAG_REL_FREQ

47.66950.226-31.906

0.555

1.373

Total VR Intervention
Duration, Non-Interactive
Action Duration,
4
NAVIGATE_REL_FREQ,SET_MAG_REL_FREQ

49.62552.821-29.921

0.360

1.646

497

To find the best regression model out of the various models developed we evaluatedthe scores of AIC, BIC, and HQIC (Ventura, M., Saulo, H., Leiva, V., & Monsueto, S., 2019).
The model having the minimum score of AIC, BIC, and HQIC is chosen as the optimum model
to predict the learning outcome. The AIC and BIC scores are low for the model having theinteraction
behavior
variables
suchas
Non-Interactive
ActionDuration,
NAVIGATE_REL_FREQ, and SET_MAG_REL_FREQ as predictors is considered as Model
1. Whereas, a minimum HQIC score is observed for the model having the variables such as
Total VR Intervention Duration, Non-Interactive Action Duration, NAVIGATE_REL_FREQ,
and SET_MAG_REL_FREQ as predictors is considered as Model 2. Hence to choose theoptimum regression model we evaluated the performance of the two models. The
performance of the regression models was evaluated using a training set size of 66% and atest set size of 34%, with 10 iterations of random sampling. Model 1, which used
Non-Interactive Action Duration, NAVIGATE_REL_FREQ, and SET_MAG_REL_FREQ as
predictors, demonstrated higher R2 (0.555) and lower root mean square error (RMSE) (1.373)
compared to Model 2 (R2=0.360, RMSE=1.646). Thus, Model 1 was chosen as the optimum
regression model to predict the learning outcome. The scores of various information criterionof the regression models is shown in Table 5.
The result of the multiple regression analysis done on the model 1 chosen as the
optimum model is shown in Table 6. The value of R = 0.55 for the optimal model indicates that
55% of the variance is explained by the model. The B value in the Table 6 indicates theaverage change in the post-test score (outcome variable) by 1 unit when the corresponding
predictor variables are changed by the given value keeping all the other variables constant.
Furthermore, the multicollinearity assumption was tested, and the results indicated that thevariance inflation factor (VIF) of all predictor variables was below 10, suggesting thatmulticollinearity was not violated.
2

Table 6. Performance of the optimum predictor model
Variables
Non-Interactive Action
Duration
NAVIGATE_REL_FREQ
SET_MAG_REL_FREQ

R2
0.555

B
-0.001p-value
0.02

VIF
0.244

-0.026-0.241

0.044
0.4

0.181
0.182

5. Conclusion
This study examined the interaction behavior of learners in VRLE and identified differences inthe duration and frequency of action events between high performers and low performers. The
duration of relevant actions performed by participants was found to significantly impact the
post-test scores, differentiating high performers from low performers. Additionally, an optimal
predictor model was developed using variables related to learners' interaction behavior,includingNon-Interactive
ActionDuration,
NAVIGATE_REL_FREQ,
andSET_MAG_REL_FREQ. The predictor model demonstrated performance with an R value of0.555. The study ensured that the variables in the model did not violate the multicollinearity
assumption.
Despite the fact that the study was effective in collecting IBD and fitting it using a linear
regression model, it was carried out with a lower sample size of 14 patients. Hence, further
study with a larger sample size is needed to make the regression model more predictable andtransportable. In addition, further work is also required to add the information related to thelearner’s views and the pace of interactive actions as important traits in understanding theinteraction behavior of the learners in VRLE.
The current research paper has contributed to 1) identifying the key difference
between the high and low performers from the perspective of interaction behavior in VRLE, 2)
designing a predictor model using significant variables of action events extracted from theinteraction behavior, and 3) measuring the impact of VR on learning the subject area of2498

electronics engineering as VR studies in electronics engineering are limited. In addition, thisstudy suggests further implications for mining behavioral patterns from the IBD and analyzing
the differences in behavioral patterns between high and low performers. Moreover, the study
proposes that the interaction behavior comprehended with the performance of the learners
can be used by the developers to design VR learner models that can offer adaptive andindividualized feedback, hints, and learning content. As a next step, we aim to expand thescope by developing a VR-based Adaptive Tutoring System based on these concepts.

References
Albus, P., Vogt, A., & Seufert, T. (2021). Signaling in virtual reality influences learning outcome andcognitive load. Computers & Education, 166, 104154.
Chavez, B., & Bayona, S. (2018). Virtual reality in the learning process. In Trends and Advances inInformation Systems and Technologies: Volume 2 6 (pp. 1345-1356). Springer International
Publishing.
Feng, Z., González, V. A., Amor, R., Lovreglio, R., & Cabrera-Guerrero, G. (2018). Immersive virtual
reality serious games for evacuation training and research: A systematic literature
review. Computers & Education, 127, 252-266.
Hamilton, D., McKechnie, J., Edgerton, E., & Wilson, C. (2021). Immersive virtual reality as apedagogical tool in education: a systematic literature review of quantitative learning outcomes andexperimental design. Journal of Computers in Education, 8(1), 1-32.
Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019). Adding immersive virtual reality to a sciencelab simulation causes more presence but less learning. Learning and instruction, 60, 225-236.
Mikropoulos, T. A., & Bellou, J. (2010). The unique features of educational virtual environments.
In Teaching and learning with technology (pp. 269-278). Routledge.
Olmos-Raya, E., Ferreira-Cavalcanti, J., Contero, M., Castellanos, M. C., Giglioli, I. A. C., & Alcañiz, M.
(2018). Mobile virtual reality as an educational platform: A pilot study on the impact of immersion
and positive emotion induction in the learning process. EURASIA Journal of Mathematics, Scienceand Technology Education, 14(6), 2045-2057.
Olade, I., Fleming, C., & Liang, H. N. (2020). Biomove: Biometric user identification from humankinesiological movements for virtual reality systems. Sensors, 20(10), 2944.
Pathan, R., Rajendran, R., & Murthy, S. (2020). Mechanism to capture learner’s interaction in VR-basedlearning environment: design and application. Smart Learning Environments, 7(1), 1-15.
Pirker, J., Holly, M., Lesjak, I., Kopf, J., & Gütl, C. (2019). Maroonvr—an interactive and immersive
virtual reality physics laboratory. Learning in a digital world: Perspective on interactive
technologies for formal and informal education, 213-238.
Prakash, A., Shaikh, D. S., & Rajendran, R. (2023, March). Investigating Interaction Behaviors of
Learners in VR Learning Environment. In 2023 IEEE Conference on Virtual Reality and 3D User
Interfaces Abstracts and Workshops (VRW) (pp. 931-932). IEEE.
Prakash, A., & Rajendran, R. (2022, July). Mechanism to Capture Learners’ Interactions in Virtual
Reality Learning Environment. In 2022 International Conference on Advanced Learning
Technologies (ICALT) (pp. 335-337). IEEE.
Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immersive
virtual reality applications for higher education: Design elements, lessons learned, and research
agenda. Computers & Education, 147, 103778.
Rajendran, R., Munshi, A., Emara, M., & Biswas, G. (2018, November). A temporal model of learner
behaviors in OELEs using process mining. In Proceedings of ICCE (pp. 276-285).
Santamaría-Bonfil, G., Ibáñez, M. B., Pérez-Ramírez, M., Arroyo-Figueroa, G., & Martínez-Álvarez, F.
(2020). Learning analytics for student modeling in virtual reality training systems: Lineworkers
case. Computers & Education, 151, 103871.
Sharma, L., Jin, R., Prabhakaran, B., & Gans, M. (2018, April). LearnDNA: an interactive VR application
for learning DNA structure. In Proceedings of the 3rd International Workshop on Interactive andSpatial Computing (pp. 80-87).
Ventura, M., Saulo, H., Leiva, V., & Monsueto, S. (2019). LogǦsymmetric regression models:
information criteria and application to movie business and industry data with economic
implications. Applied Stochastic Models in Business and Industry, 35(4), 963-977.
Wade, J., Zhang, L., Bian, D., Fan, J., Swanson, A., Weitlauf, A., ... & Sarkar, N. (2016). A
gaze-contingent adaptive virtual reality driving environment for intervention in individuals with
autism spectrum disorders. ACM Transactions on Interactive Intelligent Systems (TiiS), 6(1), 1-23.

499

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Keeping Teams in the Game: Predicting
Dropouts in Online Problem-Based
Learning CompetitionaAditya PANWARa*, Ashwin TSa, Ramkumar RAJENDRANa & Kavi ARYAa
IDP in Educational Technology, Indian Institute of Technology Bombay, INDIA
adityapanwar@iitb.ac.in

Abstract: Online learning and MOOCs have become increasingly popular in recent
years, and the trend will continue, given the technology boom. There is a dire need toobserve learners' behavior in these online courses, similar to what instructors do in aface-to-face classroom. Learners’ strategies and activities become crucial tounderstanding their behavior. One major challenge in online courses is predicting andpreventing dropout behavior. While several studies have tried to perform suchanalysis, there is still a shortage of studies that employ different data streams tounderstand and predict the drop rates. Moreover, studies rarely use a fully online
team-based collaborative environment as their context. Thus, the current study
employs an online longitudinal problem-based learning (PBL) collaborative robotics
competition as the testbed. Through methodological triangulation, the study aims topredict dropout behavior via the contributions of Discourse discussion forum‘activities’ of participating teams, along with a self-reported Online Learning
Strategies Questionnaire (OSLQ). The study also uses Qualitative interviews toenhance the ground truth and results. The OSLQ data is collected from more than
4000 participants. Furthermore, the study seeks to establish the reliability of OSLQ toadvance research within online environments. Various Machine Learning algorithms
are applied to analyze the data. The findings demonstrate the reliability of OSLQ with
our substantial sample size and reveal promising results for predicting the dropout
rate in online competition. Overall, the study contributes to online learning byaddressing the need to understand and predict dropout behavior in online courses.
The study’s methodological triangulation, involving qualitative interviews, provides
insights into such contexts' unique dynamics and challenges by utilizing a fully online
team-based collaborative environment.
Keywords: Predictive Models, Machine Learning, Discussion Forum Interaction,
Self-Regulated Learning, PBL, Collaboration, Robotics

1.

Introduction & Related Literature

There has been a tremendous increase in online courses worldwide for Undergraduate
students in engineering and all other fields. COVID-19 has further triggered MOOC
enrolment numbers to an exponential level (Baudo & Mezzera, 2021). However, the mostprominent drawback of online courses is their high attrition rate; the completion rate usually
falls between 2-10% (Maxwell et al., 2018). The success and learning in online courses may
be directly attributed to student performance, which can be further attributed to several key
factors like content interaction, student satisfaction (Zimmerman, 2012), engagement &activity with course material (Soffer & Cohen, 2019), and motivation (Loizzo, Ertmer, Watson,& Watson, 2017), to list a few. Many such factors can be observed or assessed in a
traditional classroom with limited participants. The instructor can guide, adapt and help the‘failing’ students towards success. All these factors become even more critical when dealing
with the online setting where the significant learner interactions are via the resources/content
and maybe a discussion or query forum. This necessitates exploring the reasons fordropouts to alleviate this issue and provide an opportunity to inform the instructors to adapt
the curriculum/task design, motivate, provide feedback, and devise other ways to keep500

success high. It is known that there are numerous benefits of online courses, such as richcontent, online asynchronous learning, and low cost. Still, despite that, they continue tosuffer from significant dropout rates, hampering pedagogical and economic goals (Quadri &Shukor, 2021). Predicting students' likelihood to complete (or not to complete) a MOOC
course, especially from the early weeks, has been one of the hottest research topics in thearea of learning analytics (Alamri, Alshehri, Cristea, Pereira, Oliveira, Shi, & Stewart, 2019).
Predictive models provide timely information about learners at risk of dropout to inform
interventions. Instructors and learners can benefit from the results of the predictive models(Moreno-Marcos, Alario-Hoyos, Muñoz-Merino, & Kloos, 2018), which can help modify andimprove course content and pedagogy to reduce dropouts (Moreno-Marcos, Muñoz-Merino,
Maldonado-Mahauad, Pérez-Sanagustín, Alario-Hoyos, & Delgado Kloos, 2020). There is ahuge potential for data analytics on students’ learning processes and outcomes in higher
education (Aldowah, Al-Samarraie, & Fauzy, 2019). Learning analytics is the measurement,collection, analysis, and reporting of data about learners and their contexts to understandand optimize learning and the environments in which it occurs (Siemens & Long, 2011).
Students’ interaction with online content is crucial and utilized in many research studies.
Students’ active participation in forums was found to enhance overall performance. The
completion level was also closely linked to prior online experience and educational
attainment (Meneses & Marlon, 2020). Students’ interactions are an important source forunderstanding the students’ behavioral patterns in an online learning process. Other
researchers have also claimed that students' social and personal information, such as
gender, age, or place, significantly impacted their performance and learning outcomes ingeneral (Mubarak, Cao, & Zhang, 2020). Success in an online course requires a high level ofdiscipline and self-direction, and proper time management to complete the assignments.
Thus, certain learning strategies must be developed, and the instructors should encourage
the development. Peer Learning should also be encouraged by forming project/study groups
wherever possible. Though many prediction models have been developed, they usually don’t
consider high-level factors like Self-Regulated Learning (SRL), which can greatly impact thelearners’ success. Prior works have also shown that the lack of SRL skills can be an
important factor that leads to failure and dropout in MOOCs (Terras & Ramsay, J. 2015). In a
study conducted by a research group at Harvard (Whitehill et al., 2015), multinomial logistic
regression was employed to identify students at risk of dropping out of a MOOC. Other
algorithms, including Logistic and Linear Regression, Forest Regression, Decision Tree, etc.,
have also been employed (Liao et al., 2019). It has been noticed that although a decent
amount of work has been done in predicting dropouts, they mostly focus on a single streamof data, like learners’ interaction with content. The dropout phenomena are studied
quantitatively, whereas research emphasizes that dropouts can be highly qualitative andcomplex (Simpson, 2010). Dropout studies thus lack information on importantsocio-psychological causes and contingencies (e.g., academic workload, personal
experiences, and other commitments). SRL is found to be one of the major factors in online
learning. Yet, it is not fully utilized for prediction, so a need is there to use multiple data
sources [learner interaction in an online forum, SRL Strategies, Qualitative Studies, learnerprofile, etc.] as combinations and predictors. There has been quite a lot of research going onin the field of motivation and Learning Strategies, yet there is a research gap in purely online
learning scenarios. SRL is usually studied through questionnaires; the Motivated Strategies
for Learning Questionnaire (MSLQ) has been widely used. However, it was developed in1991 and is primarily suited to face-to-face classrooms (Zhou & Wang, 2021). Efforts havebeen made to modify MSLQ part B for Distance Education (Meijs, Neroni, Gijselaers,
Leontjevas, Kirschner, & de Groot, 2019). Still, not many descriptive studies have been
conducted to establish the validity. Several other instruments like MSLQ (basic), MSLQ
modified, and Self-regulated Online Learning Questionnaire SOL-Q (Jansen, van Leeuwen,
Janssen, Kester, & Kalz, 2017), Technology Innovation Questionnaire, NSSE (Pascarella,
Seifert, & Blaich, 2010), Educause Student Technology Survey (Dahlstrom, & Bichsel, 2014),
and Self-Regulation for Learning Online (SRL-O) (Broadbent, Panadero, Lodge, &Fuller-Tyszkiewicz, 2022) are also sporadically used in studies. Online Self-Regulated
Learning Questionnaire or OSLQ (Barnard, Lan, To, Paton, & Lai, 2009; Bruso & Stefaniak,

501

2016) is one such instrument developed solely for the online environment andaccommodates the most important scales from the literature like time management, task
strategies, etc. However, many more studies with large sample sizes (Rufini, Fernandes,
Bianchini, & Alliprandini, 2021) using OSLQ are warranted.
Several dropout prediction models are created, but they are carried out as posthoc
analysis, which fails to ‘anticipate’ the dropouts; early predictions and sequentially suitableintervention strategies are thus the need of the hour. Moreover, the research almost always
involves individual participants, so there is a huge shortage of team or collaborative
research. The ability to predict a team's performance can have a strong pedagogicalpotential that has not yet been explored sufficiently in the related literature (Giannakas,
Troussas, Voyiatzis, & Sgouropoulou, 2021). Furthermore, when dealing with questionnaires,
issues related to self-reported data are always present; this can be somewhat circumvented
by conducting longitudinal studies and qualitative interviews. Through this study, we are thus
trying to tackle these gaps via
1.
Collecting self-reported self-regulated learning (SRL) data through a questionnaire
and learner activity data from the discussion forum,
2.
Analyzing the SRL data to understand its implications within a team-based
problem-based learning (PBL) environment,
3.
Developing Early Prediction models specifically designed for an online team-based
setting to reduce attrition,
4.
Augmenting the investigation with qualitative interviews conducted with teams/teammembers.
Based on these aims, we propose the following Research Questions:
RQ1:How reliably can the Online Strategies for Learning Questionnaire (OSLQ) demonstrate
the level of Self-Regulated Learning (SRL) in the context of a fully online problem-basedlearning (PBL) competition?
RQ2:How effective are the early prediction models based on discussion forum interactions
and the OSLQ in anticipating and preventing team dropouts in the online PBL environment?
In the upcoming sections of this study, we will describe the study environment and discuss
the data collection methods, specifically the utilization of the OSLQ (SRL Questionnaire) andthe incorporation of Discourse discussion forum activity data like Topics visited, Posts read,
likes received, and given. We will then proceed to build our prediction models and present
the results. Finally, we will conclude with a comprehensive discussion section that analyzes
and interprets the findings, contextualizing them within the broader scope of our researchand exploring their implications.

2.

Study Environment

To address our RQs, we use e-Yantra Robotics Competition (eYRC) as our study
environment. One of the authors worked within e-Yantra, which allowed the use of eYRC.
e-Yantra is an initiative hosted in IIT Bombay (in CSE Dept.) with the support of the Ministryof Education (MoE) to spread a wide variety of technical skills across the college. e-Yantra’s
initiatives complement the existing education system with a “problem-solving” culture
through project-based learning (PBL) approach using Simulators & Robotic Kits. eYRC
focuses on building not only “HARD'' [STEM, Image processing, control systems, ML,embedded systems, Functional programming, etc.] but “SOFT” [teamwork,
time-management, dealing with failure, etc.] skills too. eYRC is unique because it is really a
MOOC-style competition that trains participants in complex engineering while competing.
The students participate as a team of two to four student members from any year and any
branch. Participating teams try to solve gamified real-world problem statements called
Themes and are abstracted on a static arena. Incepted in 2012 (eYRC-2012), thecompetition occurs online in two stages, as shown in Figure 1. Stage 1 is usuallysimulator/programming based, comprising three tasks-Task 0, Task 1, and Task 2, occurring
during September-November. Teams are then selected based on their performance and502

advance into Stage 2, which involves hardware and learnings from Stage 1. It consists of
Task 3, Task 4, Task 5, and Task 6 conducted between December-March. Every teammember has access to all resources, the discussion forum, eYRC website (that hosts
questionnaires, feedback forms, etc., needed to be filled out individually by each member).
The submission of each task is, however, team-based rather than individual. Each theme
comprises tasks and constant feedback provided through various online discussion forums,
including Discourse, Discord, and Google Hangouts. The selection of eYRC as the researchsetting is influenced by its unique attributes as a comprehensive online collaborative robotics
competition. Furthermore, segmenting problem statements into distinct deadline-driven tasks
within eYRC offers an opportune platform for applying algorithms in early prediction
endeavors. First, we gather individual members’ SRL questionnaire responses via thewebsite and each member’s interaction data using the Discourse discussion forum. It is
crucial to consider that interaction data should be analyzed within the specific time frame
assigned to each task, as each task has its own deadline. Semi-structured interviews arealso conducted with the eYRC participants for our study to delve deeper into dropout
behavior. Then, using methodological triangulation, we try to achieve the study goals.

Figure 1. eYRC Structure3.

Research Design & Methodology

Our study employs methodological triangulation that uses more than one kind of method tostudy a phenomenon. It is beneficial in confirming findings, more comprehensive data,
increasing validity, and enhancing understanding of studied phenomena (Bekhet &Zauszniewski, 2012). This framework underpins our use of the SRL questionnaire,
Discussion forum, and qualitative interviews, as detailed in the following sections.

3.1 Learning Strategies Questionnaire
For this study, OSLQ: Online Strategies for Learning Questionnaire, which has six
sub-scales, namely, Goal setting, Environment Setting, Task Strategies, TimeManagement, Help Seeking, and Self Evaluation, is used. OSLQ is rated via a 5-point
Likert Scale, with 1 being Strongly Disagree and 5 being Strongly Agree. OSLQ is a
developed model; hence no Exploratory Factor Analysis (EFA) will be necessary;
Confirmatory Factor Analysis (CFA) will be required to establish reliability and address RQ1.
CFA enables us to establish construct validity by testing the relationship between observed
factors (question items) and latent factors (subscales) proposed by the model. This step isnecessary to ensure that subscale scores within the SRL framework can be distinguishedand thus reported separately. Jamovi being open-source and cloud-based, was tested for503

performing the CFA. OSLQ was administered to participants via the web portal with barely
any modifications to 10011 individual participants representing 2610 teams registeredentering Stage 1 of the competition to attempt Task 0 [basic software installation and a
supplementary task]. 4907 individual participants, representing 1520 teams, had filled
out the questionnaire, and further analysis was performed on this dataset.

3.2 Discourse Discussion Forum Data
To address RQ2, Discourse (an open-source discussion forum) was utilized in thecompetition. Participants were enrolled here to access various resources, announcements
and post their queries. Discourse readily provides forming “groups,” where an instructor can
add only certain participants to a certain group relevant to their 'theme.' Six separate groups
were formed for six Themes in the analyzed year (2022-23). The total number of individual
participants from these six themes was 7604 representing 2214 teams.
Discussion forum learner activity was gathered using the Data Explorer feature of
Discourse, where full-fledged Structured Query Language (SQL) queries can be written.
Queries like Poll Statistics, Lurkers, High Likers, and many more can be explored by simply
running them on the cloud. We also leveraged SQL queries to extrapolate each group's datausing their unique group_id. This data was first separately collected for each theme groupbased on the tasks’ timeline and then later combined for overall analysis. Major features
identified from the data were Topics Entered, Posts Count, Likes Given, and Likes
Received (see Appendix A)1. Since these values were individual-based, they had to befurther processed to fit the team-based style of the competition. The MAX values of eachfeature were finalized because of the nature of the competition that even if one teammember “visits” the resources, it can be constituted to the whole team. Hence, we finallyutilized four features for further analysis and model building for each team: Max Likes
received, Max Likes Given, Max Topics Entered, and Max Posts Read. Furthermore, thetime interval (deadline) for the above features between tasks was also crucial since we aimfor early predictions. For example, for task1 dropout prediction, we only used the features
extracted before the task1 deadline, i.e., Task 0 and Task 1. A similar process was followed
for other predictions.
3.2.1 Building Machine Learning Models
All algorithms were run in Python using Google Colab. Since the OSLQ data is in Likert
Scale, it was first standardized using suitable Python libraries.
3.2.1.1. Procedure
To build our model, we first performed data pre-processing. An overview of the steps:
1. OSLQ data with aggregate (1520 teams) added with the team’s Discourse activity.
2. After this, each team’s task submission data was added. Task submission data eitherscored (value >=0) or value NULL, representing a team not submitting task
3. NULL value and scores were suitably standardized to 0 (Not Submitted) and 1(Submitted) using label encoders. Performance or scoring is not considered to keep
the model simple (see Appendix B1 for the Merged dataset description).
4. As our environment resembles a MOOC model, there was a lot of imbalance in thedataset. An example of this is shown in Figure 2; teams selected in Task 0 of thecompetition.
5. We used SMOTE as the technique to balance the data. SMOTE stands for Synthetic
Minority Over-sampling Technique, and it works by generating synthetic examples ofthe minority class by interpolating between existing examples in the minority class.
Similar data balancing is also used to balance the task-wise data since we focus onearly prediction. Figure 3 shows Task 0 composition after SMOTE.
1Appendix Document504

Figure 2. Task 0 Imbalanced Dataset

Figure 3. Task 0 Balanced Dataset

6. We then employed competing ML methods: Logistic Regression, Decision Trees, andRandom Forest (see Appendix B2 for details).
7. After building the above models based on a training and test set division of
80%/20%, we also evaluated the models for all performance metrics: accuracy,
precision, recall, F1-score, and AUC-ROC (see Appendix B2 for details).
In conclusion, the data pre-processing for our machine learning models involved combining
OSLQ data with team Discourse activity and incorporating task submission data. We
standardized the task submission data to represent submission status. Due to our
MOOC-like environment, we encountered dataset imbalance, which we addressed using the
SMOTE and employed Logistic Regression, Decision Trees, and Random Forest ascompeting ML methods. Finally, their performance was evaluated using various metrics suchas accuracy, precision, recall, F1-score, and AUC-ROC.

3.3 Qualitative Data
To establish further reliability, a qualitative study was conducted with eYRC participants thatwere selected from a pool of eYRC 2021-22 (last year’s version) who were not able toperform well in Stage 1 (dropped out before Task 2) but participated again in the current
version (eYRC 2022-23) and cracked the Stage 1 to enter the hardware-based Stage 2.
Eight participants representing six teams were invited to participate and consented to a
study. The overarching objective was to comprehend the team's view of the discussion
forum, unravel the causes of early dropout, analyze the role of discussion forum activities,
elucidate their contributions, and triangulate this with our Early Prediction models. These
interviews were conducted online over Google Meet. Audio and video were both recorded foranalysis purposes. To facilitate the discussion with the participants, guiding questions were
provided. Each group member was asked these initial questions to further the conversation(see Appendix C)2. Otter (a speech-to-text app) was employed, and handwritten notes weretaken as a backup.

4.

Results and Analysis

In this section, we present the results and analysis of the OSLQ (Self-Regulated Learning
Questionnaire) data addressing RQ1. Descriptive statistics and Cronbach's Alpha were used
to assess the internal consistency of the questionnaire. We also present the CFA results andthe adequacy of the model fit to the data as a satisfactory model fit is crucial for the reliabilityof the questionnaire. We then present the performance and evaluation of our Early
Prediction models to address RQ2. Finally, we discuss the analysis of interviews.

2

Appendix Document505

4.1 OSLQ Data Analysis
To answer our RQ1, initially, Descriptive Statistics were performed on the resulting data fromOSLQ (n = 4907). Internal consistency for all six subscales was determined using
Cronbach’s Alpha, as shown in Table 1. All subscales lie in an acceptable range of (0.70 to0.95) of Cronbach’s Alpha and are sufficiently reliable for analysis.
Table 1. OSLQ Internal Consistency
Sub Scales
Goal Setting
Environment Setting
Task Strategies
Time Management
Help Seeking
Self Evaluation

Cronbach’s Alpha (α)
0.864
0.814
0.729
0.812
0.799
0.846

In performing the CFA, five statistics reflecting fit were reported: the chi-square goodness offit statistic (χ2); the ratio of chi-square statistic to degrees of freedom (χ2/df); theComparative Fit Index (CFI); the Tucker Lewis Index (TLI) [also known as the Non-Normed
Fit Index (NNFI)]; and the root mean square error of approximation (RMSEA). The model fit
results are shown in Table 2 and Table 3 below.
Table 2. Exact Model Fit
χ²
2924

df
237

p
< .001

Table 3. Fit Measures
CFI
0.948

TLQ
.94

RMSEA
.0531

RMSEA 90% CI
Lower
Upper
.0514
0.548

Cronbach's Alpha confirmed the reliability of the questionnaire's subscales. CFA establishedthe validity of the questionnaire, ensuring that subscale scores within the SRL framework
could be reliably distinguished. The fit statistics of the CFA were examined, and a good fit
was found. Descriptive statistics, CFA loadings, and Model Fit Indices description are addedin Appendix D3.

4.2 Machine Learning Models Performance
From the initial pool of 1520 teams, a dataset comprising 1290 teams was selected for
addressing RQ2. Initial model performance was evaluated by constructing models without
oversampling using Logistic Regression, Decision Trees, and Random Forest. Subsequently,
the model was refined by incorporating minority oversampling for task0, task1, task2, andstage2 as target variables—early Prediction utilized Discussion Forum activities data before
each task's deadline. The results are presented in Table 4, Table 5, and Table 6.
Table 4. Logistic Regression report for task0, task1, task2 and stage2 prediction
task0
3

Precision
0
0.96

Recall0
1

F1 Score
00.98

Appendix Document

506

AUC-ROC
0.49

Accuracy0.95

task1task2stage2

0.61
0.96
0.85
0.89
0.92
0.71

0.43
1
0.94
0.73
0.93
0.69

0.5
0.98
0.9
0.8
0.93
0.7

0.63

0.7

0.83

0.86

0.8

0.88

Table 5. Decision Tree model report for task0, task1, task2 and stage2 prediction
task0
task1
task2
stage2

Precision
0.11
0.97
0.5
0.75
0.83
0.69
0.94
0.68

Recall
0.45
0.94
0.58
0.69
0.84
0.72
0.91
0.78

F1 Score
0.18
0.95
0.54
0.72
0.83
0.7
0.93
0.73

AUC-ROC
0.64

Accuracy
0.82

0.64

0.65

0.77

0.79

0.84

0.88

Table 6. Random Forest model report for task0, task1, task2 and stage2 prediction
task0
task1
task2
stage2

Precision
0.27
0.96
0.58
0.78
0.89
0.79
0.97
0.71

Recall
0.36
0.99
0.65
0.83
0.87
0.81
0.91
0.88

F1 Score
0.31
0.97
0.61
0.80
0.88
0.80
0.94
0.79

AUC-ROC
0.65

Accuracy
0.93

0.70

0.71

0.84

0.85

0.90

0.91

The study employed machine learning models - Logistic Regression, Decision Tree, andRandom Forest - to predict task completion and stage advancement in the e-Yantra Robotics
Competition. Results indicated varied model performance across tasks and stages. Notably,
the Random Forest model consistently outperformed others, demonstrating higher accuracy
and AUC-ROC values. For instance, in predicting Task 0 (task0), Random Forest achieved
an accuracy of 0.93 and AUC-ROC of 0.65. Similarly, Task 1 (task1) prediction achieved an
accuracy of 0.71 and an AUC-ROC of 0.70. In Task 2 (task2) and Stage 2 (stage2)
predictions, Random Forest maintained competitive accuracy and AUC-ROC values,
solidifying its predictive prowess.

4.3 Qualitative Data Analysis
Through the analysis of the interviews, several themes and issues emerged. The
participants in the study found the Discourse Discussion Forum highly effective, being more
active on it this year, assessing queries before posting. Participants found it beneficial toobserve the progress of other participants in the forum. They felt more satisfied andreassured when they discovered that others were facing similar challenges or getting stuckat similar points in the competition while acknowledging helpful solutions with likes andappreciating the forum's structure and navigation. Additionally, they demonstrated improved
time management between academics and competition, consistent effort in completingtasks, and reported the benefits of prior participation. Participants also identified issues
about the importance of diligently choosing team members and being shy to post their
queries in their previous participation, owing to how they’ll be perceived on the forum.

507

5.

Conclusion

This study aimed to predict dropout behavior in an online collaborative PBL competition
using a methodological triangulation approach. The study combined data from participatingteams' Discourse discussion forum activities, a self-reported Online Learning Strategies
Questionnaire (OSLQ), and qualitative interviews to enhance the results. OSLQ descriptive
results revealed that subscales Goal Setting (μ=4.06), Environment Setting (μ=4.1525), Help
Seeking (μ=4.03), and Self Evaluation (μ=4.06) were highly evidenced, while Task Strategies(μ=3.8225) and Time Management (μ=3.85) were on the lower side. A reliability analysis ofeach subscale was performed, and all subscales were found to be within the acceptable
range of 0.70 and above, making OSLQ valid and consistent. Using CFA, we established thereliability of OSLQ using a large sample size from various institutions with participants
representing various disciplines of Undergraduate Studies in a team-based dynamics. The
qualitative interviews also pointed toward participants' perception of the importance of TimeManagement as a success indicator. Instructors can thus create some interventions right at
the beginning of the competition to encourage participants. For building machine learning
early prediction models, we kept the general aim of keeping the models fairly simple andused discussion forum features. The models were built till Stage 1 (or Stage 2 selection) as
we focus on Early Predictions. The performance was poor at the start of the competition(task0) but gradually improved. Random Forest and Decision Trees performed better thanthe Logistic Regression models. Decision Trees and Random Forest seemingly worked wellas they incorporated non-linear relationships between the variables. These findings highlightthe potential of machine learning models, particularly Random Forest, in predicting taskcompletion and stage advancement in the online Competition. The results contribute to ourunderstanding of the factors influencing the team’s potential early dropouts and can informthe design of interventions to enhance their learning outcomes. The findings of this study
have implications for online course design, instructor interventions, and learner support
strategies. By identifying teams at risk of dropout early on, instructors can offer timely
support, adaptive feedback, and personalized interventions to improve success. However, it
is essential to acknowledge some limitations of this study. The self-reported nature of the
OSLQ data introduces potential biases and may not capture the full complexity of learners'
strategies and behaviors. In the ongoing work, we are exploring the role of individual
participants’ data like Year of Study, Prior participation, Engineering Branch, Gender, andother demographic variables that could provide even more insights on team-based researchin the online scenario. In conclusion, the contribution of this ongoing study is multifaceted;first, it cements the OSLQ’s validity and reliability through a large sample size which has
been missing in the literature. Secondly, it contributes to the growing body of research on
predicting dropout behavior in online courses, especially in a fully online longitudinal
environment that is also collaborative.

References
Baudo, V., & Mezzera, D. (2021, September). Insights from a MOOC Platform during COVID-19
Pandemic in Italy. In EDEN Conference Proceedings (No. 1, pp. 401-409).
Maxwell, W. D., Fabel, P. H., Diaz, V., Walkow, J. C., Kwiek, N. C., Kanchanaraksa, S., ... &Bookstaver, P. B. (2018). Massive open online courses in US healthcare education: practicalconsiderations and lessons learned from implementation. Currents in Pharmacy Teaching and
Learning, 10(6), 736-743.
Zimmerman, T. (2012). Exploring Learner to Content Interaction as a Success Factor in Online
Courses. International Review of Research in Open and Distributed Learning, 13(4), 152–165.
Soffer, Tal & Cohen, Anat. (2019). Students' engagement characteristics predict success andcompletion of online courses. Journal of Computer Assisted Learning. 35. 10.1111/jcal.12340.
Loizzo, J., Ertmer, P. A., Watson, W. R., & Watson, S. L. (2017). Adult MOOC Learners as
Self-Directed: Perceptions of Motivation, Success, and Completion. Online Learning, 21(2), n2.

508

Quadri, A. T., & Shukor, N. A. (2021). The Benefits of Learning Analytics to Higher Education
Institutions: A Scoping Review. International Journal of Emerging Technologies in Learning,
16(23), 4–15. https://doi.org/10.3991/ijet.v16i23.27471
Alamri, A., Alshehri, M., Cristea, A., Pereira, F. D., Oliveira, E., Shi, L., & Stewart, C. (2019).
Predicting MOOCs dropout using only two easily obtainable features from the first week’s
activities. In Intelligent Tutoring Systems: 15th International Conference, ITS 2019, Kingston,
Jamaica, June 3–7, 2019, Proceedings 15 (pp. 163-173). Springer International Publishing.
Moreno-Marcos, P. M., Alario-Hoyos, C., Muñoz-Merino, P. J., & Kloos, C. D. (2018). Prediction in
MOOCs: A review and future research directions. IEEE transactions on Learning Technologies,
12(3), 384-401.
Moreno-Marcos, P. M., Muñoz-Merino, P. J., Maldonado-Mahauad, J., Pérez-Sanagustín, M.,
Alario-Hoyos, C., & Delgado Kloos, C. (2020). Temporal analysis for dropout prediction using
self-regulated learning strategies in self-paced MOOCs. Computers and Education, 145(October2019). https://doi.org/10.1016/j.compedu.2019.103728
Aldowah, H., Al-Samarraie, H., & Fauzy, W. M. (2019). Educational data mining and learning analytics
for 21st century higher education: A review and synthesis. Telematics and Informatics, 37, 13-49.
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE
review, 46(5), 30.
Meneses, J., & Marlon, X. (2020). Dropout in online higher education: A scoping review from 2014 to2018. In Universitat Oberta de Catalunya (UOC).
Mubarak, A. A., Cao, H., & Zhang, W. (2020). Prediction of students’ early dropout based on theirinteraction logs in online learning environment. Interactive Learning Environments.
Terras, M. M., & Ramsay, J. (2015). Massive open online courses (MOOCs): Insights and challenges
from a psychological perspective. British Journal of Educational Technology, 46(3), 472–487.
https://doi.org/10.1111/bjet.12274
Whitehill, J., Williams, J., Lopez, G., Coleman, C., & Reich, J. (2015). Beyond prediction: First steps
toward automatic intervention in MOOC student stopout. Available at SSRN 2611750.
Liao, S. N., Zingaro, D., Thai, K., Alvarado, C., Griswold, W. G., & Porter, L. (2019). A robust machinelearning technique to predict low-performing students. ACM transactions on computing education(TOCE), 19(3), 1-19.
Zhou, Y., & Wang, J. (2021). Psychometric Properties of the MSLQ-B for adult distance education in
China. Frontiers in psychology, 12, 620564.
Meijs, C., Neroni, J., Gijselaers, H. J., Leontjevas, R., Kirschner, P. A., & de Groot, R. H. (2019).
Motivated strategies for learning questionnaire part B revisited: New subscales for an adultdistance education setting. The internet and higher education, 40, 1-11.
Jansen, R. S., van Leeuwen, A., Janssen, J., Kester, L., & Kalz, M. (2017). Validation of theself-regulated online learning questionnaire. Journal of Computing in Higher Education, 29(1),6–27. https://doi.org/10.1007/s12528-016-9125-x
https://www.concordia.ca/content/dam/artsci/research/cslp/docs/TIQ2.pdf
Pascarella, E. T., Seifert, T. A., & Blaich, C. (2010). How Effective are the NSSE Benchmarks in
Predicting Important Educational Outcomes? Change: The Magazine of Higher Learning, 42(1),
16–22. https://doi.org/10.1080/00091380903449060
Dahlstrom, E., & Bichsel, J. (2014). ECAR Study of Undergraduate Students and Information
Technology, 2014. Educause.
Broadbent, J., Panadero, E., Lodge, J. M., & Fuller-Tyszkiewicz, M. (2022). The self-regulation forlearning online (SRL-O) questionnaire. Metacognition and Learning, August.
Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. L. (2009). Measuring self-regulation in online
and blended learning environments. Internet and Higher Education, 12(1), 1–6.
https://doi.org/10.1007/s11409-022-09319-6
Bruso, J. L., & Stefaniak, J. E. (2016). The Use of Self-Regulated Learning Measure QuestionnairesasaPredictorofAcademic
Success.
TechTrends,
60(6),
577–584.
https://doi.org/10.1007/s11528-016-0096-6
Rufini, S. É., Fernandes, J. G., Bianchini, L. G. B., & Alliprandini, P. M. Z. (2021). Brazilian Version of
Online Self-Regulated Learning Questionnaire (OSLQ): Evidence of Validity. Psicologia: Teoria e
Pesquisa, 37.
Giannakas, F., Troussas, C., Voyiatzis, I., & Sgouropoulou, C. (2021). A deep learning classification
framework for early prediction of team-based academic performance. Applied Soft Computing,106, 107355.
Bekhet, A. K., & Zauszniewski, J. A. (2012). Methodological triangulation: An approach tounderstanding data. Nurse researcher, 20(2).

509

CUMTEL510

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Fostering Students’ Dialogic Engagementwith the Use of Visual Learning Analytics
as a Teaching Assistant Tool in Primary
School Classrooms
Fan CHENa, Pengjin WANGa, Deliang WANGa & Gaowei CHENa*
Faculty of Education, The University of Hong Kong, Hong Kong, China*gwchen@hku.hkaAbstract: Using visual discourse tools can be a valuable approach for teachers tofoster academically productive talk in the classroom. However, teachers often pay
considerable attention to highly engaged students and fail to create sufficient
opportunities for less engaged students to participate in classroom dialogue. This study
seeks to enhance learning by incorporating social network analysis to elucidate
students’ levels of engagement in classroom discourse. Over a three-week period, thisstudy analyzed both overall and individual student’s dialogic engagement in theclassroom. The results demonstrate that as teachers purposefully lead class dialogue
and limit their own speech, students gradually speak more. Additionally, those who
initially speak less are given more opportunities to engage in classroom dialogue.
These results emphasize the significance of using visualization tools to assist teachers
in orchestrating and optimizing classroom dialogue.
Keywords: Visual learning analytics, dialogue visualization, dialogic engagement,social network analysis, primary school classroom

1. Introduction
Classrooms are dynamic social and physical environments where teaching and learning
occur through interactions between teachers and students. To facilitate effective learning,teachers must organize and manipulate environmental structures, instructional activities, andclassroom resources to guide students’ attention and learning, thus allowing students toacquire a wide range of knowledge and skills. However, teachers often encounter
considerable challenges when attempting to ensure that students are engaged in learningand discussion in the classroom (Saunders-Stewart et al., 2012). Despite the availability ofvarious support tools in research (e.g., visualization-based scaffolding), few are used inpractice, and there is a lack of sufficient strategies to aid teachers in optimizing theirteaching approaches (Echeverria et al., 2019; Urhahne et al., 2009). This phenomenon isparticularly pronounced in primary school classrooms where students’ learning performanceis highly dependent on practices that place high demands on teachers (Wolery & Hemmeter,
2011). More empirical evidence is needed to explore how to support talk-based instruction inprimary school classrooms and enable students to engage in learning more effectively(Mavrikis et al., 2019; Urhahne et al., 2009).
Visual learning analytics is a combination of learning analytics and visualization
techniques that represent learning and teaching information. This approach goes beyond
simply summarizing and visualizing data and presents the classroom learning process in aform that is easy for users to understand (Hsiao & Lin, 2017). These visual approaches canalso function as teaching assistant tools. For example, Chen (2020) developed theclassroom discourse analyzer (CDA) to aid teachers in reflecting on classroom dialogue andimproving classroom teaching practices. They found that visualization-based scaffolding
enhanced teachers’ self-efficacy in orchestrating effective classroom dialogue andsignificantly impacted classroom teaching behaviors (Chen et al., 2020). While this visual511

tool had a positive impact on middle and high school classrooms, its effectiveness inelementary schools, where children are more likely to be distracted and disengaged (Downer
et al., 2007), requires further investigation. Furthermore, besides focusing on students who
are more proficient in the classroom, these analytical methods must also be extended tofocus on marginal learners and the participation of the entire class. Therefore, this study
attempts to introduce social network analysis to further support classroom teaching andinvestigate its impact on students’ engagement in classroom dialogue. The researchquestions are as follows:
RQ1: What are the influences of discourse-based visual tools on the overall dialogic
engagement in the classroom?
RQ2: What are the influences of discourse-based visual tools on individual students’
dialogic engagement in the classroom?
RQ3: Do students exhibit significant improvements in their learning performance
when discourse-based visual tools are incorporated into classroom learning?

2. Literature Review
2.1 Talk-Based Instruction
Talk-based instruction is beneficial for student learning and skill development, particularly forelementary school students with less learning experience who require more dialogue
guidance from teachers to participate in the classroom. According to the “two-thirds rule”
uncovered by Flanders (1964), verbal communication dominates classroom practice and its
quality is crucial to what and how students learn during lesson time (Khong et al., 2017).
Therefore, it is critical for teachers to consciously guide students’ awareness of their
behaviors and discourse.
Michaels and O'Connor (2015) put forward a form of talk referred to as academically
productive talk (APT), which holds several talk moves that encourage teachers to give
students more space to (1) share or clarify their initial ideas (e.g., “Can you say more aboutit?”), (2) listen carefully to one another (e.g., “Do you repeat his idea in your word?”), (3)deepen their reasoning (e.g., “Why do you think that?”), and (4) think together (e.g., “Do you
add on to his idea? Why?”) (Michaels & O’Connor, 2015; O’Connor & Michaels, 2019). APT
prompts teachers to encourage students to elaborate, reason, argue, and share thoughts toaid students’ learning. It creates an atmosphere of equal teacher-student dialogue, whichhelps students to generate new ideas in collaboration, thus deepening their own
understanding and communication skills. Studies have shown positive relationships between
productive dialogue and students’ academic achievements in mathematics (Chen et al.,2020), English (Vetter et al., 2021), and reading (Barak & Lefstein, 2021).

2.2 Discourse Visualization Tool
Discourse visualization is a technical scaffolding used in the talk-based classroom that
employs visual learning analytics to provide a more structured way for teachers to makesense of dialogue information. This approach is widely used in the learning environment toincrease students’ collaborative discourse engagement (Celepkolu et al., 2022) and foster
diverse students’ science learning (Ryoo & Bedell, 2019). The Classroom Discourse
Analyzer (CDA) is this kind of discourse visualization platform (Chen, 2020), as shown in
Figure 1. The CDA comprises dynamic, interactive statistical graphs that represent talkmoves between teachers and students (Chen, 2020; Chen & Chan, 2022) and social
network analysis that models classroom interaction (Mameli et al., 2015; Wolf et al., 2022).

512

Figure 1. The Classroom Discourse Analyzer’s (CDA) Visualization of the Whole-class
Talk in a 35-min Lesson.
Dynamic interactive statistical graphs use bubbles, tables, and statistics to showclassroom discourse information, as shown on the left side of Figure 1. This information
includes the number of words, turns, and student-teacher talk patterns (Chen, 2020). Eachbubble in the graph represents a participant’s talk, and the size of the bubble corresponds tothe number of the participant’s words spoken. The picture in the lower left corner of Figure 1
illustrates turn-taking among participants and represents the number of student-student andteacher-student talks in the real classroom setting.
Social network analysis identifies overall discursive interaction and individual
contribution in regular classroom activities, as shown on the right side of Figure 1. In theseanalyses, ties indicate the quantity of network participants, and relationships show theamount of messages sent and received by each network member (Bokhove, 2016). Network
density is a global metric that represents the total number of possible ties and the overall
tightness of the network. Generally, the higher the frequency of interaction, the higher thedensity of the network. The nodal degree involves in-degree ties (the quantity of messages
received from others) and out-degree ties (the quantity of messages sent to others) (Mameli
et al., 2015). The degree of centrality measures the relevance of a network participant's
amount of dialogues taking place in a classroom (Bokhove, 2016).

2.3 Dialogic Engagement
Learner engagement refers to whether and how students respond to the learning
environment, which correlates with academic performance, persistence, and satisfaction(Bergdahl, 2022; Yang et al., 2022). Dialogic engagement is a critical component ofclassroom engagement that indicates whether learners actively participate in classroom
instruction and exchange ideas with peers or teachers. Analyzing discourse could contributeto understanding students’ classroom engagement. Therefore, this study views studentstudent and student-teacher dialogue as an indicator to reveal classroom engagement.
Social network analysis is widely used to explore classroom interaction. For example,
Bokhove (2016) proposed dynamic social network analysis (SNA) to model classroom

513

interaction through video and transcript data, representing the changes in overall interaction
patterns over time. Mameli et al. (2015) utilized a similar approach to identify the discursive
patterns of each classroom activity. Previous studies have shown that the single social
network presents the overall interaction of the classroom, and adding temporal analysis canreveal the classroom participation of individual learners. Thus, this study attempts to utilize
social network analysis to uncover the overall and individual dialogic engagement over time.

3. Methods
3.1 ParticipantsThis study was conducted in a second-grade primary classroom in Hong Kong, China, withthe participation of 20 students and their teacher. The teacher holds a master’s degree andis a senior teacher with seven years of teaching experience. Prior to the official launch of theintervention, we informed students’ parents of the interventional plan and obtained theirconsent. The classroom consisted of 7 boys and 13 girls with an average age of eight years.
The teaching content focused on learning basic programming concepts through graphicaloperations, such as sequences, loops, control, and events.

3.2 Design of the Intervention
The intervention was divided into two stages, as shown in Figure 2. Prior to the intervention,
we conducted two workshops to train teachers on the use of APT and CDA. This guaranteedthat the teacher could proficiently use APT as the discourse framework during the class andgain basic knowledge of CDA used by teacher-research collaboration before each class. In
the follow-up stage, we carried out three sessions of the intervention over the course of three
weeks. During each session, we used video cameras to record the complete classroom forapproximately 35 minutes per lesson. We then used CDA to analyze and visualize
classroom dialogue interaction by converting the audio from the videotape to text. Based on
the CDA analysis, we co-designed a plan for instruction improvement, such as providing
opportunities for students who spoke less during the previous class or adding somediscussion questions to guide students thinking.

Figure 2. Design of the Intervention.

3.3 Data Collection and Analysis
This study aimed to investigate the impact of using CDA as a teaching assistant tool tofacilitate students’ dialogic engagement. To achieve this goal, video data of the instructor’steaching were utilized as the primary data source of analysis. The analysis process involved
two main steps. First, audio data from the videotape was transcribed into text data with thetimestamp, teacher ID, and student ID, using the transcription tool iflyrec. Second, the text
data were converted into SNA network data using Python packages such as numpy andnetworkx, following the methodology outlined in Bokhove (2016) and Mameli et al. (2015)
research. In this approach, the sender is considered the source node, the receiver is thetarget node, and the word number of utterance is the weight of the edge. For example, if514

student A spoke to student B for 20 words, this would correspond to the data in SNA as
source node A, target node B, and the weight of the edge as 20. Additionally, when theteacher speaks to the whole class, the teacher is regarded as the source node, and all thestudents in the class are the target nodes. The class responds as a group to the teacher in asimilar manner. Based on the SNA data, we examined the dialogic engagement of students
as a whole and also analyzed the dialogic engagement of the individual learner.
Six subject-specific tests provided by the teacher were utilized to measure thelearning performance of students. We administered a pre-test and a post-test before andafter the intervention, respectively, with each test being completed in approximately 10
minutes. To facilitate ease of calculation, we converted test scores to percentages. A pairedsample t-test was employed to determine if any differences existed between the pre- andpost-test results.

4. Results
4.1 Overall Dialogic Engagement
Table 1 represents the overall dialogic engagement in the classroom, revealing that theteacher (node 22) was the central node in all three lessons. As the intervention progressed,
classroom dialogue interactions gradually increased, as evidenced by the visual graphs andnetwork data, where the network density significantly increased from 4.76% to 30.52%, andthe number of arcs grew from 22 to 141. In the first lesson, only 22 talk turns were
established in the entire class, while there was a slight increase in the second lesson. In thethird lesson, the number of classroom dialogue interactions increased to 141, indicating
more opportunities for student-student and student-teacher dialogue. The core nodes
correspondingly added student node 7 and node 18.
Table 1. Networks and Basic Metrics of Overall Dialogic Engagement in the Classroom
Lesson 1

Lesson 2

Lesson 3

Visualization

Number of
22
22
22
Nodea
Number of
22
30
141
arcs
Network
4.76%
10.82%
30.52%
densityCore nodes
node 22
node 22
nodes 22, 7, 18
a. Nodes include node 21 for ‘entire class’ and node 22 for the teacher. Node labels from 1 to 20
represent student IDs.
In Figure 3, we observe the relationship between the out-degree of node 22 (teacher)and network density. The out-degree of node 22 refers to the number of times the teacher
speaks to the students. Interestingly, Figure 3 indicates an inverse relationship between theout-degree of node 22 and network density. Specifically, as the number of times the teacher
speaks to the students decreases, the overall network density gradually increases. In thefirst lesson, the teacher spoke 75 times, and the overall network was relatively sparse with adensity of 4.76%; in the second lesson, the teacher spoke 59 times, and the network density

515

increased to 10.82%; And in the third lesson, the number of teacher explanations decreased
to 48, while the network density increased to 30.52%.
75

80
70
60
50
40
30
20
10
0

35.00%
30.52% 30.00%

59
48

25.00%
20.00%
15.00%

10.82%

10.00%

5.00%

4.76%

0.00%
Lesson 1

Lesson 2

Lesson 3

Out-degree of Node 22 (teacher talk)Density of network

Figure 3. Out Degree of Node 22 (Teacher) and Network Density.

4.2 Individual Dialogic Engagement
Figure 4 displays individual students’ dialogic engagement in the classroom during the threelessons. In the first lesson, the average number of times students spoke was 2.5, and the
maximum number of times a student spoke was 8 (node 9). Six students did not participate
in the classroom dialogue (e.g., nodes 3, 5, 10), and three students participated in only one
class talk (e.g., nodes 4, 7, 13). In the second lesson, the average number of times students
spoke was slightly lower at 2.35, and the maximum number of times a student spoke was 8(node 1). Seven students did not speak in class (e.g., nodes 7, 10, 13, 14), and three
students spoke only once (e.g., nodes 3, 6, 19). In the third lesson, the average number oftimes the students spoke was 7.3, and the maximum number of times a student spoke was
20 (node 7). Only one student did not speak (node 18), and three students only spoke once
(e.g., nodes 5, 14, 15).

Degree of node

25
20
15
10
5
0

8

11

18

4

7 5
6
2

6

20 14
1

3

1

7

5

12

1
3

6 16

1
1
8
6 5 7
3 7
2
6
6
5
4
2
2
2 3
1 3
1 2
1 1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

4 4Nodes (Student ID)
Lesson 1

Lesson 2

Lesson 3

Figure 4. Individual Dialogic Engagement in the Classroom.
Figure 4 also reveals slight differences in each student’s engagement in classroom
dialogue during the three lessons. The participation of some students in classroom dialogue
gradually increased (10 students, including nodes 3, 4, 7, 8, 10, 11, 13, 16, 17, 20), whilesome students’ participation in classroom dialogues decreased (5 students, including nodes
1, 5, 14, 15, 18). Some students’ participation in classroom dialogues fluctuated to a certain

516

extent (4 students, including nodes 2, 6, 9, 19), while others remained relatively constant (1student, node 12).

4.3 Learning Performance
Table 2 illustrates the statistical outcomes of overall learning performance, including the pretest (M = 62.28, SD = 16.08) and the post-test (M = 82.35, SD = 13.75). Results from thepaired-sample t-test indicated a significant difference (t = -4.697, p <. 001) in learning
performance before and after the intervention, with the post-test scores significantly higher
than the pre-test scores.
Table 2. Paired-Samples t-test of the Pre- and Post-test Learning Performancen20
20

Pre-test
Post-test

Mean
62.28
82.35

SD
16.08
13.75

t
-4.697p<.001

Figure 5 presents the individual learning performance of students, with more thanhalf of the students demonstrating higher scores on the post-test than the pre-test. When
dividing students into low- and high-engagement groups based on the mean value of
dialogic engagement during the three sessions, the analysis results revealed that thelearning performance of the students in the high-engagement group (n = 9, M = 84.97) was
slightly higher than that of the students in the low-engagement group (n = 11, M = 80.21);
however, there was no statistically significant difference in learning performance between
the two groups (t = -.76, p = .457).
100.00
90.00
80.00
70.00
60.00
50.00
40.00
30.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Pre-test

Post-test

Figure 5. Individual Learning Performance of Students.

5. Discussion and Conclusion
The present study aimed to investigate the impact of visual learning analytics as a teacher
assistance tool on students’ engagement in classroom discourse. While previous studies
have demonstrated that dialogic frameworks or analytics-based tools can benefit classroom
instruction (Chen, 2020; Chen & Chan, 2022), there remains a lack of effective scaffoldings
to support teachers’ improvement regarding talk-based classroom instruction. In this study,
we introduced a richer classroom dialogue analyzer that included interactive statistical andsocial network graphs that served as a teacher assistance tool to visualize classroom
dialogic progress information and provide evidence to support the improvement of classroom
instruction. Our findings from social network analyses indicated that visualization-based
support contributed to both overall dialogic engagement and individual student engagement.
Furthermore, subsequent tests found that students’ academic performance also increased.

517

From the perspective of overall classroom engagement, we observed that the lessthe teacher spoke, the more students expressed their thoughts during lessons, as indicatedby the growth of network density. The visual graphs may have helped the teacher become
more aware of her speech and understand that a high level of teacher talk meant that
opportunities for students to speak were more limited. Consequently, she consciously used
APT to guide students to think and speak in subsequent classes (Major, Smørdal, et al.,2022; Major, Warwick, et al., 2018). Similar findings can be reported in van der Veen et al.
(2017) and Chen et al. (2020), where teachers consciously encouraged and supported
student talk, leading to improved communicative performance. The visualization-based tool
can further guide teachers and researchers in negotiating how to improve classroom
instruction and provide targeted support. To a certain extent, the tool responds to the needfor evidence-based support for classroom instruction (Urhahne et al., 2010). As
Amarasinghe et al. (2022) have suggested, designing and orchestrating daily classroom
scenarios is a demanding task, and advanced technologies, such as learning analytics ordata mining, can assist teachers in tackling the complexity involved in their daily work.
Individual student’s dialogic engagement improved, with over half of the students
demonstrating more engagement in classroom talk during the three lessons, as indicated by
the out-degree of students’ nodes. However, it should be noted that, given the fixed lesson
time, some students may have multiple opportunities to speak, leading to fewer opportunities
for others. This highlights the need to consider adding other technical tools to support
students’ talks when designing activities. For example, online discussion forums could
provide students with opportunities to express their ideas (e.g., Ratan et al., 2022);
additionally, an adaptive learning system could be incorporated to respond to individual
learning needs (e.g., Wu et al., 2017).
Our analysis of learning performance data revealed a statistically significant increasein students’ academic achievement from pre- to post-test stages. We also compared student
groups with varying levels of dialogic engagement and found that the high-engagement
group slightly outperformed the low-engagement group, although this difference was notstatistically significant. We will further investigate the relationship between students’ dialogic
engagement and their learning performance by considering students’ other characteristics
and learning preferences (Bond & Bedenlier, 2019; Halverson & Graham, 2019). In doing so,
we can help teachers obtain a better understanding of students’ engagement in classroom
dialogue and how this relates to their academic achievement.

6. Limitations and Future Work
This study investigated the influence of teachers (supported by a discourse-based visualtool) on students’ dialogic engagement in the classroom. The findings indicated that thisapproach has the potential to foster students’ dialogic engagement. However, as this is an
ongoing project, there are some limitations to the current study. First, the three-weekduration of the intervention made it difficult for us to estimate its long-term impact on student
classroom participation. In the coming months, we will continue to conduct extended
interventions to explore the impact of visualization tools on both teachers and students.
Second, the lack of a comparison group and the small scale of the intervention may limit thegeneralizability of the research findings. In future research, we will scale up the intervention
and explore the impact of discourse-based visualization on both teachers and students.
Third, we focused on data sources derived from classroom videos and achievement tests,
without considering data sources regarding changes in the participation processes ofstudents or teachers. Consequently, we were unable to conduct a thorough analysis of the
impact of visual support on both teachers and students throughout the intervention. In thenext stage of the project, we will collect additional sources of data, such as teachers’
reflective logs, classroom observation notes, and interviews with both teachers and students.
These additional data sources will provide a more comprehensive understanding of theintervention and improve the accuracy of our overall conclusions.

518

Acknowledgements
This work was supported by Hong Kong Research Grants Council, University Grants
Committee (Grant No.: 17605221) and by the Innovation and Technology Commission of theGovernment of the HKSAR (Grant No.: ITB/FBL/7026/20/P).

References
Amarasinghe, I., Michos, K., Crespi, F., & Hernandez-Leo, D. (2022). Learning analytics support toteachers' design and orchestrating tasks. Journal of Computer Assisted Learning, 16.
https://doi.org/10.1111/jcal.12711
Barak, M., & Lefstein, A. (2021). Opening Texts for Discussion: Developing Dialogic Reading Stances.
Reading Research Quarterly, 57(2), 449-468. https://doi.org/10.1002/rrq.413
Bergdahl, N. (2022). Engagement and disengagement in online learning. Computers & Education,
188, 104561. https://doi.org/10.1016/j.compedu.2022.104561
Bokhove, C. (2016). Exploring classroom interaction with dynamic social network analysis.
International Journal of Research & Method in Education, 41(1), 17-37.
https://doi.org/10.1080/1743727x.2016.1192116
Bond, M., & Bedenlier, S. (2019). Facilitating Student Engagement Through Educational Technology:
Towards a Conceptual Framework. Journal of Interactive Media in Education, 2019(1).
https://doi.org/10.5334/jime.528
Celepkolu, M., Galdo, A. C., & Boyer, K. E. (2022). Fostering Balanced Contributions Among Children
Through Dialogue Visualization. Ieee Transactions on Learning Technologies, 15(4), 454-466.
https://doi.org/10.1109/tlt.2022.3193648
Chen, G. (2020). A visual learning analytics (VLA) approach to video-based teacher professional
development: Impact on teachers’ beliefs, self-efficacy, and classroom talk practice.
Computers & Education, 144. https://doi.org/10.1016/j.compedu.2019.103670
Chen, G., & Chan, C. K. K. (2022). Visualization- and analytics-supported video-based professional
development for promoting mathematics classroom discourse. Learning, Culture and Social
Interaction, 33. https://doi.org/10.1016/j.lcsi.2022.100609
Chen, G., Chan, C. K. K., Chan, K. K. H., Clarke, S. N., & Resnick, L. B. (2020). Efficacy of videobased teacher professional development for increasing classroom discourse and student
learning. Journal of the Learning Sciences, 29(4-5), 642-680.
https://doi.org/10.1080/10508406.2020.1783269
Downer, J. T., Rimm-Kaufman, S. E., & Pianta, R. C. (2007). How Do Classroom Conditions andChildren's Risk for School Problems Contribute to Children's Behavioral Engagement inLearning? School Psychology Review, 36(3).
https://doi.org/10.1080/02796015.2007.12087938
Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards Collaboration
Translucence: Giving Meaning to Multimodal Group Data 濁 Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems,
https://doi.org/10.1145/3290605.3300269
Flanders, N. A. (1964). Interaction Analysis in the Classroom: A Manual for Observers. School of
Education, University of Michigan. https://books.google.com.hk/books?id=DN9EAAAAYAAJ
Halverson, L. R., & Graham, C. R. (2019). Learner Engagement in Blended Learning Environments: A
Conceptual Framework. Online Learning, 23(2). https://doi.org/10.24059/olj.v23i2.1481
Hsiao, I. H., & Lin, Y.-L. (2017). Enriching programming content semantics: An evaluation of visual
analytics approach. Computers in Human Behavior, 72, 771-782.
https://doi.org/10.1016/j.chb.2016.10.012
Khong, T. D. H., Saito, E., & Gillies, R. M. (2017). Key issues in productive classroom talk andinterventions. Educational Review, 71(3), 334-349.
https://doi.org/10.1080/00131911.2017.1410105
Major, L., Smørdal, O., Warwick, P., Rasmussen, I., Cook, V., & Vrikki, M. (2022). Investigating digitaltechnology’s role in supporting classroom dialogue: integrating enacted affordance into
analysis across a complex dataset. International Journal of Research & Method in Education,
1-19. https://doi.org/10.1080/1743727x.2022.2032632
Major, L., Warwick, P., Rasmussen, I., Ludvigsen, S., & Cook, V. (2018). Classroom dialogue anddigital technologies: A scoping review. Education and Information Technologies, 23(5), 19952028. https://doi.org/10.1007/s10639-018-9701-y

519

Mameli, C., Mazzoni, E., & Molinari, L. (2015). Patterns of discursive interactions in primaryclassrooms: an application of social network analysis. Research Papers in Education, 30(5),546-566. https://doi.org/10.1080/02671522.2015.1027727
Mavrikis, M., Geraniou, E., Santos, S. G., & Poulovassilis, A. (2019). Intelligent analysis and datavisualisation for teacher assistance tools: The case of exploratory learning. British Journal of
Educational Technology, 50(6), 2920-2942. https://doi.org/10.1111/bjet.12876
Michaels, S., & O'Connor, C. (2015). Conceptualizing Talk Moves as Tools: Professional
Development Approaches for Academically Productive Discussions. In Socializing IntelligenceThrough Academic Talk and Dialogue (pp. 347-361). https://doi.org/10.3102/978-0-93530243-1_27
O’Connor, C., & Michaels, S. (2019). Supporting teachers in taking up productive talk moves: Thelong road to professional learning at scale. International Journal of Educational Research, 97,
166-175. https://doi.org/10.1016/j.ijer.2017.11.003
Ratan, R., Ucha, C., Lei, Y., Lim, C., Triwibowo, W., Yelon, S., . . . Hua Chen, V. H. (2022). How dosocial presence and active learning in synchronous and asynchronous online classes relate tostudents’ perceived course gains? Computers & Education, 191, 104621.
https://doi.org/10.1016/j.compedu.2022.104621
Ryoo, K., & Bedell, K. (2019). Supporting linguistically diverse students' science learning with dynamic
visualizations through discourse-rich practices. Journal of Research in Science Teaching,56(3), 270-301. https://doi.org/10.1002/tea.21519
Saunders-Stewart, K. S., Gyles, P. D. T., & Shore, B. M. (2012). Student Outcomes in Inquiry
Instruction. Journal of Advanced Academics, 23(1), 5-31.
https://doi.org/10.1177/1932202x11429860
Urhahne, D., Schanze, S., Bell, T., Mansfield, A., & Holmes, J. (2009). Role of the Teacher in
ComputerƮsupported Collaborative Inquiry Learning. International Journal of Science
Education, 32(2), 221-243. https://doi.org/10.1080/09500690802516967
Urhahne, D., Schanze, S., Bell, T., Mansfield, A., & Holmes, J. (2010). Role of the Teacher in
Computer-supported Collaborative Inquiry Learning. International Journal of Science
Education, 32(2), 221-243. https://doi.org/10.1080/09500690802516967
van der Veen, C., de Mey, L., van Kruistum, C., & van Oers, B. (2017). The effect of productiveclassroom talk and metacommunication on young children’s oral communicative competence
and subject matter knowledge: An intervention study in early childhood education. Learning
and Instruction, 48, 14-22. https://doi.org/10.1016/j.learninstruc.2016.06.001
Vetter, A., Schieble, M., & Martin, K. M. (2021). Critical talk moves in critical conversations: examining
power and privilege in an English Language Arts classroom. English in Education, 55(4), 313336. https://doi.org/10.1080/04250494.2020.1848351
Wolery, M., & Hemmeter, M. L. (2011). Classroom Instruction: Background, Assumptions, andChallenges. Journal of early intervention, 33(4), 371-380.
https://doi.org/10.1177/1053815111429119
Wolf, S. F., Sault, T. M., Suda, T., & Traxler, A. L. (2022). Social network development in classrooms.
Applied Network Science, 7(1). https://doi.org/10.1007/s41109-022-00465-z
Wu, C.-H., Chen, Y.-S., & Chen, T.-g. (2017). An Adaptive e-Learning System for Enhancing Learning
Performance: Based on Dynamic Scaffolding Theory. EURASIA Journal of Mathematics,
Science and Technology Education, 14(3), 903-913. https://doi.org/10.12973/ejmste/81061
Yang, Y. Q., Yuan, K. C., Feng, X. Q., Li, X. H., & van Aalst, J. (2022). Fostering low-achieving
students' productive disciplinary engagement through knowledge-building inquiry andreflective assessment. British Journal of Educational Technology, 19.
https://doi.org/10.1111/bjet.13267

520

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Advancing Education through Stakeholder
Engagement: An Evaluation of the
Learning Butler Chatbot's Impact on
Instructors, and Learners
Chih-Yang PENGa, Su-Hang YANGb, Pei-Yu HOa, Jen-Hang WANGc* &Gwo-Dong CHENaa Department of Computer Science and Information Engineering, National Central University, Taiwanb Department of Hospitality Management, Chien Hsin University of Science and Technology, Taiwan
c Research Center for Science and Technology for Learning, National Central University, Taiwan*harry@cl.ncu.edu.tw
Abstract: This study aims to address the limitation of existing chatbots in education,
which are designed only for learners and do not account for the needs of instructors.
To solve this problem, the study proposes a learning chatbot called the "Learning
Butler," which integrates resources for optimal learning outcomes and is developed
based on the situational theory for mobile and classroom learning environments. The
effectiveness of the Learning Butler in improving learning outcomes was evaluatedthrough an experiment conducted at a university of science and technology. The results
demonstrated that the Learning Butler significantly improved learners' performancecompared to the general chatbot, and learners and instructors provided positive
feedback on its ability to offer instant suggestions and feedback during dialogues andinteractions. This research contributes to the development of effective educational
technology by exploring the potential of chatbots in enhancing learning outcomes, withthe ultimate goal of improving the quality of education through the application ofadvanced technologies and innovative pedagogical approaches.
Keywords: Chatbot in education, Social chatbot, Situational learning, Mobile situatedlearning, Butler

1. Introduction
1.1 Background
Chatbots have become increasingly popular in education due to their ability to integrateteaching content, provide real-time assistance, and enhance learning engagement (Okonkwo& Ade-Ibijola, 2021). However, most existing chatbots for education are designed only forlearners and do not account for the needs of other stakeholders, such as instructors, which
can limit their effectiveness. In addition, based on the situated learning theory, throughdialogue interaction with roles shaped in physical or digital ways, instructors can overcome
the limitations of time and space, and provide learners with real situations to apply theirclassroom knowledge in a low-cost and convenient way (Dede, 2009).
To address this limitation, this study proposes a learning chatbot called the "Learning
Butler," modeled after the role of a butler, to serve as a communication bridge among
stakeholders. Through the Learning Butler, learners can receive assistance with time
management, emotional management, and encouragement to achieve learning goals.

1.2 Research Objectives
In this study, our main objective is to develop a learning dialogue chatbot, i.e., the Learning
Butler, designed as a butler role for learning and modeled after the situated learning theory.

521

We aim to propose design methods for a learning chatbot that can serve as a communication
bridge among stakeholders, integrate various resources, and achieve optimal learning
outcomes. The ultimate goal is to enhance the quality of education through the application ofadvanced technologies and innovative pedagogical approaches.

1.3 Research Questions
To achieve the research goals, the research questions are designed to address the following
inquiries: (1) Does the learning butler chatbot lead to better improvements in learners' learningachievements compared to a general chatbot? (2) Does the learning butler chatbot enhancelearners' learning motivations and sense of responsibility to a better extent than a generalchatbot?

2. Related Work
2.1 Application of Chatbot in Education
Chatbots have been recognized as a promising tool for improving education by addressing
the problem of insufficient manpower (Winkler & Söllner, 2018). Additionally, chatbots havebeen found to be helpful in language learning, especially for beginners (Yin & Satar, 2020).
Pham et al. (2018) designed a chatbot that assisted English language learners with generalgreetings, answering specific user requests, providing tips for learning content, and sending
reminders to users. However, maintaining learners' enthusiasm for learning has been
identified as a challenge in educational chatbot applications (Fryer et al., 2017). Furthermore,
research has indicated that the inability to express understanding to users and the lack ofemotion in chatbots may hinder their effectiveness in teaching (Gallacher et al., 2018).

2.2 Situational Learning
Situational Learning emphasizes that learners should acquire knowledge and skills in applied
contexts, based on the concept of Situated Cognition proposed by Brown et al. (1989). Dede(2009) suggested that virtual environments can be utilized to optimize the implementation ofsituational learning by simulating learning situations that are not easily accessible in the realworld. Cognitive Immersive Language Learning Environment (CILLE) is an example of a virtualenvironment that combines artificial intelligence with augmented reality technology, allowing
learners to learn Chinese by interacting with AI agents in a situational learning environment(Divekar et al., 2021). In contrast, Huang et al.'s (2016) study showed that using mobile
devices for situational learning resulted in better learning outcomes. Furthermore, somestudies suggest that mobile learning has a lasting impact on learners' learning outcomes andcan induce positive emotions such as excitement and happiness during the learning process
for up to six months after the experiment (Demir & Akpinar, 2018).

2.3 Butler and Social Chatbot
Social chatbots have the potential to establish an emotional connection with users and provide
a sense of social belonging, which can enhance users' engagement in longer and deeper
conversations with the chatbot (Zhou et al., 2020). While social chatbots have been applied invarious fields, their potential in the education field, particularly as companions and emotional
support during learning, remains largely unexplored (Liu et al., 2022).
On the other hand, butlers, in general, are known for their excellent communication
skills, organizational abilities, discretion, attention to detail, and active listening skills. These
qualities make butlers suitable for supervision, companionship, and learning assistance, yet
few studies have applied the butler model to design chatbots for the education field. To fill thisgap, this study proposes a learning butler chatbot designed to assist learners with time and522

emotion management, step-by-step completion of in-class and after-class exercises, andsituational learning environments.

3. System Design
3.1 System Structure
The system structure of the learning butler chatbot comprises two platforms: the classroom
situational learning system and the mobile situational learning platform. The chatbot provides
learners with eight functions through its user interface, including time management, emotion
management, positive reinforcement, learning goal prompts, multiple-branch scripts, dubbing
management, manual evaluation, and a reward mechanism (Figure 1). These functions areintended to create a positive and supportive learning environment by providing learners withemotional support, time management, and rehearsal opportunities. Learners can interact withthe chatbot through the classroom situational learning system, or interact with the practicewebsite and learning butler chatbot via their mobile devices, enabling them to practice andreview what they have learned at any time and place.

Figure 1. System structure.
The general chatbot, in contrast, only has basic dialogue functionality, limiting it to threefunctions: multiple situational scripts, manual evaluation, and a reward mechanism. For theother five functions, instructors provide support by reminding learners of homework before
class, actively understanding learners' emotions, giving verbal encouragement to the wholeclass, informing learners of learning goals before the class starts, and changing the tone ofvoice promptly according to learning content or learners' situations. The multiple-branchscripts used in general chatbots only provide situational information through text and audio
files, without situational information provided by pictures such as comics, and without the initial
evaluation provided by the chatbot.

3.2 Implementation
The implementation of the classroom situational learning system is based on a digital learning
theater design (Hu, 2021) and includes both a learning butler chatbot and a general chatbot.
To enable effective rehearsal, multiple branch scripts have been added. Additionally, andMicrosoft Azure's speech-to-text service is used for preliminary assessment by the learning
butler chatbot. Customized features in the voice assistant's database provide reminders,positive reinforcement, emotional support, and feedback to reduce anxiety and enhancelearners' confidence during rehearsal.
Regarding the mobile situational learning platform, respectively including the generalchatbot and the learning butler chatbot. After selecting the script, learners are briefed on the523

learning butler chatbot or general chatbot rehearsal processes. When entering the scenario
rehearsal, the learning butler chatbot first informs the learner of the learning goals and theupcoming learning schedule, then provides positive reinforcement, scenario comics,
demonstration audio files, and allows learners to record specific lines. Following the scenario
rehearsal, the learning butler chatbot offers emotional support to learners (Figure 2).

Figure 2. The learning butler chatbot reminds learning goals and provides positive
reinforcement. Also, the scenario comics, and demonstration audio files are provided.

4. Experiment
4.1 Experimental Subject
In this study, a total of 60 undergraduate students in a Hospitality Japanese course at a local
university of science and technology were recruited. The students were randomly assigned toeither the experimental or control group, with 30 learners in each group. The experimentalgroup consisted of 16 males and 14 females, while the control group was composed of 10
males and 20 females. Throughout the experiment, the two groups remained completely
independent of each other and had no impact on each other's performance. The learningmaterial for the system is a restaurant script compiled in Japanese with the assistance of the
instructor of the Hospitality Japanese course.

4.2 Experimental Process

Figure 3. Experimental Process
The experimental process (Figure 3) spanned six weeks and consisted of several stages. A
pre-test was conducted before the experiment began. Basic knowledge teaching took placein the first week to establish a solid foundation. In the second and third weeks, script teaching524

was conducted, and learners used the mobile situational learning platform. The experimentalgroup used the learning butler chatbot, while the control group used the general chatbot. In
the fourth and fifth weeks, learners rehearsed their presentation using the classroom
situational learning system. The experimental group used the learning butler chatbot, and thecontrol group used the general chatbot. In the final week, learners presented theirachievements using the classroom situational learning system. At the end of the experiment,
all learners completed a post-test and questionnaires. Some learners were selected forinterviews to gain deeper insights into their experience.

4.3 Research Instrument
To evaluate the effectiveness of the learning butler chatbot, the study employed a pre-test anda post-test before and after the experiment, as well as a questionnaire administered during
the post-test. The tests were developed in collaboration with the Hospitality Japanese course
instructor and included multiple-choice, true/false, and fill-in-the-blank questions related to thecourse content, with a maximum score of 100. The pre-test aimed to gauge learners' priorknowledge, while the post-test aimed to measure the impact of different learning chatbots onlearning effectiveness, analyzed using ANCOVA. In addition to the tests, a Likert five-point
scale questionnaire was used to gather data on learners' perceptions of the learning butler
chatbot. The questionnaire included 17 questions covering five dimensions, four of whichmeasured the elements proposed by the ARCS motivational model: attention, relevance,
confidence, and satisfaction (Keller, 1987). The fifth dimension assessed learning
responsibility and was developed by the researchers after review by experts.

5. Result and Discussion
ANCOVA was employed to compare the experimental and control groups in post-test scoreswith pre-test as covariate. The results show that the average and adjusted average scores ofthe experimental group are 82.40 and 81.70, respectively, while those of the control group are
72.16 and 72.85, respectively. Furthermore, there is a significant difference between the twogroups (F = 6.255, p = .015 < .05). The partial eta square value was .099, with a range
from .058 to .138, which indicates the effect size is at a medium level (Cohen, 1992). The data
indicate that using a learning butler chatbot can significantly improve learners' performancecompared to a general chatbot.
The questionnaire, with a Cronbach's Alpha score of 0.986, indicated high reliability.
The results of independent sample t-tests revealed no significant differences between theexperimental and control groups, implying that the use of a learning butler chatbot did notaffect learners' motivation and learning responsibility.
This study conducted follow-up interviews with four learners and the instructor who
participated in the experiment. The majority of learners believed that the learning butler
chatbot played a supervisory role and increased their interest in learning. However, a small
number of learners felt that general chatbots helped them enter a state of focused learning,which may explain why no significant differences were shown in the five dimensions of the
post-questionnaire. Regarding the instructor, she emphasized the unique feature of thelearning butler chatbot, which is its ability to determine whether learners are using the correct
tone and style of Japanese during dialogues and interactions, and offer instant suggestions
and feedback.

6. Conclusion
This study introduces a new learning approach using a learning butler chatbot on mobile andclassroom situational learning platforms. It integrates resources, enhances communication
efficiency, and enables real-time understanding of learners' needs for personalized andadaptive learning. Learners benefit from time and emotion management and a situational

525

learning environment, while instructors can conveniently plan, track progress, and provide
individualized support. The results of this study show that using a learning butler chatbot
significantly improves learning performance without negatively impacting learners' motivation
and responsibility. Additionally, the majority of the interviewed learners and instructors
provided positive feedback.
For future research, psychological theories and user experience should be considered
to meet different learning needs, and more interactions should be integrated, such as adding
texts and images to the mobile situational learning platform or incorporating facial recognition
into the classroom situational learning system to provide more interactive experiences.

Acknowledgements
This study is supported by the National Science and Technology Council, Taiwan.
Grand Number: MOST 111-2410-H-008-012-MY3 and NSTC 112-2811-H-008-006.

References
Hu, L. L. (2021). Design and produce a digital learning theater software architecture that is easy tocontinue to evolve [Unpublished master’s thesis]. National Central University, Taiwan.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational
Researcher, 18(1), 32-42.
Dede, C. (2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66-69.
Demir, K., & Akpinar, E. (2018). The Effect of Mobile Learning Applications on Students' Academic
Achievement and Attitudes toward Mobile Learning. Malaysian Online Journal of Educational
Technology, 6(2), 48-59.
Divekar, R. R., Drozdal, J., Chabot, S., Zhou, Y., Su, H., & Chen, Y. & Braasch, J. (2021). Foreign
language acquisition via artificial intelligence and extended reality: Design and evaluation.
Computer Assisted Language Learning, 1-29.
Gallacher, A., Thompson, A., Howarth, M., Taalas, P., Jalkanen, J., Bradley, L., & Thouësny, S. (2018).
“My robot is an idiot!”-students’ perceptions of AI in the L2 classroom. Future-proof CALL: language
learning as exploration and encounters-short papers from EUROCALL, 70-76.
Huang, C. S., Yang, S. J., Chiang, T. H., & Su, A. Y. (2016). Effects of situated mobile learning approachon learning motivation and performance of EFL learners. Journal of Educational Technology &Society, 19(1), 263-276.
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of
Instructional Development, 10(3), 2-10
Liu, C. C., Liao, M. G., Chang, C. H., & Lin, H. M. (2022). An analysis of children’ interaction with an AI
chatbot and its impact on their interest in reading. Computers & Education, 104576.
Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review.
Computers and Education: Artificial Intelligence, 2, 100033.
Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., & Cao, T. T. H. (2018, November). Chatbot as
an intelligent personal assistant for mobile language learning. In Proceedings of the 2018 2ndInternational Conference on Education and E-Learning (pp. 16-21).
Winkler, R., & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the-art
analysis. In Academy of Management Annual Meeting (AOM).
Yin, Q., & Satar, M. (2020). English as a foreign language learner interactions with chatbots: Negotiation
for meaning. International Online Journal of Education and Teaching (IOJET).
Zhou, L., Gao, J., Li, D., & Shum, H. Y. (2020). The design and implementation of xiaoice, an
empathetic social chatbot. Computational Linguistics, 46(1), 53-93.
Cohen, J. (1992). Statistical power analysis. Current directions in psychological science, 1(3), 98-101.

526

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31 st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Building Students’ Learning Habits on
Slack: An Application of the IDC Theory
Veenita SHAHa Sahana MURTHYa Sridhar IYERa
IDP in Educational Technology, IIT Bombay, India*veenita.shah@iitb.ac.in
Abstract: The integration of ubiquitous technologies with classroom interactions has
the potential to foster positive learning habits in students. The instructors at
Educational Technology Program in IIT Bombay tackled the challenge of remotelearning during the COVID-19 lockdown by conceptualizing the use of the Slackplatform to promote learning habits among their graduate students. Grounded in the
IDC theory specifically developed for the Asian population, the design of thispedagogical intervention aimed to promote students’ habit of conveying complex
ideas in an accessible manner through two learning activities aligned with theirinterests. Conversation analysis was used to examine instructors' actions to develop
learning habits in students, and member check interviews were conducted with
instructors to verify the pedagogical approach and findings. The study demonstrated
how the instructors' pedagogical intervention on the Slack platform supported thedevelopment of learning habits among graduate students. The results exhibited themapping of individual components of the habit loop, i.e., cuing, routine and harmony
with the design and outcome of the approach. Students’ satisfaction with routine
activities and impatience for creations from the learning activities indicated harmony
which eventually leads to habit formation. The continual pursuit of learning activities
without instructor involvement also suggested habit activation. The present study
contributes to the current understanding of effective pedagogical practices to foster
good learning habits by leveraging ubiquitous learning tools.
Keywords: Learning habits, ubiquitous learning tools, Slack platform, IDC theory,
habit loop, interest-driven learning activities, graduate students

1. Introduction
Developing good learning habits is crucial for students to become lifelong learners andacquire essential competencies for the 21st century, such as critical thinking, self-regulated
learning, problem-solving, and collaborative skills (Chan et al., 2018). Hence, educational
institutions must prioritize these competencies to foster students' professional growth bypromoting the cultivation of good learning habits.
The Educational Technology (ET) Program at IIT Bombay in India encompasses
Masters and PhD students, postdoctoral scholars, academic staff and faculty members.
During the COVID-19 lockdown, the instructors in the ET Program at IIT Bombay faced thechallenge of enhancing learning without the ability for physical interactions. It was during thistime that the instructors conceived the idea of using Slack, a ubiquitous learning tool
supported by mobile and computers, to connect the team and promote learning. They
recognized the potential of this cloud-based instant messaging platform (Slack Technologies,
Inc., n.d.) to foster good learning habits among the PhD and MTech students. Slack isdesigned for team communication and collaboration, offering real-time channels, direct
messages, and calls. It also integrates a range of productivity and collaboration tools(Montrief et al., 2021), making it a versatile platform for remote learning.
The pedagogical design of this intervention was grounded in the interest-driven
creator (IDC) theory which was specifically developed in the context of the Asian population(Chan et al., 2018). The Asian education system and curriculum drives students to the habitof rote learning and achieving academic satisfaction with high scores in grade-level exams
1

The first author is now a part of the Tata ClassEdge Limited.

527

(Chan et al. 2018; Kirkpatrick & Zang 2011). Forming interest-driven or skill-based habits oflearning to promote professional growth of students have not been sufficiently considered informal education. Hence, the instructors in this study applied the IDC theory to encourage
positive learning habits among their graduate student community utilising the Slack platform.
They achieved this by incorporating creative activities that aligned with students' interests
into their daily learning routines. The Slack platform was pivotal in cultivating these habits instudents among a diverse team of individuals. With the aid of valuable feedback from both
instructors and peers, the platform facilitated increased engagement and led to marked
improvements in creative idea generation and effective communication of complex
information.
This study aims to demonstrate how Slack, as a ubiquitous learning tool, can be used
to promote positive learning habits in an academic context. The learning activities were
designed to develop the habit of conveying complex information in an accessible manner.
Specifically, these activities, entailing the creation and sharing of newsletters and memes,
involved fostering critical thinking abilities to break down complex concepts and ideas into
simpler, more manageable parts that can be easily understood by different audiences.
Hereby, we present a pedagogical approach for promoting learning habits among students
on Slack by implementing the IDC theory. This contributes to the existing knowledge ofeffective educational practices with ubiquitous learning tools.

2. Literature Review
2.1 Interest-Driven Creator Theory
The IDC theory consists of three anchored concepts: interest, creation and habit which areinterconnected in various ways (Chan et al., 2018). The theory assumes that learners, whendriven by interest, can be engaged in knowledge creation, and the repetition of suchprocesses in their daily learning routines will help in forming interest-driven creation habits
for them.
The concept of habit formation is an integral component of the IDC theory. The habit
loop suggests that habits are formed through a three-step process: cuing environment(arrangement of place, time, people, or incidents), routine (repetitive pattern of activities),and harmony (Chen et al., 2020). The theory posits that learning activities can be developed
as interest-driven creation activities through a rigorous design process that prioritizes thecultivation of students' interests. The creation activities must be sufficient to arouse curiosity,
engage students in the activities, and extend the scope of those activities in the pursuit of
personal interests. To ensure that these activities have lasting effects, the study proposesthat they should be incorporated into the daily routine so that students can develop a habit oflearning through daily engagement with these activities.
Several researchers have employed the IDC theory as a guiding framework to design
learning activities and curriculum in computer programming education (Kong et al., 2018;
Kong & Li, 2016) and mathematics course (Huang et al., 2020). A recent study highlighted
the implementation of an IDC experimental school in nurturing teachers' pedagogicalpractices and encouraging academic growth in students (Looi et al., 2023). An example ofan IDC-based learning activity intervention focused on habit development is the Modeled
Sustained Silent Reading (MSSR) (Wong et al., 2020). The main objective of MSSR was toassist students in developing a reading habit. The intervention involved exposing students toa "modeled" and "silent" environment repeatedly. Another recent study designed learning
activities using the IDC theory for students to develop an interest in podcast English
learning, be immersed in the learning process and, by repeating this process in their daily
routines, strengthen learning habits (Chen et al., 2022). The findings provided useful insights
for podcasters to invest in establishing learners’ interest factor and stimulating experiences
to improve their continuous learning intention. While the IDC theory has been utilized invarious applications that focus on interest and creation loops (Huang et al., 2020; Kong et528

al., 2019; Wong & Wong, 2020), there has been a lack of attention to the habit loop aspect of
the theory.

2.2 Habit Formation in Academia Using Ubiquitous Learning Tools
Instilling good learning habits in students is highly valued by teachers and parents for theirlong-term learning and development, as they significantly impact academic achievements
and future success (Chan et al., 2018; Ebele & Olofu, 2017). A habit is a pattern of behavior
that is repeated regularly and often happens unconsciously. In the context of this study, habit
is used to refer to mental processes which become evident as behavioural inclinations when
students are continuously interacting with tasks or actions of interest-driven learning (Chenet al., 2020).
Though there is enough research on the role of ubiquitous learning tools, including
social media in academia (Aljawarneh, 2020; Dontre, 2021), there is a lack of empiricalevidence on the correlation between these learning tools and academic habit formation.
Slack is a ubiquitous learning tool predominantly geared towards professional use inbusinesses and organisations. A recent study investigated the use of Slack as acommunication platform to enhance collaborative learning and online activities in online andflexible programming courses for in-service teachers (Rouhani, 2020). The results indicatedSlack as one of the factors that possibly contributed to increased activity and learning in thecourse, specifically through the role of instantaneous feedback and discussions. Though
Slack is regularly used by students and educators in higher education for teamcommunication and collaborative learning (Menzies & Zarb, 2020; Ross, 2019), there is a
gap in the research that explores the potential use of the platform for fostering habit
development. Additionally, none of the previous studies have applied the IDC theory inacademic contexts through ubiquitous and collaborative learning tools to develop learning
habits.
The instructors in the ET Program at IIT Bombay hypothesized that if students’curiosity, creativity and routine was stimulated on the Slack platform, they were likely toacquire good learning habits and evolve into enthusiastic creators. In this approach, theinstructors aimed to foster habits of conveying complex and creative ideas in an accessible
manner among their graduate students. To achieve this goal, utilising the Slack platform, theinstructors integrated learning activities that were aligned with students’ pre-existing interests
into their daily learning routines. The instructors methodologically harnessed the habit loop
components of the IDC theory by finding the right triggers as cues for starting the habit,
following a routine and providing repeated tasks with small goals to students, and examining
the outcomes of these activities.

3. Pedagogical Design
3.1 Learning Activities
Utilizing the Slack platform as a ubiquitous learning tool, the pedagogical design of thisintervention was based on the habit loop components of the IDC theory. As part of thisintervention, the instructors created two specific learning activities to develop the habit ofconveying complex ideas and concepts in an accessible manner.
Learning activity 1: For this learning activity, the instructors tapped on the known
interest of their students, i.e., to communicate with each other through social media. In
March 2020, an email was received by all PhD and MTech students of the department toengage in the creation and sharing of a daily ET Newsletter on Slack platform. In terms ofcontent creation of these newsletters, the activity was open to include academic andnon-academic issues as well as students' personal experiences. Since this activity was
conducted in the period of COVID lockdown, it prompted a multitude of articles with creativeand diverse insights from the team members dispersed across the country. On Slack, the529

newsletters served as an opportunity to showcase students’ ideas, creating a sense ofcommunity and collaboration within the team. The articles comprised diverse sections ofentertainment including research, fiction, philosophy, cartoons, creativity corner, health,travel, etc. The key goal of this activity was to cultivate the habit of communicating intricate
and innovative concepts in a way that is easily understood by a wide cohort. The purpose of
fostering this habit was to enable individuals to deconstruct complex ideas or concepts into
simpler and understandable terms which can be consumed by a wider audience. This is anextremely important habit in the field of academia which can help individuals become
effective communicators and improve their ability to connect with others.
Learning activity 2: In the second learning activity, instructors fostered students’pursuit to create memes as a voluntary activity. Meme creation and sharing was an activityof interest to students which was carried out by them on their personal chat groups. This wasnoticed by instructors in an informal discussion, after which the students were asked to start
sharing these memes on the Slack platform with all team members. Furthermore, an
instructor incorporated the idea of EdTech memes activity as a regular practice associatedwith assessments for his courses, namely Human Computer Interactions and Learning
Sciences. As part of the assessment, students were given specific guidelines for creating
memes related to each course element, and they were required to upload their creative
outputs on Slack. To create an effective meme, students must extract complex ideas into abrief message and present it in a way that is easily understandable and engaging. Thisrequires not only creativity but also critical thinking skills to convey and determine how best
to present it. Through memes creation, the instructors aimed at developing the habit ofconveying complex ideas in an accessible and engaging way, which can be a valuable skill
in many personal and professional contexts.

3.2 Instructor Actions
The intervention focused on nurturing students' interests, fostering opportunities formeaningful creation and supporting the development of good habits on the Slack platform.
Hence, the instructors’ actions involved in this approach were categorized into three primary
components, namely tapping creative interests of students, creation of learning activities andmotivating students to not only initiate but also maintain engagement in learning activities
until they become habitual (Fig. 1).
Instructors tailored creation-based learning activities to student interests and context,
thereby aiding them in overcoming their initial resistance to building a new habit. Student
interests were identified through informal conversations (on Slack or in person) andclassroom observations, which was a continuous and implicit effort by the instructors (Fig.
1). Additional design characteristics were employed to facilitate the cognitive engagement ofstudents with the learning activities. These features included creating a focused question to
guide the learning activity, developing activities which were relevant, required criticalexamination, and were open to diverse and valid perspectives.
After posting the learning activities and creating dedicated channels on Slack, theinstructors dedicated their efforts towards inspiring and motivating the students to sustain
their engagement in the tasks. By sharing detailed written guidelines, instructors ensured
that the purpose of the learning activity was clear to the students before the start of theactivity which made it goal-directed.
Modelling of learning activities by instructors is crucial to habit development due tothe nature of humans (Chan et al., 2018). The instructors modeled their own creations on
Slack showing a clear example of how the task and skill should be performed. This helped
the students get a better understanding of the expected outcome. Also, when the students
witness the instructor getting involved in the learning activity, they are more likely to become
motivated to do the same, which enhances their focus and engagement in the task.

530

Figure 1. The figure illustrates the instructors' pedagogical design categorized into three
primary components.
Furthermore, the instructors often demonstrated positive learning behaviour byrecognizing the new knowledge gained from other students and providing feedback on theircreation. Here, this aspect holds significant importance as these activities foster thedevelopment of habits by combining self-practice along with learning from the strengths andweaknesses of others. Instructors also motivated the students through constructive feedback
on each creation.
As the process of habit development for complex tasks is an interplay between one’s
agency and situational resources (Chen et al., 2020), the instructors ensured that thestudents were not restricted in their creativity due to technological access or any otherconstraints. Students were provided with the autonomy to use simplest resources orcomplex tools to come up with their products. The following excerpt from an instruction
provided to students, for learning activity 1, on the Slack platform was an example of suchcreative freedom: “Good morning all. Thank you all for your contributions in all aspects. For
the future broadcasters here is a small reflection: Do it the way you want to and areinterested in doing it. The most important part is to have fun and enjoy doing this. It’s notnecessary to follow any format. Make your own format and try to involve as many people aspossible. They may not be able to give you exactly what you want but that is ok they havetheir own challenges. Take what you get and use your freedom to create anything you like.
Most importantly, have fun.”
The instructors chose the Slack platform to facilitate these learning activities andfoster a culture of sharing among the students. By encouraging them to share their work withpeers, a sense of community and collaboration was created, which served as anothermotivating factor for the students to perform the tasks diligently and receive feedback.

4. Methodology
4.1 Research Question
While the link between habits and learning is widely recognized, there is much less researchthat investigates how learning habits are formed in various circumstances using ubiquitous
learning tools. The aim of this study was to explore the development of learning habits
among students on Slack using the IDC theory. Hence, the following research question was531

investigated in this study: How does the instructors' pedagogical design, based on the habit
loop of the IDC theory, support the development of learning habits on the Slack platform?

4.2 Participants
The learning activities were offered on the Slack platform to foster learning habits among
PhD and MTech students of the ET Program at IIT Bombay. However, these learning
activities were also participated by other team members and the alumni community. For thelearning activity 1, there were a total of 5 instructors, 30 students, 12 alumni and 10
academic staff members who participated on the Slack platform through creation activities
and feedback mechanisms. However, for the learning activity 2, there were 5 instructors, 51
students, 8 alumni and 2 staff members as participants.

4.3 Data Sources and Data Analysis
The data source of this research came from the time-stamped chat logs of Slack, which was
created for the team members of the ET Program in 2018. Instructors often initiated casualconversations on different Slack channels to keep the students connected and motivated. In
March 2020, the pedagogical approach, based on the IDC theory, was utilised on two new
Slack channels named newsletter_et and et_memes. The chat logs were downloaded fromthese channels, and organised chronologically along with relevant documents, images andother media shared within the Slack workspace. The number of posts and threads related to
the learning activities on Slack were quantified and presented in the results section.
The chats on each channel were screened to filter the threads showing evidence fordifferent components of the habit loop and instructor actions. Conversation analysis(Meredith, 2019) was used to analyse the participation of instructors and students in theconversations and study the structure and specific aspects of these discussions, including
instructor and peer involvement, quality of discussions, frequency of feedback, etc. Patterns
were examined in the data to find relevant evidence for the research question, involving
linguistic resources such as use of word choices, sentence structure and language,
emoticons, and team members’ responses. Analysis and patterns were used to map the
instructors’ actions aimed at transforming routine activities into learning habits of conveying
complex ideas in simple ways. Additionally, member check interviews were conducted with
two of the instructors who played significant roles in the design and conduct of theselearning activities. Member check was performed to provide any corrections, clarifications, oradditional insights on the pedagogical approach and its relation to the findings.

5. Results
The habit loop of the IDC theory, consisting of cuing environment, routine and harmony wasused as the framework to design the pedagogy for these activities. The learning activities
chosen in the pedagogical intervention were focused to promote the habit of conveying
complex ideas and concepts in a way that is easily understood by a wide cohort. Herein, we
analyse the interplay of habit loop components of the IDC theory in building this learning
habit among the graduate students employing the Slack platform.

5.1 Cuing Environment
In order to encourage students to start with creative activities without hesitation, a cuingenvironment should be provided (Chen et al., 2020). For both learning activities, there were
three primary context cues that were utilised: a) friendly and social environment of Slack,
open to team members, which can spark interest in such creations (Fig. 2a), b) the
instructors who modeled such interest-driven creation activities and insightful discussions
which could be mimicked by the students (Fig. 2b), and c) constructive feedback provided by532

instructors and peers which helped students gain perception and motivated them to excel(Fig. 2c).

Figure 2: Primary context cues for learning activities: a) friendly Slack platform among teammembers b) instructor modeling for interest-driven creation activities, and c) constructive
feedback from team members.
The possibility of using a ubiquitous technology in sharing such creative outputs also
acts as a trigger in this context. On Slack, students were able to share their creations
horizontally in a safe environment with peers and also vertically across different professional
cohorts, including faculty, alumni and different years of PhD and MTech students. Feedback
and discussions on these creations were rich and insightful, and free of judgement which
made the students feel safe to express their honest opinions and thoughts (Fig. 2). The
well-articulated creations and discussions from peers also exposed students to new ideas,
perspectives and arguments, which is important to improve the ability to evaluate complex
information and communicate in an effective manner.

5.2 Routine
By making interest-driven creation activities a regular part of students’ routine, they are more
likely to develop a habit of pursuing their interests (Chan et al., 2018; Wong et al., 2020).
The instructors consistently motivated the students to upload their creative outputs fromlearning activity 1 on the Slack platform by 10 AM each morning. This enabled their peers toread and discuss the same throughout the day. As a result, there were a total of 31 issues of533

newsletters that were created over a period of 4 weeks by 15 broadcasters while all 57
participants were involved in active and rich discussions which entailed 221 thread posts.
Memes creations were significantly contributed by junior PhD and MTech students(38 students); however, the activity was sporadically participated by all 66 participants,
including instructors and alumni. Though there was no specific time allotted for this activity
by the instructors, they were continuously encouraged to create and share them on a regularbasis. There were a total of 185 memes with 28 thread posts and hundreds of emoticons
associated with them from November 2020 to September 2021, when it was the final activity
provided by the instructor. The instructors' encouragement to establish a routine forinterest-driven creations and discussions amongst the students led to not only consistentengagement with these activities, but also a high volume of creative output.

5.3 Harmony
In the habit loop, harmony refers to the affective outcome of the routine activity as well as
the integration or stabilization of habits (Chen et al., 2020). The participants expressed theirsatisfaction from their engagement with these learning activities (Fig 3a), which in turn led topursuing the routine activity whenever there was an opportunity. The participants also statedtheir impatience in anticipation of the upcoming creations from their peers (Fig. 3b),suggesting that they have developed certain habits.

Figure 3: Participants expressing a) satisfaction through these learning activities, andb) their anticipation of similar creations, suggesting development of habitual behavior.
Such habits are likely the results of repeated exposure and active engagement inrelevant activities. Both learning activities are still continued to be pursued, sporadically, by
the students on Slack platform without indulgence or moderation of the instructor. Over time,
the format of the newsletter evolved; however, even after the instructors’ involvement ended,
25 articles, with rich discussions, have been generated on that Slack channel by multiple
cohorts over a period of 2 years. Likewise, for memes, 61 additional memes were createdand shared on Slack subsequent to the instructor's last intervention. The sustained
production of these creative outputs also indicated clear evidence of habit activation.

6. Discussion
Creating interest and context-driven learning activities for students is crucial in building
habits because it provides a meaningful and enjoyable experience for them. While Slack has
previously been utilized as a communication tool to enhance collaborative learning andonline engagement (Rouhani, 2020), the present study demonstrates an effective use of theplatform for developing positive learning habits, by applying the IDC theory, among graduate
students. The pedagogical design exhibits how instructor actions can play a crucial role indeveloping good learning habits using a ubiquitous learning tool. This also contributes
towards one of the questions posed by the creators of IDC theory on training teachers tonurture the interests of students, provide opportunities for meaningful creation, and support534

the development of good habits (Chan et al., 2018). The pedagogical approach categorised
the teacher's actions into three main components: stimulating the creative interests ofstudents, designing learning activities, and encouraging students to not only start but alsosustain engagement in these activities until they become habitual. A crucial element
observed during the discussions between instructors and graduate students on their creative
outputs was the instructors' openness to embracing constructive disagreements about
varying perspectives. The instructors deliberately embraced this attitude to encourage thedevelopment of critical thinking to foster learning good habits in their students.
The study highlighted the role of the Slack platform, instructors’ actions in the design
and application of this intervention, and the interplay of the habit loop components innurturing the habit of conveying complex ideas in a simple and accessible manner among
graduate students. The cuing environment was established through a social and friendly
environment provided by the Slack platform, where the instructor modelled interest-driven
creation activities and insightful discussions. The use of Slack as a platform enabled
graduate students to receive constructive feedback from instructors and peers, leading toenhanced engagement and effective communication of complex information. The routine
was established through regular practice of sharing creative products from learning activities
on the Slack platform. The harmony, or the affective outcome of the routine activity, wasevident in the satisfaction of learning, anticipation expressed by the students for theupcoming creations and continual pursuit of learning activities. The findings of this study canprovide valuable insights for educators in this digital world, especially in light of the recent
review suggesting the crucial role of the schools in developing good habits for students(Ekman et al, 2022).
Recognizing the positive response to IDC practices in Asia (Looi et al., 2023), thiswork represents a small attempt to build good learning habits in students through its
application. Nevertheless, there were a few limitations associated with the study. The study
could not control for potential confounding variables that could influence habit formation,such as prior habits, personality traits, and other external factors. Also, the study measured
the development of habits over a short period of time of a few months, and the long-term
effects of the pedagogy were not assessed. Further research would be needed to explore
the sustainability of the approach. Additionally, the study relied on subjective measures suchas satisfaction and motivation to assess habit development. To provide more robust
evidence, future research would incorporate markers for quantitative analysis to monitor andrecord the frequency and duration of engagements in the targeted habit-forming activities.

7. Conclusion
The study utilised the Slack platform to demonstrate a pedagogical approach, rooted in the
habit loop of the IDC theory, in building students’ learning habits. Cultivating the habit ofconveying complex ideas and concepts in a clear and comprehensible manner can greatly
benefit graduate students in their career development. This skill not only enhances theirproficiency in effective communication, but also improves their ability to establish meaningfulconnections with others. Leveraging the Slack platform, the instructors supported thedevelopment of this learning habit and fostered a culture of peer learning by tapping creativeinterests of students, designing relevant learning activities and motivating students toregularly engage in these activities. The results of the study have important implications foreducators using ubiquitous learning tools to improve their pedagogical practices andcultivate positive learning habits in their students.

Acknowledgements
We would like to thank the Next Education Research Lab for providing the financial support
for our project. We are also thankful to the Tata ClassEdge Limited for its support in thepresentation of this work. Our gratitude to the faculty members, alumni, staff, and researchscholars, from the ET Program at IIT Bombay, who contributed to the study.

535

References
Aljawarneh, S. A. (2020). Reviewing and exploring innovative ubiquitous learning tools in higher
education. Journal of computing in higher education, 32, 57-73.
Chan, T. W., Looi, C. K., Chen, W., Wong, L. H., Chang, B., Liao, C. C., ... & Ogata, H. (2018).
Interest-driven creator theory: Towards a theory of learning design for Asia in the twenty-first
century. Journal of Computers in Education, 5, 435-461.
Chen, W., Chan, T. W., Wong, L. H., Looi, C. K., Liao, C. C., Cheng, H. N., ... & Pi, Z. (2020). IDC
theory: habit and the habit loop. Research and Practice in Technology Enhanced Learning, 15(1),1-19.
Chen, M. C., Chan, T. W., & Chen, Y. H. (2022). How to encourage students continue learning English
on podcast? The perspectives of stimulus-organism-response and interest driven creator theory.
Interactive Technology and Smart Education, (ahead-of-print).
Dontre, A. J. (2021). The influence of technology on academic distraction: A review. Human Behavior
and Emerging Technologies, 3(3), 379-390.
Ebele, U. F., & Olofu, P. A. (2017). Study Habit and Its Impact on Secondary School Students'
Academic Performance in Biology in the Federal Capital Territory, Abuja. Educational Researchand Reviews, 12(10), 583-588.
Ekman, R., Fletcher, A., Giota, J., Eriksson, A., Thomas, B., & Bååthe, F. (2022). A Flourishing Brainin the 21st Century: A Scoping Review of the Impact of Developing Good Habits for Mind, Brain,
Well‐Being, and Learning. Mind, Brain, and Education, 16(1), 13-23.
Huang, M. C. L., Chou, C. Y., Wu, Y. T., Shih, J. L., Yeh, C. Y., Lao, A. C., ... & Chan, T. W. (2020).
Interest-driven video creation for learning mathematics. Journal of Computers in Education, 7,395-433.
Kirkpatrick, R., & Zang, Y. (2011). The negative influences of exam-oriented education on Chinese
high school students: Backwash from classroom to child. Language testing in Asia, 1(3), 36.
Kong, S.C., Chiu, M.M. and Lai, M. (2018), “A study of primary school students’ interest, collaboration
attitude, and programming empowerment in computational thinking education”, Computers andEducation, Vol. 122, pp. 178-189.
Kong, S. C., & Li, P. (2016, May). The interest-driven creator theory and coding education. In
Workshop proceedings of the 20th global chinese conference on computers in education (pp.
116-119).
Kong, S. C., & Wang, Y. Q. (2019). Nurture interest-driven creators in programmable robotics
education: an empirical investigation in primary school settings. Research and practice intechnology enhanced learning, 14(1), 20.
Looi, C. K., Wong, S. L., Kong, S. C., Chan, T. W., Shih, J. L., Chang, B., ... & Liao, C. C. (2023).
Interest-Driven Creator Theory: case study of embodiment in an experimental school in Taiwan.
Research and Practice in Technology Enhanced Learning, 18.
Menzies, R., & Zarb, M. (2020). Professional communication tools in higher education: A case study
in implementing Slack in the curriculum. In 2020 IEEE Frontiers in Education Conference, pp. 1-8.
Meredith, J. (2019). Conversation analysis and online interaction. Research on Language and Social
Interaction, 52(3), 241-256.
Montrief, T., Haas, M. R., Gottlieb, M., Siegal, D., & Chan, T. (2021). Thinking outside the inbox: useof Slack in clinical groups as a collaborative team communication platform. AEM Education andTraining, 5(1), 121.
Ross, S. M. (2019). Slack it to me: Complementing LMS with student-centric communications for themillennial/post-millennial student. Journal of Marketing Education, 41(2), 91-108.
Rouhani, M. (2020). Utilizing slack as a communication platform in a flexible learning trajectory
course: supporting the learning process. In Proceedings of the 9th Computer Science Education
Research Conference, pp. 1-7.
Slack Technologies, Inc. (n.d.). Slack: Where work happens. Retrieved from https://slack.com/
Watanabe-Ito, M., Kishi, E., & Shimizu, Y. (2020). Promoting healthy eating habits for college
students through creating dietary diaries via a smartphone app and social media interaction:
Online survey study. JMIR mHealth and uHealth, 8(3), e17613.
Wong, L. H., Chan, T. W., Chen, W., Looi, C. K., Chen, Z. H., Liao, C. C., ... & Wong, S. L. (2020). IDC
theory: interest and the interest loop. Research and Practice in Technology Enhanced Learning,15(1), 1-16.
Wong, S. L., & Wong, S. L. (2019). Relationship between interest and mathematics performance in atechnology-enhanced learning context in Malaysia. Research and Practice in Technology
Enhanced Learning, 14(1), 1-13.

536

" !   !" " "   #" !
#" !"$  #" ! #"                        !!!"
 "    !"   

A2E492A2?<<F 24 E9   6G6=@A:?8 2 D6=7C68F=2E65 32D65 A6CD@?2=:K65 @?=:?6 =62C?:?8 DJDE6>
(%$!(7@C=62C?:?8724E@C:K2E:@?2:>65E@AC@G:562AAC@AC:2E6 =62C?:?8>2E6C:2=DE92E
2==@HDEF56?EDE@=62C?724E@C:K2E:@?2?54@?EC@=E96>D6=G6DE@244@>A=:D9E96:CE2C86E
)96 DEF5J 2:>65 E@ :?G6DE:82E6 E96 :>A24E @7 FD:?8 (%$!( @? DEF56?ED =62C?:?8
249:6G6>6?E 2?5 E96:C A6C46AE:@? @7 @?=:?6 D6=7C68F=2E65 23:=:EJ )96 A2CE:4:A2?ED
4@?D:DE65@72D:?8=68C@FA4@>AC:D:?8D6G6?EJEH@6:89E98C256DEF56?ED)96JH6C6
25>:?:DE6C65AC62?5A@DEE6DED367@C62?527E6C4@>A=6E:?82=6DD@?@?724E@C:K2E:@?
55:E:@?2==J E96J H6C6 C6BF:C65 E@ C2E6 E96:C D4@C6D @? 2? @?=:?6 D6=7C68F=2E65
BF6DE:@??2:C6367@C62?527E6CFE:=:K:?8(%$!( )96C6DF=EDD9@H65E92E:?4@CA@C2E:?8
(%$!(=65E@2DE2E:DE:42==JD:8?:7:42?E:?4C62D6:?E962G6C286DEF56?ED249:6G6>6?E
@?E96A@DEE6DE7@CE96?F>6C:42=724E@C:K2E:@?4@>A2C65E@E96:CA6C7@C>2?46@?E96
AC6E6DE@H6G6CFD:?8(%$!(7@C=62C?:?8A@=J?@>:2=724E@C:K2E:@?5:5?@E6=:4:E2
DE2E:DE:42==J D:8?:7:42?E 492?86 :? 249:6G6>6?E D4@C6D "@C6@G6C :? A2CE @7 E96 D6=7
C68F=2E65 23:=:EJ 27E6C FD:?8 (%$!( A2CE:4:A2?ED D4@C:?8 @? AC6 2?5 A@DED6=7
C68F=2E65BF6DE:@??2:C6D9:DBF2C6:>A=:6D5:D4@G6C:?8E96C6=2E:@?D9:A36EH66?EH@
42E68@C:42= G2C:23=6D )96 C6DF=E D9@H65 E92E D6G6?E66? BF6DE:@??2:C6 :E6>D H6C6
DE2E:DE:42==JD:8?:7:42?E27E6CFD:?8(%$!(  %6CD@?2=:K65 @?=:?6 =62C?:?8 DJDE6> D6=7C68F=2E65 724E@C:K2E:@?
F3:BF:E@FD=62C?:?8>2E96>2E:4D

 $" %$ )?6 DEC6?8E9 @7 :>A=6>6?E:?8 E649?@=@8J :? E96 4=2DDC@@> H2D 2DD:DE:?8 E96 @G6CD:89E 2
86?6C2=4=2DDC@@>92C5=JC62496D -@ E96:562@7A6CD@?2=:K65=62C?:?8FDF2==J2AA62CD:?
E964@?E6IE@7@?=:?6=62C?:?8DJDE6>DD:?46:E6?23=6D2446DDE@DF492DJDE6>2?JH96C6 
2?JE:>6 *6CD@?2=:K2E:@?D:?E6?EDAFCC65>@E:G2E:@?2=2?5>6E24@8?:E:G6DE2E6D=:<6A@D:E:G6
27764E2?57@4FD652EE6?E:@?$?8<2G2C26E2=  *2?;23FC666E2=  02=<:?8E@?
6C?24<:        #@H6G6C  E96 A6CD@?2=:K65 @?=:?6 =62C?:?8 DJDE6> >:89E ?@E DE2?5
2=@?6H:E9@FE2=62C?:?82AAC@249D:?46E967C665@>E@FD6@7A6CD@?2=:K65@?=:?6=62C?:?8
DJDE6>42?42FD6=62C?:?8:DDF6D -F492D DEF56?ED56=2J7:?:D9:?8E964@FCD62?52C6=@DE:? A66C :?E6C24E:@?  $E C67=64ED :? E96 =24< @7 23:=:EJ E@ 4@?EC@= E96>D6=G6D E@ 244@>A=:D9 E96E2C86E D2JDD6=7 C68F=2E65=62C?:?8  -6=7 C68F=2E65=62C?:?8-,&:DE96D6=7 AC@46DD2?5D6=7 36=:67DE92E6?23=6=62C?6CD
E@ EC2?D7@C> E96:C >6?E2= 23:=:E:6D :?E@ 24256>:4 A6C7@C>2?46 D<:==D  0:E9 E9:D  -,& :D
4@?D:56C65E96AC@24E:G6AC@46DDE92E@?6FD6DE@5C:G6E96>D6=G6DE@244@>A=:D9=62C?:?8
7@C:?DE2?46 8@2=D6EE:?8 DEC2E68:6DD6=64E:@? D66<:?896=A @CD6=7 6G2=F2E:@?1:>>6C>2?   0:E92==E96D6-,&DEC2E68:6D :EEFC?D2>2DD:G62>@F?E@7H@C<7@CDEF56?EDE@42CCJ
2?5244@>A=:D9:?E9686?6C2=4=2DDC@@>D6EE:?8E92E:DH9JE649?@=@8J364@>6D2?677:4:6?E
2DD:DE2?E -@ E9:DC2:D6DE96492==6?86D@7H6== 4@>3:?2E:@?D6=7 C68F=2E65=62C?:?8DEC2E68:6D
:?E@2A6CD@?2=:K65@?=:?6=62C?:?8DJDE6>E@96=ADEF56?ED249:6G6:?E96=62C?:?84@?E6?E 537)E96CA@:?EDC6=2E65 2D7C@>2564256C6G:6H2CE:4=6D  @7&2:2?5#H2?8
C6A@CE65 E96C6H6C6;FDE7:G6DEF5:6D7C@>29F?5C657@4FD65@?FD:?8E96-,&DEC2E68J:?6=62C?:?87@C>2E96>2E:4D&2:#H2?8    $E:DH6==<?@H?E92E'2E96>2E:4D:D2DF3;64E
E92E4@?E6?E>2:?=JAC6D6?ED2D?F>6C:42= DJ>3@=:4 2?5G2C:23=6 .96C67@C6 >@DEDEF56?ED
H@F=5365:77:4F=E6G6?E@368:?@C8:G6FA@?=62C?:?8 0:E9E9:D 2?F?56CDE2?5:?8@7E9@D6
23DEC24E?@E:@?D2?5E96:C4@?46AED:DAC@A@D65E@564C62D64@?7FD65=62C?:?82?52=D@82:?
E9623:=:EJE@=62C??6H<?@H=6586C6=2E65E@E96AC6G:@FD4@?46AECF?6C%6??6J   &:<6H:D6  BF25C2E:4 A@=J?@>:2= 724E@C:K2E:@? :D @?6 42E68@C:K65 2D 7F?52>6?E2= :? =62C?:?8
=863C2DE:== 76HC6D62C496CDC6=:65@?E9:D4@?E6?E:?A6CD@?2=:K65@?=:?6=62C?:?8DJDE6>D
H:E9 -,& DEC2E68:6D  .96C67@C6  56G6=@A:?8 2 D6=7 C68F=2E65 32D65 A6CD@?2=:K65 @?=:?6
=62C?:?8DJDE6>-*)&-364@>6D492==6?8:?87@CE9:DDEF5J3JC676CC:?8E@2?6I:DE:?856D:8?
@7 @FC AC6G:@FD DEF5J $?8<2G2C2 6E 2=     0:E9 EH@ A@:?ED =@@<:?8 7@C  5@6D =62C?:?8
249:6G6>6?E 5:776C H96? DEF56?ED C646:G6 E96 -*)&- 7@C =62C?:?8 724E@C:K2E:@? :?
>2E96>2E:4D 2?55@DEF56?EDA6C46AE:@?D@7E96:C@?=:?6-,&5:776C367@C62?527E6CFD:?8
E96-*)&-7@C=62C?:?8724E@C:K2E:@?:?>2E96>2E:4D & !$ "%$ #"# )"(#$%$ #"# ( "'#$ -*)&- :D 56G6=@A65 7C@> E96 AC:?4:A=6D @7 =62C?:?8 AC676C6?46 32D65 =62C?:?8 DJDE6>D  $E
DE2E6D E92E :?5:G:5F2=D 4@F=5 C646:G6 AC@A6C =62C?:?8 >2E6C:2=D C6=6G2?E E@ E96:C AC676C6?46D
2?5 AC@G:56 6249 4@?46AEF2= =62C?:?8 @FE4@>6 $?8<2G2C2 6E 2=     $? 255:E:@?  -,&DEC2E68:6D2C625565E@2DD:DEDEF56?ED:?249:6G:?8E96:C=62C?:?8 2?5E96FD:?8DE6AD2C6
56D4C:3652D7@==@HD.96DJDE6>AC@G:56D=62C?:?8>2E6C:2=D4@CC6DA@?5:?8E@A6CD@?2==62C?:?8DEJ=6D ==2C62D<65E@D6EA6CD@?2=8@2=D2?5D6BF6?46=62C?:?84@?E6?ED!:8FC6  
67@C6:?G@=G:?8=62C?:?824E:G:E:6D 2AC6 E6DE?665DE@365@?6E96DJDE6>5:28?@D6D
=62C?:?823:=:EJ2?5AC@G:56D=62C?:?8DE2EFD H9:492==@HDDEF56?EDE@25;FDE8@2=D2=@?8
E96=62C?:?8AC@46DD!:8FC6 
@>A=6E6 A@DE E6DED 27E6C 7:?:D9:?8 2== 4@?E6?ED E96 DJDE6> 5:DA=2JD 86EE:?8 D4@C6D 7@C
6249!:8FC6 
*6CD@?2= =62C?:?8 2?2=JD:D DF>>2C:K6D E96 @G6C2== =62C?:?8 249:6G6>6?E  @FE4@>6
4@>A2C:D@? 36EH66? AC6 2?5 A@DE E6DE  DF446DD A6C46?E @7 =62C?:?8 249:6G6>6?E  2?5DF446DDA6C46?E@7=62C?:?8E:>68:G6?!:8FC6 
7E6C 7:?:D9:?8 2== =6DD@?D  -*)&- 2==@HD C6E6DE:?8 2?5 C6=62C?:?8 H:E9 @E96C =62C?:?8
>2E6C:2=D  "$"#  " " $ "($ $$#.966 =62C?:?8>2E6C:2=D2C6EJA:42==J4@?46C?65:?EH@A2CED )?6H2D23@FEG2C:@FD=62C?:?8
>2E6C:2=D 2?5E96D64@?5H2D4@?46AEF2=<?@H=6586:?724E@C:K2E:@? !@FC=62C?:?8>2E6C:2=D
H6C625@AE65C682C5:?8!6=56C -:=G6C>2?D'@56=E@4@G6CA6CD@?2==62C?:?8AC676C6?46DE@
7:EE96@?=:?6=62C?:?8DJDE6>2DAC@A@D65:?2AC6G:@FDDEF5J*2?;23FC66-C:D2H2D5:   
.96C6 C6=:6D @? 7@FC 492C24E6C:DE:4D =:<6D 24E:G6 G:DF2=  C67=64E:G6 G:DF2=  24E:G6 G6C32=  2?5C67=64E:G6 G6C32= !@C 4@?46AEF2= <?@H=6586  E9:D DJDE6> AC@G:565 EH@ >2:? 4@?E6?ED : 6  ?F>6C:42=
2?5 BF25C2E:4 A@=J?@>:2= 724E@C:K2E:@?  /D:?8 G:CEF2= 2=863C2 E:=6D :D E96 4@C6 :562 :?
56G6=@A:?82==E96D6=62C?:?8>2E6C:2=D!@D?2F89':E496==  "2CKL?2FE:DE2   
-EF56?ED24BF:C65724E@C:K2E:@?<?@H=65867C@>E96C6=2E:@?@7C64E2?8=62C62E92E5:DA=2JD2D
2AC@5F4E@7EH@?F>6C:42=@C6G6?DA64:7:4EH@ E6C>DH:5E92?5=6?8E9 



538



             

      

      *6CD@?2=8@2=D6EE:?8  44@>A=:D9>6?E:?4@?E6?E                                     

  



 



 



 

 

  44@>A=:D9>6?E:?E96=6DD@?    

       &62C?:?82?2=JD:D 








#"$   (.96 ?@? 6IA6C:>6?E H:E9 2 BF2?E:E2E:G6 2AAC@249 H2D 2AA=:65 :? E9:D DEF5J E@ :?G6DE:82E6
DEF56?ED 249:6G6>6?ED 367@C6 2?5 27E6C E96 :?4@CA@C2E:@? @7 -*)&- 2E E96 D2>6 E:>6 
DEF56?EDA6C46AE:@?@7@?=:?6-,&23:=:EJ367@C62?527E6CH2D2=D@7@F?5@FE $?E9:DDEF5J 6:89E9 8C256DEF56?EDFD65-*)&-:?E96D64@?5D6>6DE6C  .96C67@C6 @?64@F=5
DEF5JE9C@F89-*)&-H:E9:?2>@?E9244@C5:?8E@@?6DDEC2E68JE@244@>A=:D92===6DD@?D !:CDE=J DEF56?EDH6C62D<65E@E2<62AC6 @?=:?6D6=7 C68F=2E65=62C?:?8BF6DE:@??2:C6)-&+2?52AC6 =62C?:?8249:6G6>6?EE6DE?F>6C:42=724E@C:K2E:@?7@==@H653J24E:G:E:6D .96?E2<6
E96 A@DE =62C?:?8 249:6G6>6?E E6DE ?F>6C:42= 724E@C:K2E:@? 27E6C 7:?:D9:?8 2== =6DD@?D -64@?5=J  E96 AC6 =62C?:?8 249:6G6>6?E E6DE A@=J?@>:2= 724E@C:K2E:@? H2D 2D<65 E@ E2<6
367@C68@:?8H:E9A@=J?@>:2=724E@C:K2E:@?24E:G:E:6D ?57:?:D9653JE2<:?8E96A@DE =62C?:?8
249:6G6>6?EE6DEA@=J?@>:2=724E@C:K2E:@?2?5D4@C:?8@?E96A@DE )-&+ .967@FC>F=E:A=6 49@:46D4@C6D2H2C5657@C62494@CC64E2?DH6CH6C6FD65E@2?DH6C
E96 7:CDE C6D62C49 BF6DE:@?  .96C6 2C6 7:7E66? E6DE :E6>D 7@C ?F>6C:42= 724E@C:K2E:@? 4@?E6?E 
H9:=6 EH6?EJ 7:G6 :E6>D 7@C A@=J?@>:2= 724E@C:K2E:@?  D2J 7:G6 BF6DE:@?D A6C DF3 =6DD@?D  $?
255:E:@? E@2?DH6C:?8E96D64@?5C6D62C49BF6DE:@? .96)-&+H2D4@?5F4E65:?E9:DDEF5J
2C?2C56E2=   .96C6H6C6D:I6G2=F2E:@?DF3D42=6D EH6?EJ 7@FC:E6>DH:E927:G6 A@:?E
&:<6CE D42=6  .96 :?E6C?2= 4@?D:DE6?4J @7 E96 D4@C6 3J DF3D42=6 G2=F6D 7@C C@?3249 2=A92
C2?865 7C@>  E@   :>A=J:?8 2 C6=:23=6 BF6DE:@??2:C6  .9:D DEF5J 4@?5F4E65 324<EC2?D=2E:@?E@6?DFC6E964@?E6?E2?54@>>F?:42E:@?G2=:5:EJ@7E96BF6DE:@??2:C6 #%$# " &$
7E6C2== 52E2H2D4@==64E65 56D4C:AE:G62?2=JD:DH2D4@>AFE65E@56D4C:36AC6 2?5A@DE E6DE
52E2D6ED3J86?6C2E:?8DF>>2C:6D23@FE52E2D2>A=6D .967:?5:?8DC6AC6D6?EEH@4@9@CED
C6=2E65E@E964@?E6?EDEF5J !@CE96?F>6C:42=724E@C:K2E:@?=6DD@? E96C6DF=EDA@:?E65@FEE96
:?4C62D6@7>62?D4@C6:?249:6G6>6?E27E6CDEF56?EDFD6-*)&-7@C=62C?:?8'  '
 2?5E96249:6G6>6?ED4@C6D9@H65>@C6C6=2E:G6=J4@?D:DE6?E:?E96A@DE E6DE
-    -    09:=67@CE96A@=J?@>:2=724E@C:K2E:@?=6DD@? E96>62?D4@C6

539

@7 E96 A@DE E6DE 2 =:EE=6 3:E :?4C62D65 '    '   :? 4@?EC2DE  E96 C6=2E:G6=J
4@?D:DE6?ED9@H2564C62D:?8:?E96A@DE E6DE-  
-    .96 0:=4@I@? D:8?65 C2?< E6DE H2D 4@?5F4E65 E@ D66 2 4@>A2C:D@?  $? E96 42D6 @7
?F>6C:42=724E@C:K2E:@? DEF56?ED92529:896CD4@C6@?E96A@DE E6DEE92?@?E96AC6 E6DE #@H6G6C DEF56?ED925?@492?86:?E96:CD4@C6D H9:=6 92529:896CD4@C6:?E96AC6E6DE   0:=4@I@? D:8?65 C2?< E6DE D9@H65 E92E FD:?8 -*)&- 7@C =62C?:?8 ?F>6C:42=
724E@C:K2E:@?6=:4:E652DE2E:DE:42==JD:8?:7:42?E492?86:?=62C?:?8249:6G6>6?E1  A  !@C E96 A@=J?@>:2= 724E@C:K2E:@?  ?62C3J C6DF=ED :?5:42E65 E92E E6? DEF56?ED 925 ?@
492?86:?D4@C6 ?6I4:E:?8H2DE92EDEF56?ED8@E9:896CD4@C6D@?E96A@DE E6DEH9:=6 
DEF56?EDH6C6@?E96AC6 E6DE .96D6C6DF=ED2C62D:8?:7:42?E4@?46C?D:?46:E:D23@FE 
E92EDEF56?EDA6C7@C>?@E2DH6== 0:=4@I@?D:8?65 C2?<E6DED9@H65E92EFD:?8-*)&-7@C
=62C?:?8724E@C:K2E:@?:?A@=J?@>:2=D5:5?@E6=:4:E2DE2E:DE:42==JD:8?:7:42?E492?86:?=62C?:?8
249:6G6>6?E1  A   "!$  "%$ " $'.96DEF56?EDH6C62D<65E@C2E6E96:CA6C46AE:@?E@H2C5)-&+ AC6 2?5A@DE FD:?8-*)&-
7@C=62C?:?8724E@C:K2E:@? 9: DBF2C6DH6C64@?5F4E65E@5:D4@G6CE96C6=2E:@?D9:A36EH66?
AC6  2?5 A@DE A6C46AE:@?D  !:?5:?8D C6G62=65 E92E DEF56?ED A6C46AE:@? H2D DE2E:DE:42==J
2DD@4:2E65H:E9 BF6DE:@??2:C6:E6>D4@G6C:?87:G6-,&DEC2E68:6D@FE@7D:I8@2=D6EE:?8
"-  E2D< DEC2E68:6D .-  E:>6 >2?286>6?E .'  96=A D66<:?8 #-  2?5 D6=7 6G2=F2E:@?
-   .96"-DEC2E68JC6DF=E65:?E9C66DF3 :E6>DE92ED9@H65DE2E:DE:42==JD:8?:7:42?EH:E9O   A  O  A
2?5O
 A  C6DA64E:G6=J *9: G2=F6 E6DED @7 E96D6 E9C66 DF3 :E6>D D9@H65 E96 DEC6?8E9 @7 2DD@4:2E:@? 36EH66? E96G2C:23=6D 2E 2 >@56C2E6 =6G6=   N   '62? DEF56?ED A6C46:G65 8@2= D6EE:?8 2?5
>2:?E2:?:?829:8924256>:4DE2?52C556DA:E62?@?=:?64@FCD6 2?5DEF56?ED28C665>@C6E@
?@E4@>AC@>:D6@?E96BF2=:EJ@72DD:8?>6?ED3642FD6E9:DH2D2?@?=:?6DEF5J C6DA64E:G6=J '@C6@G6C  E96 :?4C62D:?8 EC6?5 @7 -,& A6C46AE:@? 27E6C FD:?8 -*)&- 7@C E96D6 E9C66 DF3:E6>D H:E9DEF56?EDC2E:?8DDEC@?8=J28C66:?82=>@DE5@F3=65 $?4@?EC2DE E966?G:C@?>6?E
DECF4EFC:?8 -DEC2E68JH2DE96@?=J@?6E92EH2D?@EDE2E:DE:42==JD:8?:7:42?E '@DE@7E96*9:
G2=F6DD9@H65E96DEC6?8E9@72DD@4:2E:@?36EH66?E96G2C:23=6D2E2H62<=6G6= 09:=6E96
DFCAC:D6H2D@?E96.-DEC2E68J DEF56?EDA6C46AE:@?:?5:42E65E92E27E6CFD:?8-*)&- E96:C
A6C46AE:@?DH6C69:896DE7C@>2==DF3 :E6>D !@FC@7E96>D9@H652DE2E:DE:42==JD:8?:7:42?E
5:776C6?46O  A  O   A O   A O  AC6DA64E:G6=J .96>@DEC2E657C6BF6?4J:?DEC@?8=J28C66:DE92EE96J
28C665E@AC24E:46D@=G:?8AC@3=6>D@?-*)&-E@364@>6>@C6AC@7:4:6?E !@CE96.'DEC2E68J E9C66DF3 :E6>DD9@H65DE2E:DE:42==JD:8?:7:42?EH:E9O  AO
 A  O  A  #@H6G6C E96C6H2D@?6DF3:E6>E92ED9@H652C6=2E:G6=JDEC@?8=6G6=@72DD@4:2E:@? H9:49>@DEDEF56?EDC2E65492?8:?8
7C@>?6FEC2=E@28C662?5DEC@?8=J28C66=6G6=D27E6CFD:?8-*)&- '62?E92EDEF56?ED
D6E>@C6E:>67@C=62C?:?83642FD6E96J<?@HE9:D:D2E:>6 4@?DF>:?824E:G:EJ3FE>:89E?@E
36DEC:4E =:<6E96D2>6E:>6@?E96D2>652J $?2H9:=6 #-DEC2E68J E9C66DF3 :E6>DD9@H65
DE2E:DE:42==JD:8?:7:42?EH:E9O   A O  A
2?5O   A   C6DA64E:G6=J  .9:CEJ E9C66 DEF56?ED C2E65 E96 DEC@?8=J 28C66 =6G6= 27E6C
FD:?8 -*)&-  $E :?5:42E65 E92E E96J D92C65 AC@3=6>D H:E9 E96:C 4=2DD>2E6D H96? FD:?8
-*)&- D@E96J<?6HH92EE96JH6C6DECF88=:?8H:E92?59@HE@D@=G6:E &2DE=J - DEC2E68J 
2==DF3 :E6>DC6G62=65DE2E:DE:42==JD:8?:7:42?E O  AO   A O  A 2?5O   A C6DA64E:G6=J @?D:56C:?8E96
C2E:?87C6BF6?4J E969:896DE7C6BF6?4J  H2D23@FEDEF56?ED28C66:?8E@DF>>2C:K6@?=:?6
DEF5:6DE@4964<E96:CF?56CDE2?5:?8@7H92EE96J=62C?65  #%## 
%# 

.96 AC6 2?5 A@DE 724E@C:K2E:@? E6DED 56D4C:AE:G6 2?2=JD:D H2D D6A2C2E65 :?E@ EH@ 4@?E6?ED 

540

?F>6C:42=2?5A@=J?@>:2=724E@C:K2E:@? $?E96?F>6C:42=724E@C:K2E:@? E96>62?@7E96A@DE
E6DE :D 9:896C E92? E96 AC6 E6DE '   '    E96 D2>6 2D E96 >62? @7 E96A@=J?@>:2=724E@C:K2E:@?'   '    @?D:56CE96>62?5:776C6?46@76249
4@?E6?E  (F>6C:42= 724E@C:K2E:@? 4@?E6?E D9@H65  DEF56?ED   H9@ :>AC@G65 E96:C
724E@C:K2E:@? 23:=:EJ 27E6C FD:?8 -*)&-  $? 4@>A2C:D@?   DEF56?ED    8@E =6DD E92?
367@C6 E96 DEF5J  2?5  DEF56?ED   DE:== E96 D2>6 E96D6 C6DF=ED D9@H DE2E:DE:42==J
D:8?:7:42?E 2E A 
 $? 4@?EC25:4E:@?  E96 A@=J?@>:2= 724E@C:K2E:@? 4@?E6?E H2D ?@E
DE2E:DE:42==JD:8?:7:42?E #@H6G6C 4@?D:56C:?8E96>62?5:776C6?46AC@323=J7@F?5E96ECFE9E92E
>@DE@7E96DEF56?ED42?:>AC@G6E96:C23:=:EJ:?724E@C:?8A@=J?@>:2=D3JFD:?8-*)&- DEF56?ED    3FE H:E9 E96 ?62C6DE @7 DEF56?ED H9@ D9@H65 2 564C62D:?8 D4@C6   
DEF56?ED >:89EF?4=62C=JE@>2<62DEC@?84@?7:C>2E:@?E92EE96FD6@7-*)&-42?
:>AC@G6E9623:=:EJ:?724E@C:?82A@=J?@>:2= )G6C2== =62C?:?8249:6G6>6?ED=:89E=J4@>A=:6D
H:E9AC6G:@FDDEF5:6DE92EFD:?82A6CD@?2=:K65@?=:?6=62C?:?8DJDE6>42?AC@>@E6=62C?:?8
249:6G6>6?E 96? 6E 2=     9F 6E 2=     *2?;23FC66 6E 2=     #@H6G6C  @?6A@:?E>6?E:@?65 C676CC:?8E@E96C6DF=E@7E96249:6G6>6?ED4@C6@7A@=J?@>:2=724E@C:K2E:@?
4@?E6?E :DE92E:EH@F=53636EE6C7FCE96CE@D66<E96=625:?842FD6@7E9:D:DDF6 !FCE96C:?
56AE9 DEF5J  :?4=F5:?8 :?5:G:5F2= :?E6CG:6HD  >:89E 4@?46C? E2D< DEC2E68:6D 2?5 E9:D 8C@FAD
=62C?:?8A2E9 
-EF56?EDD4@C65E96:CA6C46AE:@?@?)-&+2>@?8FD:?8-*)&-7@C=62C?:?8 )G6C2== E96C6DF=EA@:?E65E@ DE2E:DE:42==JD:8?:7:42?E:E6>D27E6CFD:?8-*)&- #@H6G6C @?6-,&
DEC2E68JH2D?@EDF3DE2?E:2=  - @?D:56C:?8E967C6BF6?4J@7D4@C6C2E:?8 :E7@F?5E92E>@DE
DEF56?EDC2E65DEC@?8=J28C66@?E9623:=:EJE@49@@D624@>7@CE23=6A=2467@CDEF5J 3FEE9:D
7C6BF6?4JDE:==564C62D657C@>367@C6 -@ 7:?5:?82=62C?:?8DA246E92E=6EDE96>@44FAJE96:C
7F==A@E6?E:2=:DAC6EEJ92C5 .CF=JE92E2?2AAC@AC:2E6=62C?:?8E@@=7@C2446DD:?8-*)&-:D2
4@>AFE6C@C=2AE@AE9:D>:89E86?6C2E62>6DDJE9:?87@CDEF56?EDD:?46:E>:89EDE2E6:?E96
AF3=:42C62@729@FD6 -@ 56G6=@A:?82DJDE6>@C24E:G:EJE92E7:EDA@CE23=656G:46D DF492D
D>2CEA9@?6D  E96 E23=6E 42? =625 E@ 7FCE96C DEF5J  (6IE  E9C66 DF3 :E6>D @7 D6EE:?8 8@2=D :?
=62C?:?8D9@H65D:8?:7:42?46 >62?E@H96E96CD9@CE@C=@?8 E6C> E96JDE:==D6E8@2=D2?5?@E
4@>AC@>:D:?8E96BF2=:EJ@72DD:8?>6?ED;FDE3642FD6E9:D:D2?@?=:?6DEF5J -:?46-*)&-D
762EFC6 2==@HD DEF56?ED E@ D6E 2 8@2= 2E E96 368:??:?8 2D =@?8 2D C6D6E E@ >66E E96:C
4@>7@CE23=6 :ED=:89E=JE2:=@CDE96E2C86E H9:49>@E:G2E6DE96>E@249:6G62EE966?5 H9:49
4@CC6DA@?5DE@E96A@:?EE92E6G6?E9:D:DE96@?=:?64@FCD6E96JH:==?@EC65F46E96BF2=:EJ:?
=62C?:?8 .9:DA@:?E4@CC6DA@?5DE@AC6G:@FDDEF5:6DE92EE649?@=@8J4@F=5DFAA@CEDEF56?ED:?
D6EE:?8 E96:C =62C?:?8 8@2=D 2?5 :?DE2?E=J >@?:E@C:?8 E96:C =62C?:?8 AC@46DD &2:  #H2?8     !@C .- DEC2E68JD C6DF=ED D9@H65 D:8?:7:42?46 :? A6C46AE:@? 86?6C2==J DE2E6 4@?5F4E
?@E6E2<:?8 E2=<:?82=@F5H96?5@:?82?24E:G:EJ AC6A2C:?85:D4FDD:@? BF6DE:@? 2?5AC24E:4:?8
>@C6:?E96DJDE6> $E:DE96C67@C6 -*)&-AC@G:56DA6CD@?2=:K65=62C?:?8>2E6C:2=H9:497:ED
E96:C=62C?:?8DEJ=6 H9:49E9:DA@:?E>:89EDFAA@CE=62C?:?8DEC2E68:6D:?5:G:5F2==J &:<6H:D6 @E96CAC6G:@FDDEF5:6DE92EAC@G:56:?5:G:5F2==62C?:?8>2E6C:2=2?524E:G:EJ25;FDE:?832D65@?
E96:CAC676C6?46>:89E96=ADEF56?ED249:6G6 9F6E2=   02=<:?8E@?6C?24<:   .'DEC2E68J H9:497@4FD6D@?D6EE:?86IEC2E:>6E@=62C? D9@H65DE2E:DE:42==JD:8?:7:42?E:?
E92E-*)&-H2D?@EC6BF:C656G6CJ7:I65E:>6E@DEF5J62494@?E6?E3FEH2D4@?46C?6523@FE
E9623:=:EJE@4@?EC@=2?5249:6G6H92EH2DD6E367@C6 ?57@CE96#-DEC2E68JC6=:65@?9@H
E96JD@F89E96=A2?56I492?865:562DH:E97C:6?5D2?5E62496CD .9:DDE286C67=64E65E92E
DEF56?ED <?@H E96>D6=G6D :? E96 AC@3=6> @?6 762EFC6 E92E 96=AD E96> :? -*)&- :D 6249
=62C?:?84@?E6?EDE2EFD EFC?:?82H2C6?6DDE@:>AC@G6F?E:=249:6G65 H9:49:D=:<6- DEC2E68J 
H9:49 >6?E:@?D E96 23:=:EJ E@ 6G2=F2E6 E96>D6=G6D  $E 86?6C2==J 56D4C:36D E96 A6C46AE:@? @7-,&E@H2C5FD:?8-*)&-7@C=62C?:?8 2DH92E:D<?@H?2D-,&:D24J4=:42=AC@46DD:?H9:49
DEF56?E A=2?D  >@?:E@CD E96:C A6C7@C>2?46  2?5 E96? C67=64ED @? E96 @FE4@>6 1:>>6C>2?  .964J4=642?C6A62E2D@?6FD6DF?E:=249:6G65 D@:E5@6D?@E7:E2==2D=:?62C .92E:D
H9J:ED9@F=536E2:=@C657@CA6CD@??6=H:E9DA64:7:4=62C?:?8A2E9D =D@ E649?@=@8J364@>6DE962?DH6CE@E9:DA@:?E 6?92?4:?84@8?:E:G62?56IA64E65=62C?:?823:=:EJ $? DF>>2CJ  FD:?8 -*)&- 7@C =62C?:?8 724E@C:K2E:@? :? >2E96>2E:4D 42? 6?92?46
=62C?:?8 249:6G6>6?E :? ?F>6C:42= 4@?E6?E  $? 4@?EC2DE  E96 A6C46AE:@? :? -,& D9@H65
D:8?:7:42?46 :? D6G6?E66? @7 E96 DF3 :E6>  !FCE96C>@C6  EH@ DEC2E68:6D : 6  E2D< DEC2E68:6D
2?5D6=7 6G2=F2E:@?D9@H65DE2E:DE:42==JD:8?:7:42?E:?2==DF3 :E6> :?5:42E:?8E92EFD:?8-*)&-

541

42?6?92?46D6=7 C68F=2E6523:=:EJ 6DA64:2==J:?E2D<DEC2E68:6D2?5D6=7 6G2=F2E:@?  '$#

.9:DC6D62C49AC@;64E:DDFAA@CE653JE96(2E:@?2=,6D62C49@F?4:=@7.92:=2?5(,.@?EC24E(@ (   "#
2C?2C5!!2?+-)@-"%2E@?*$!2:(!
"62DFC:?8D6=7C68F=2E:@?:?
@?=:?62?53=6?565=62C?:?86?G:C@?>6?ED', *' ,'$"# *-,$('  N
9EEAD 5@:@C8  ;:965F4  
CF?6C( 6??6J '6AC6D6?E2E:@?2?5"2E96>2E:4D!62C?:?8('("*)#+(!,# 
($ ,/!(* + *#$'#$% . %()& ',   9EEAD 5@:@C8   
96?2>3=6!66.F&!
@C>2E:G62DD6DD>6?EH:E9:?E6C24E:G6
H9:E63@2C5D@?6J62C=@?8:EF5:?2=DEF5J@7AC:>2CJDEF56?ED>2E96>2E:42=A6C7@C>2?46(&)-, *+-,$('  9EEAD 5@:@C8  ;4@>A65F 9F96?" F@'-2?8("6G6=@A>6?E@72?52AE:G62>62D65
:28?@DE:42?5'6>65:2=!62C?:?8(JDE6>2D65@?E96@?46AE7764E"@56=7@C>AC@G:?8
!62C?:?849:6G6>6?ED:?"2E96>2E:4D #'(%("/($ ,/  N 
9EEAD 5@:@C8  )(
1 

@D?2F89!":E496==%)6249:?8)C:4<D2?6)C:4<J# %%$'($+,# &,$+ # *  N 446DD65"2J
/$?=:?60G2:=23=62E
9EEA ;@FC?2=:4E>@C8 :?56IA9A :>E 2CE:4=6 5@H?=@25   
2CKL?2FE:DE2
*:CEF2==863C2):=6DA6528@8:42=E@@=E@E62492?5=62C?2=863C2
E9C@F8986@>6ECJ(-*'%(!(&)-, *++$+,  *'$'" N 
9EEAD 5@:@C8 
;42=

?8<2G2C2)%2?;23FC66%(C:D2H2D5:#(2;;2A2?C@;()96FD6@72A6CD@?2=:K65
=62C?:?82AAC@249E@:>A=6>6?E:?8D6=7C68F=2E65@?=:?6=62C?:?8(&)-, *+'-,$('*,$!$$%', %%$" '  
9EEAD 5@:@C8  ;4262:
!2:!H2?8
(EC2E68:6D7@C6?92?4:?8D6=7C68F=2E:@?:?6=62C?:?82C6G:6H@7
D6=64E65;@FC?2=AF3=:42E:@?D7C@>
E@', *,$.  *'$'"'.$*('& ',+ N 
9EEAD 5@:@C8     
%2?;23FC66% @>2=2H2C592?2#?8<2G2C2)
446AE2?46@7A6CD@?2=:K656=62C?:?8
DJDE6>D242D6DEF5J@74@?46AE67764EC6=2E:@?D9:A2AAC@249@?D4:6?46E649?@=@8J2?5
>2E96>2E:4D4@FCD6D(-*'%(!(&)-, *+$'-,$('  N 
9EEAD 5@:@C8  D  
%2?;23FC66%(C:D2H2D5:#?:?E68C2E65=62C?:?8DEJ=6D2?5D4:6?E:7:4:?G6DE:82E:@?
32D65A6CD@?2=:K65H632AAC@2492C6DF=E@?4@?46AEF2==62C?:?8249:6G6>6?ED2?5
A6C46AE:@?D@79:89D49@@=DEF56?ED(-*'%(!(&)-, *+$'-,$('    N  
9EEAD 5@:@C8  D     +2=<:?8E@?6C?24<:"!
(,$.,$'",- ',+/ *+('%$0$'" *'$'"*(-''$.$-%', * +,+('+$ *,$('(!# (*/ +$"''&)% & ',,$('++- +AA N9EEAD 5@:@C8  (  +2=<:?8E@?6C?24<:"!
%6CD@?2=:K:?8=863C2E@(EF56?ED?5:G:5F2=?E6C6DED:?2?
?E6==:86?E)FE@C:?8(JDE6>"@56C2E@CD@7>A24E', *',$('%(-*'%(!*,$!$$%', %%$" ' $'-,$('  N9EEAD 5@:@C8  D     .:>>6C>2? 64@>:?82D6=7C68F=2E65=62C?6C+9:492C6E96<6JDF3AC@46DD6D
(', &)(**/-,$('%+/#(%("/ N
9EEAD 5@:@C8   ,

.:>>6C>2?
?G6DE:82E:?8(6=7'68F=2E:@?2?5"@E:G2E:@?:DE@C:42=24<8C@F?5"6E9@5@=@8:42=6G6=@A>6?ED2?5FEFC6%C@DA64ED& *$'-,$('% + *#(-*'%  N  9EEAD 5@:@C8 


 



542

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Effects of Visualization Strategies on
Students’ Learning Outcomes inAugmented Reality Contexts
Pei-Shan Tsai*
Teacher Education Center & Graduate Institute of Technological & Vocational Education,
National Taipei University of Technology, Taiwan* sandra.pstsai@gmail.com
Abstract: This study attempted to investigate the effects of visualization strategies onelectrical-machinery course of vocational high school students’ learning outcomes inaugmented reality (AR) contexts. Hence, A learning environment combined withvisualization strategies and AR technology were developed in this study. To exploring
the effects of visualization strategies, a quasi-experimental design was conducted inthis study. The participants of this study included 63 seventh-grade senior high school
students from two classes in Taiwan. Two classes were randomly assigned into control
and experimental groups. Students in control group learned with AR technology while
those in experimental group learned with visualization strategies and AR technology.
The results revealed that students in experimental group outperformed than those incontrol group.
Keywords: Augmented reality, visualizations, electrical-machinery, vocational highschool

1. Introduction
The emergence of the AR technology has been continuously rebuilding students’ learning
experience. One of the major advantages of the AR technology includes its potential toauthentically augment people’s experience in the real world (Wu, Hwang, Yang, & Chen,
2018), which may help learners obtain abstract concepts or perceive unobservablephenomena in a multidimensional way (Xu, Su, Hu, & Chen, 2022). Research results havecommonly suggested that individuals may benefit from the AR-enhanced learning
environment with increased motivation and engagement (Cai, Chiang, & Wang, 2013),
positive attitudes (Behzadan & Kamat, 2013), learning gains in language learning (Cai, Pan,
& Liu, 2022), and high levels of reading comprehension (Bursali & Yilmaz, 2019). However,while it is believed that students may profit from the AR technology in improving academic
achievement (Lu, Liu, Chen, & Hsieh, 2020; Sahin & Yilmaz, 2020), some studies do notreveal such positive effects (Lai & Chang, 2021). Moreover, in a review conducted by
Chang, Binali, Liang, Chiou, Cheng, Lee and Tsai (2022), learners seem to show morepositive responses (e.g. learning motivation or attitude) in learning arts-related subjects thanthose in science.
Indeed, most research has shown promising efficacy on the AR technology inpromoting students’ learning outcomes (e.g. Chang & Hwang, 2018; Sahin & Yilmaz, 2020),
and some research has suggested that the AR technology utilized with the collaborative
pedagogical approach may display higher impact on students’ learning outcomes (Garzón
Kinshuk, Baldiris, Gutiérrez, & Pavón, 2020). However, it is noted that relative few studies
have been found in suggestion of pedagogical strategies in implementing the AR-enhanced
educational context. As the importance of visualization strategies have been emphasis inscience edcuation (Ainsworth, Prain, &Tytler, 2011; Chang, Quintana, & Krajcik, 2010).
Hence, it is worthwhile to investigate the effects of visualization strategies in the AR-543

enhanced educational context so that better AR-enhanced learning environment may beestablished.

2. Methods
2.1 Participants
The participants of this study included 63 seventh-grade vocational high school students
from two classes in Taiwan. Two classes were randomly assigned into control andexperimental groups. Students in control group (n = 35) learned with AR technology while
those in experimental group (n = 28) learned with visualizations and AR technology.

2.2 Research Process
All the students in two groups enrolled in electrical-machinery courses with the same
instructor. At first, all the students were asked to fill in pretest to assess their priorknowledge. During the learning activity, the students in experimental group were learned
with visualizations and AR technology while those in control group were learned with AR
technology. After the learning activity, all the students in two groups were asked to complete
a posttest to understand their learning performance.

Figure 1. Research process in this study.

2.3 Instruments
2.3.1 AR technology
A learning environment combined with visualization strategies and AR technology weredeveloped in this study. The learning materials in text book and AR context were shown in
Figure 1 and Figure 2.

544

Figure 2. The learning materials in textbook.

Figure 3. The learning materials in AR context.
Figure 3 shows a student learns with AR technology. The student can not onlybrowse the learning materials in mobile devices, but also interact with the learning materials
to acquire more abstract knowledge or to conduct inquiry learning activities.

Figure 4. An example of a student learns in AR context.
2.3.2 Learning Outcomes
The learning outcomes in pre-test and post-test were developed by two vocational highschool teachers. Each test consisted of 15 questions by using two-tier tests. The questions
were validated by a teacher and a professor for expert validity.
2.3.3 Data Collection and Analysis545

To explore the effects of the visualizations on the students’ learning outcomes in AR
contexts, this study used the students’ pre-test data as the covariate and post-test data as adependent variable to run the one-way ANCOVA analysis

3. Results
A one-way ANCOVA was used to compare the effect of the two teaching strategies while
controlling for the pre-test scores. As shown in Table 1, the results revealed that students inexperimental group outperformed than those in control group (F(1,59) = 4.24, p < 0.05). It
showed the effects of visualization strategies in AR learning outcomes.
Table 1. ANCOVA results
SourceIntercept
Pre-test
Teaching strategies

SS
275911.20
122.511152.29

DF
1
1
1

MS
275911.20
122.511152.29F
233.16*0.45
4.24*

4. Conclusion
This study investigated the effects of visualization strategies on vocational high school
students’ learning outcomes in augmented reality (AR) contexts. The results revealed thatstudents in experimental group outperformed than those in control group. It implied theimportance of using suitable teaching strategies into innovative technology into instructions.
It is suggested that future studies can investigate the effect of combinations of different
teaching strategies and AR technology in AR learning contexts.

Acknowledgements
This study is supported by the National Science and Technology Council, Taiwan, under
grant numbers MOST 109-2511-H-027-001-MY3.

References
Ainsworth, S., Prain, V., &; Tytler, R. Drawing to learn in science. Science, 333(6046), 1096-1097.
Bursali, H., & Yilmaz, R. M. (2019). Effect of augmented reality applications on secondary school
students' reading comprehension and learning permanency. Computers in Human Behavior, 95,
126-135.
Cai, S., Chiang, F. K., & Wang, X. (2013). Using the augmented reality 3D technique for a convex
imaging experiment in a physics course. International Journal of Engineering Education, 29(4),856-865.
Cai, Y., Pan, Z., & Liu, M. (2022). Augmented reality technology in language learning: A metaanalysis. Journal of Computer Assisted Learning, 38(4), 929-945.
Chang, H.-Y., Quintana, C. &; Krajcik, J. S. (2010). The impact of designing and evaluating molecularanimations on how well middle school students understand the particulate nature of matter.
Science Education, 94(1), 73-94
Chang, H. Y., Binali, T., Liang, J. C., Chiou, G. L., Cheng, K. H., Lee, S. W. Y., & Tsai, C. C. (2022).
Ten years of augmented reality in education: A meta-analysis of (quasi-) experimental studies toinvestigate the impact. Computers & Education, 191, Article 104641.
Chang, S. C., & Hwang, G. J. (2018). Impacts of an augmented reality-based flipped learning guiding
approach on students’ scientific project performance and perceptions. Computers & Education,
125, 226-239.

546

Garzón, J., Kinshuk, Baldiris, S., Gutiérrez, J., & Pavón, J. (2020). How do pedagogical approaches
affect the impact of augmented reality on education? A meta-analysis and research synthesis.
Educational Research Review, 31, Article 100334.
Lai, J. Y., & Chang, L. T. (2021). Impacts of augmented reality apps on first graders’ motivation andperformance in English vocabulary learning. Sage Open, 11(4).
Lu, S. J., Liu, Y. C., Chen, P. J., & Hsieh, M. R. (2020). Evaluation of AR embedded physical puzzle
game on students’ learning achievement and motivation on elementary natural science.
Interactive Learning Environments, 28(4), 451-463.
Sahin, D., & Yilmaz, R. M. (2020). The effect of Augmented Reality Technology on middle school
students' achievements and attitudes towards science education. Computers & Education, 144,
Article 103710.
Wu, P. H., Hwang, G J., Yang, M. L., & Chen, C. H. (2018). Impacts of integrating the repertory grid
into an augmented reality-based learning design on students’ learning achievements, cognitive
load and degree of satisfaction. Interactive Learning Environments, 26(2), 221-234.
Xu, W. W., Su, C. Y., Hu, Y., & Chen, C. H. (2022). Exploring the effectiveness and moderators ofaugmented reality on science learning: a meta-analysis. Journal of Science Education andTechnology, 31, 621-637.

547

" !   !" " "   #" !
#" !"$  #" ! #"         
   72+'2)-  -&#'   &,%&- 
4'1('1  '/-)'   ! "!!" $!# !$ 
"! ! $!# !$ 
7>A8:>B78 8ID<8 5:H>C> D 02 9?   %01; ;<=,A -@847:-, <0- 87;;1*141<A 7. <-)+016/ )6)4A<1+; =<141B16/ ,)14A
4-):616/47/,)<):-+7:,-,16@".7:5)<.7:+4);;)+<1>1<A>1;=)41B)<1762=617:01/0
;+0774:-),16/)+<1>1<A,-;1/6-,<7=<141B-;->-:)4%<774;,1;<:1*=<-,)576/5=4<184-
84)<.7:5; ?); >1;=)41B-, %0- >1;=)41B-, +4);; )+<1>1<A ?); ;07?6 <7 ) 4-):616/
,-;1/6-:7.<0-:-),16/)+<1>1<A=61<)6,<0-6);3-,?0)<;0-:-)41B-,?0-6:->1;16/<0- +4);; )+<1>1<A ,-;1/6 ; ) :-;=4< ?- .7=6, <0)< <0- 16<-/:)<-, 4-):616/ 47/;
8:7+-;;-, .:75 <0- @" +7=4, >1;=)41B- <0- ,1..-:-6+-; 16 <0- )+<=)4 )+<1>1<1-;
+758):-,?1<0<0-4-;;7684)6)6,07?)+<1>-<0-4-):6-:;?-:-6),,1<176?0-6<0-4-):616/,-;1/6-:;)?<01;>1;=)41B)<176;0--@8:-;;-,0-:,-;1:-<7+0)6/-<0-)+<1>1<A,-;1/6<7:-,=+-<0-<:)6;1<1767.)+<1>1<1-;,=:16/+4);;);-,76<0-;-:-;=4<;?-+76+4=,-,<0)<<0-,)14A4-):616/47/,)<):-+7:,-,16<0-@"+7=4,>1;=)41B-<0-<15-)6,+76<-6<7.<0-)+<1>1<1-;8-:.7:5-,*A<0-4-):6-:;%0-:-;=4<;;07?<0-87;;1*141<A
7.+)8<=:16/)6,>1;=)41B16/<0-8:7/:-;;7.)+4);;*A+:7;;84)<.7:5)6)4A;1;=;16/@"16;<-),7.=;16/5=4<184-;8-+1)4;-6;7:;

 %-)+016/6)4A<1+;@"-):616/-;1/6-):616/6)4A<1+; 
=0;HI8=60=3E8BD0;8I8=62;0BB02C8E8C84B8B20;;43C40278=60=0;HC82B0=38B4BB4=C80;5>A
A4B40A270=3C74?A>54BB8>=0;34E4;>?<4=C>5C40274AB!>H>B.4;JB@D4I   +4E4A0;
BCD384B70E4144=2>=3D2C43C>20?CDA4;40A=8=6CA024B5A><5024 C> 5024;40A=8=62>=C4GCB ,74?A8<0AH0??A>027C>C78B8B&D;C8<>30;%40A=8=6 ,40278=6=0;HC82BF8C7E0A8>DBCH?4B>5
30C0BD270B0D38>E834>C4GCB?442764BCDA4B0=34H4 60I4(27>04C0;  )A84C>
4C0;  '3D:F40=84;  &0AC8=4I &0;3>=03>4C0;    !>F4E4A;40A=8=6
C>>;B70E4144=8=2A40B8=6;HDB438=C74308;HC40278=62>=C4GCF7827>5C4=4=01;4;>668=6>5
DB4A8=C4A02C8>=B8=C74BHBC4< 338C8>=0;;HB><4;40A=8=6C>>;BBD??>ACG)"4=01;8=6
;40A=8=6;>68=C46A0C8>=0=32A>BB ?;0C5>A<0=0;HB8B)&0=60A>B:04C0;   )
F8C7G)"8B4G?42C43C>F>A:0B0=0;C4A=0C8E4B>;DC8>=5>A<D;C8<>30;B4=B>AB 
8E4=C78B102:6A>D=3C78BBCD3H4G?;>A43C74?>BB818;8CH>5DC8;8I8=6308;H;40A=8=6;>6
30C0A42>A3438=G)"5>A2;0BB02C8E8CHE8BD0;8I0C8>=8=C40278=60=0;HC82B /4E8BD0;8I43C74
A4038=602C8E8CHF7827F0B34B86=43C>DC8;8I4B4E4A0;",C>>;B38BCA81DC4302A>BB<D;C8?;4
?;0C5>A<B0=34G0<8=43C748=B867CBC70CC74E8BD0;8I0C8>=2>D;3?A>E834C>A45;42C>=0=3
A4E8B4C742;0BB02C8E8CH34B86= ,74A4B40A27@D4BC8>=B5>AC78BBCD3H0A40B5>;;>FB•  ,>F70C4GC4=C20=C74308;H;40A=8=6;>68=G)"5>A<0C20?CDA42;0BB02C8E8C84B5>A
C40278=60=0;HC82B
 
 !  ##$
,78BBCD3HC0A64C43;40A=8=602C8E8C84BC70C703C0B:B02A>BB<D;C8?;4?;0C5>A<B %40A=8=6
02C8E8CHE8BD0;8I0C8>=F0B?4A5>A<43DB8=6C74;40A=8=6;>630C02>;;42C43=0CDA0;;H3DA8=6C74
;40A=8=602C8E8CH 

548

86DA4 ?A4B4=CB0=>E4AE84F>5C78BBCD3H %%40A=8=60=3E834=24=0;HC82B
A0<4F>A:8BC74"=C46A0C43%40A=8=6=E8A>=<4=C8=F7827C74;40A=8=602C8E8C84B8=C78B
BCD3HF4A42>=3D2C43 ,74%BHBC4<2>=B8BCB>50;40A=8=6<0=064<4=CBHBC4<%&+
&>>3;44 >>: >>:*>;;;40A=8=6A42>A3BC>A4%*+0=3;40A=8=60=0;HC82BC>>;B;>6
?0;4CC4B(60C04C0;   ;;;40A=8=6;>6B5A><C74B4?;0C5>A<B0A4BC>A438=>=4%*+
8=C74G)"5>A<0C0=3;>6B20=14A42>A343DB8=60D=85843--">=C74%*+     
# 
# 






                   !"
$"

 
  #     

%
  !   

" .8BD0;8I0C8>=&4C7>35>A ;0BB2C8E8C84BF8C7 A>BB ?;0C5>A<%>60C0
0B43>=2A>BB ?;0C5>A<8=C46A0C43;>630C0F4E8BD0;8I43C74;40A=8=602C8E8C84B>50
2;0BB ,74;4BB>=?;0=0=3E8BD0;8I0C8>=F4A4B7>F=C>0;40A=8=634B86=4A0=30;40A=8=6
0=0;HC82B A4B40A274AF7>34B86=43C74A4038=602C8E8CHD=8C 0=3F0B0B:43F70CC74H
A40;8I435>AA4E8B8=6C74;40A=8=634B86=0=3;4BB>=?;0= !%!

,78B BCD3H 5>2DB4B>=0#0?0=4B49D=8>A7867B27>>;=6;8B72;0BBF8C7C745>;;>F8=62;0BB
34B86= ,74C0A64C2;0BBDB430C427=8@D420;;4330C0 4=70=24302C8E4A4038=6 ,>H>:0F0
4C0;  F78278B0A4038=62;0BBC70CDB4B4 >>:B0=3;40A=8=60=0;HC82BC>>;C>38B?;0H
;>6B 86DA4 B7>FBC74;4BB>=?;0= 0=3C74;40A=8=6?;0C5>A<B DB43>=C70C30H
 









    

 


  





 
    









 

                    
 


 
        
 
     " 4C08;43%4BB>=);0=B 0=3-B43%40A=8=6,>>;B,74;4BB>=?;0=BC0AC43F8C7A>DC8=402C8E8C84B382C0C8>= <8= A4038=64C25>;;>F43
1H0?A4 @D8I ,78BF0B5>;;>F431H?A4 A4038=6F78272>=B8BC43>5A4038=6C74C4GC0=3
FA8C8=66D4BB4B0=3@D4BC8>=B8=C744 >>:<4<> ,74=0B0)08A/>A: 02C8E8CH
02C8E8C84BF4A4?;0==43C>0224BBC7430B71>0A30=3B70A46D4BB4B0=3@D4BC8>=BF8C7
?44AB1H2742:8=6C7430B71>0A3 "=)A4 A4038=6BCD34=CBF4A44G?42C43C>A403C74
?0BB064B8;4=C;H0=3A42>A3C748AA4038=6C8<4F8C7C744 >>:C8<4A ,74HC74=0224BB43C74
02C8E4A4038=630B71>0A30=3A468BC4A43F8C7C748AF>A3B ?4A <8=DC4 /)& '4GCC74H

549

A4A403C74C4GC0=37867;867C43D=:=>F=F>A3B0=34BB4=C80;?>8=CBF8C7C744 >>:<0A:4A 8=0;;H0B8=)08A/>A:C7430B71>0A3F0B0224BB430=3D=:=>F=F>A3B0=34BB4=C80;
?>8=CBF4A438B2DBB43F8C7?44AB1HA42742:8=6C7430B71>0A3
B34B2A814301>E4C74;42CDA4DC8;8I430E0A84CH>5;40A=8=6?;0C5>A<B0=3F0B
34B86=43C>2><18=48=38E83D0;0=32>;;01>A0C8E402C8E8C84B0;C4A=0C4;H DAC74AC78B2;0BBF0B
=>C34B86=435>AC748=BCAD2C>AC>68E40;42CDA48=5A>=C>5C74BCD34=CB1DCA0C74A5>AC74
BCD34=CBC>?A>2443F8C7C7402C8E8C84BC74<B4;E4B022>A38=6C>C748AC40274AB8=BCAD2C8>=B

 !# !#!
86DA4 B7>FBC748=C4A5024B>5C74C7A44?;0C5>A<BDB438=C74;40A=8=602C8E8C84B /74=
DB4AB2;82:40271DCC>=>A;8=:C74;40A=8=6;>68BBC>A438=C74%*+0;>=6F8C7C74E4A1B
B7>F=>=C74B834 .8BD0;8I0C8>=F0B?4A5>A<4310B43 >=C74;>6BBC>A438=C74 %*+ CC78B
C8<4C><0:4C7402C8E8CH6A0=D;0A8CHBD8C01;45>AC74E8BD0;8I0C8>=>5C78B;4BB>=?;0=B><4
E4A1BF4A4039DBC430B6A>D?43>A38E83438=C>B4E4A0;02C8E8C84B 0BB7>F=8=86DA4   


  



    

   



 






  

0%40A=8=6&0=064<4=C+HBC4<    
     








 

 







 




     



 
   

14 >>:       
     



           2%40A=8=6=0;HC82B,>>;B" ,74"=C4A5024B >5%40A=8=6);0C5>A<B0=3.4A1B

550



  
86DA4B7>FBC7402C8E8C84B3DA8=6C74 <8=2;0BB066A460C434E4AH<8=DC4 ,74B8I4>5
40273>CA4?A4B4=CBC74=D<14A>502C8E8C84B ,74E4AC820;0G8BA4?A4B4=CBC74;40A=8=6
02C8E8C84BB7>F=8=,01;4 0=3C747>A8I>=C0;0G8BA4?A4B4=CBC8<4 02724;;8=C74?;>CB
A4?A4B4=CB0BCD34=CNB02C8E8C84B0=3C741>CC><24;;B7>FBC74C40274ANB02C8E8C84B D4C>
B?024;8<8C0C8>=B>=;H ;40A=4AB>DC>5BCD34=CBF4A4A0=3><;HB4;42C435>A86DA4 "%40A=8=62C8E8CH.8BD0;8I0C8>=5A><"=C46A0C43G)"%>60C0  $!%!!% !!"



!#! 0B43>=C74B4E8BD0;8I0C8>=B8=86DA4F458ABC0=0;HI43C>F70C4GC4=CG)";40A=8=6;>6
30C020=20?CDA40=3E8BD0;8I42;0BB02C8E8C84B -41&#!*00./-!##"#"
,74?;>CB8=C74586DA48=3820C4F7>DB43C74C>>;F74=0=37>F<0=HC8<4B ,7DBC78B
E8BD0;8I0C8>=20=?A>E8340=>E4AE84F>5C742;0BBNB?A>6A4BB ,74A4BD;CBB7>F43C74>A34A
8=F7827C74C>>;BF4A4DB43C7438554A4=24B8=C>>;DB4C8<8=60=3C74=D<14A>5C8<4B8C
F0BDB4334?4=38=6>=C74?4AB>= >A4G0<?;4F8C7C748=C46A0C43G)";>6B20?CDA8=6
0=3?>ACA0H8=67>FC742;0BB02CD0;;H?A>2443430;<>BC0;;BCD34=CB8=C78B2;0BB58ABCDB43
C744 >>:1;D43>CB5>;;>F431HC7430B71>0A36A44=3>CB0=3C74%&+A433>CB0C
C744=3>5C742;0BB "=0338C8>=C78BE8BD0;8I0C8>=20?CDA4BE0A80C8>=B8=7>F<0=HC8<4B
B8I4>53>CB0=37>F<D27C8<4F83C7>50B4A84B>53>CB4027BCD34=C?4A5>A<4302C8>=B #3'1'-,$/-+1&#*#00-,.*,"2/',%1&#!12*!1'3'16
><?0A8=6C74?;0=0=3C74F7>;42;0BB02C8E8C84BC74%&+02C8E8C84BA4?A4B4=C431HC74A43
3>CBF4A4?;0==430CC741468==8=6>5C742;0BB1DCF4A402CD0;;H2><?;4C430CC744=3>5
C74;4BB>= ,78B20=148=C4A?A4C430BC74C40274A270=68=6C74;4BB>=?;0= "=0338C8>=8C
F0B?;0==43C70CC74BCD34=CBF>D;3DB4C74A42><<4=30C8>=8=C744 >>:C>6D834C748A
0224BBC>C7430B71>0A31DC=>>=4DB438C ,78B2>D;3141420DB4C748=BCAD2C8>=B5>AC74
A42><<4=30C8>=5D=2C8>=F4A4=>C2;40A0BC74H383=>CA40;8I4C745D=2C8>= "=C78BF0HC78B
E8BD0;8I0C8>=20=0;B>0BB8BC8=0BB4BB8=67>FF4;;C74;4BB>=?A>6A4BB430B?;0==43  &#1&#/1&#012"#,10!-2*"$-**-41&#!1'3'16,"1--*1/,0'1'-,0*460A38=6C7402C8E8CHF8C7C7430B71>0A3A4?A4B4=C431HC746A44=3>CBB><4BCD34=CB
0224BB438CC7A44C8<4B0B?;0==431DC>C74AB0224BB438C>=;H>=240=3054F383=>C


551



0224BB8C4E4=>=24 (=4>5C74A40B>=B5>AC78B2>D;314C70CC7430B71>0A3F0B=>C0
5D=2C8>=8=C744 >>:1DCF0B>=;40A=8=60=0;HC82BC>>;B038554A4=C?;0C5>A<;0D=27435A><%&+ ,7DBBCD34=CB5>D=38C270;;4=68=6C>0224BB8C "=0338C8>=C7402C8E8C84BDB8=6C74
30B71>0A3F4A42>;;01>A0C8E4C74A45>A4C74A4F4A47DA3;4B "=>C74AF>A3BC78B
E8BD0;8I0C8>=20=4G?;08=C74385582D;CH>5CA0=B8C8>=B0<>=6C>>;B0=302C8E8C84B  #!&#/0 #&3'-/,""#3'1'-,$/-+1&#*#00-,.*,,"012"#,108!1'3'16
8=0;;H4E4=C7>D67C40274AB0224BB430;;C74%&+4 >>:B0=3;40A=8=60=0;HC82BC>>;B
C74C8<8=6383=>C=424BB0A8;H<0C27C74;4BB>=?;0=0=3C70C>5C74BCD34=CB "=0338C8>=C74
C40274A0224BB43C74@D8I5>A%&+A433>CB0=3C7430B71>0A35>A;40A=8=60=0;HC82BC>>;B
6A44=3>CB0=3>=;HC74A4038=60=3C8<4A5>AC744 >>:1;D43>CB ,7DBF45>D=3C70C
4E4=C7>D678C8BC40278=60=0;HC82B8C8B=424BB0AHC>?0H0CC4=C8>=C>C74;>6B>5C74C40274A
0=3C74;40A=4A4B?4280;;H8=C7420B4>502;0BB34B86=438=BD270F0HC70CBCD34=CB
34E4;>?02C8E8C84B022>A38=6C>C74C40274ANB8=BCAD2C8>=BA0C74AC70=148=6;431HC74C40274A  ! !!# "&!#!#!!
 

+42>=3F4B7>F43C74B4E8BD0;8I0C8>=BC>0A4B40A274AF7>34B86=43C78BA4038=602C8E8CH
0=30B:4374AF70CB74=>C8243 /40;B>0B:43F74A40=37>FB742>D;38<?A>E4C74
;4BB>=?;0=0=3;40A=8=634B86=   /70CC74;40A=8=634B86=4A=>C82435A><C74E8BD0;8I0C8>=
,#5.#!1#",2+ #/-$.%#1/,0'1'-,0,"-.#/1'-,0,74;40A=8=634B86=4A?>8=C43>DCC70CC74=D<14A>5A4038=60=3C8<4A;>6BF0B78674AC70=
4G?42C43 ,74A4038=6<0C4A80;2>=B8BC43>5>=;H5>DA?064B4027>5F78272>AA4B?>=343C>
C7402C8E8C84B8=C74;4BB>=?;0=0BB7>F=8=86DA4 /74=8CF0B34B86=438CF0B
BD??>B43C>14A403>=;H>=24?4A?0645A><C745A>=C !>F4E4AA4038=6;>6B0??40A438=0
10=3B70?48<?;H8=6C70C?064CA0=B8C8>=B0A4?4A5>A<43<D;C8?;4C8<4B ,7434B86=4A
<4=C8>=43CF>?>BB81;48=C4A?A4C0C8>=B>=48BC70CBCD34=CBF>A:4338;864=C;H0=3C74>C74A
8BC70CC74<0C4A80;F0B=>C40BHC>5>;;>F0=3A4@D8A43A4E84FB>5C74?A4E8>DB?064B 2!&1'+#0.#,1-,',"'3'"2*!1'3'16,"*#00!-** -/1'3#!1'3'1'#0
,7434B86=4A0;B>5>2DB43>=C7438554A4=2414CF44=4 >>:0=330B71>0A3>?4A0C8>=;>6B >=B834A8=6C70CC744 >>:A45;42CB8=38E83D0;02C8E8C84B0=3C7430B71>0A3A45;42CB
2>;;01>A0C8E402C8E8C84B8=C78B34B86=B74A45;42C430B5>;;>FB"0<70??HC74HF>A:4370A3
>=4 >>:02C8E8C84B1DC8=38E83D0;02C8E8C84B<0H70E4144=<>A4385582D;CC70="4G?42C43 "F>D;3;8:4C><0:42>;;01>A0C8E402C8E8C84B<>A402C8E40=3<>385HC74?;0==>CC>5>2DB>=
8=38E83D0;02C8E8C84B>E4A;H M   /70CBD664BC8>=BC7434B86=4A7035>A8<?A>E8=6C74;4BB>=?;0=0=3;40A=8=634B86= !1'3'16"#0'%,'+./-3#+#,1 1/,0'1'-, #14##,!1'3'1'#0,".*1$-/+0+74=>C43C70CC7434B86=43;42CDA470302C8E8CHBF8C278=60=3<>E4<4=C14CF44=
?;0C5>A<B4B?4280;;HF8C7B><4;40A=4AB=>C?4A5>A<8=6C7402C8E8C84B0B?;0==43C7DBB74
F>D;3;8:4C>A43D24102:0=35>AC714CF44=02C8E8C84B0=3?;0C5>A<BC>A4<43HC74?A>1;4< B02>=2A4C4<40BDA4B74F>D;3B8<?;85HC7402C8E8C84BB;867C;H<>A4F78;4148=620A45D;=>C
C>;>B4C74?DA?>B4>5402702C8E8CH 601#+"#0'%,'+./-3#+#,1 %1&#/1&#$2,!1'-,0$-/-,#!1'3'16',-,#.*1$-/+
+740;B><4=C8>=43C74BHBC4<8<?A>E4<4=C B<4=C8>=438= C7430B71>0A3F0B
=>C05D=2C8>=8=C744 >>:0=3B><4BCD34=CB0224BB438C>=;H>=24>A383=>C0224BB8C
4E4=>=24 0B43>=C78BB74B083C70C85C7430B71>0A32>D;314E84F435A><F8C78=C744
>>:C74=D<14A>5BCD34=CBF7>2>D;3E84FC7430B71>0A3F>D;38=2A40B40=3
2>;;01>A0C8E402C8E8C84BF>D;314<>A402C8E4 


552



 
"=C78BBCD3HF4B4CC74A4B40A27@D4BC8>=0BL,>F70C4GC4=C20=C74308;H;40A=8=6;>68=
G)"5>A<0C20?CDA42;0BB02C8E8C84B5>AC40278=60=0;HC82BM,>0=BF4AC74@D4BC8>=F4
E8BD0;8I43C74A4038=602C8E8CHF7827F0B34B86=43C>DC8;8I4B4E4A0;",C>>;B02A>BB<D;C8?;4?;0C5>A<B0=34G0<8=43C748=B867CBC70CC74E8BD0;8I0C8>=20=?A>E8345>AA45;42C8=6>=0=3A4E8B8=6C7402C8E8CH34B86= ,74A4BD;CBB7>F43C70CC748=C46A0C43G)";>6B2>D;3E8BD0;8I4F74C74AC74A4F4A4
0=H38554A4=24B8=C7402CD0;02C8E8C84B2><?0A43F8C7C74;4BB>=?;0=0=37>F02C8E4C74
;40A=4ABF4A4 "=0338C8>=C74;40A=8=634B86=4A<4=C8>=4374A34B8A4C>270=64C7402C8E8CH
34B86=C>A43D24C74CA0=B8C8>=>502C8E8C84B3DA8=62;0BB 0B43>=C74B4A4BD;CBF4
2>=2;D343C70CC74308;H;40A=8=6;>630C0A42>A3438=C74G)"2>D;3E8BD0;8I4C74;40A=8=6
02C8E8C84B5>AC40278=60=0;HC82B ,78BBCD3H703B><4;8<8C0C8>=B 8ABC>DA<4C7>320?CDA430=3E8BD0;8I430=4=C8A4
2;0BB02C8E8CH0=32><?0A438CF8C70;40A=8=634B86= !>F4E4A8CB7>D;314=>C43C70CC74
C0A64C2;0BBF0B34B86=43F8C702C8E8C84BC70CDB43C>>;B5A4@D4=C;H >A4G0<?;48502;0BB
DB4B>=;H4 >>:BF8C754FC>>;B>A02;0BBC70C3>4B=>CDB4C>>;B0C0;;C74<4C7>3
20?CDA4B>=;H0?>AC8>=>5C7402C8E8CH +42>=38=C78BBCD3HF4?A4B4=C43C74E8BD0;8I0C8>=
A4BD;CBC>C74;40A=8=634B86=4AC>B447>FC74HF>D;38=C4A?A4CC74< !>F4E4AF4F>D;3
;8:4C>B7>FC78BC>C40274AB0=3>1B4AE47>FC74H8=C4A?A4CC74E8BD0;8I0C8>= "=0338C8>=
C40274AB3>=>C=424BB0A8;H70E4BD558284=C30C0;8C4A02HC>8=C4A?A4C;40A=8=60=0;HC82BA4BD;CB'3D:F40=84;   ,>B>;E4C78B?A>1;4<5443102:B7>D;370E46D838=6?4AB?42C8E4B
8=BC403>5B8<?;H<8AA>A8=6C7402C8E8CH+>;;4A4C0;   
;C7>D67C78BA4B40A278BBC8;;8=0=4G?;>A0C>AH?70B48CB7>FBC74?>BB818;8CH>5
20?CDA8=60=3E8BD0;8I8=6C74?A>6A4BB>502;0BBDB8=6G)"8=BC403>5<D;C8?;4B4=B>AB  

,78BBCD3HF0BBD??>AC431H'(#)')
#)$ 0=3#)!   0=3#+)+$$'!" A0=C'D<14AB



7A7;     &-4C;9=-B      %-)+016/ 6)4A<1+; +=::-6< +0)44-6/-; )6, .=<=:-,->-4785-6<)&'%#!%""# #&  $%",  )6/):7;3)&-;161)66)37;  ):+0:7;;84)<.7:5)6)4A<1+;;<-8<7?):,;
8-:;76)41B)<176)6,),)8<)<17616-,=+)<176"%#"&#' '"'%"'#" #"%"#"
%"" " +'&"#* 88  ):<16-B)4,76),7#+0->-::1)&-:6)6,-B 1-<7=+316/0)5$0=5$  8:14
:75 ,)<) <7 16;1/0<;  4)A-:-, ;<7:A<-4416/ )88:7)+0 .7: 5=4<157,)4 4-):616/ )6)4A<1+;
6%#"&#'#"%"#"(!"'#%&"#!$('"&+&'!&88  ,=3?-)61-4  %-)+016/6)4A<1+;>)4=-)6,<774;.7:<-)+0-:,)<)41<-:)+A
;A;<-5)<1+ )6, <:18):<1<- )88:7)+0"'%"'#"  #(%"  # ('#"  "# #+ " %
('#"    !+07)')6/$1-5-6;  =4<157,)44-):616/)6)4A<1+;"### %""
" +'& !/)<)  )2=5,): # 3D)8G6):  );616-    4)6)/)6     -A76, -):616/6)4A<1+; :)5-?7:3 .7: %-+06747/A60)6+-, >1,-6+-);-, ,=+)<176 )6, -):616/ 16
":7+--,16/;7.<0- <06<-:6)<176)476.-:-6+-76758=<-:;16,=+)<176  ":1-<7"$0):5)1,B16;31I#7,:E/=-B%:1)6)144-6*7=:/"  =4<157,)4
%-)+016/ 6)4A<1+; =<75)<-, -@<:)+<176 7. 7:+0-;<:)<176 /:)80; .:75 ?-):)*4- ;-6;7:
,)<)#(%" ##!$('%&&&' %""     
$744-:):<E6-B-:5)66"=-04-6*:7+3  :7551::7:16/<7/=1,16/:->1-?
7.;<)<-7.<0-):<<-+06747/A.7:;=887:<16/+744)*7:)<1>-4-):616/"'%"'#" #(%" # %' 
"' "" ('#"    %7A73)?)()2=5,):#76,7%7:137;01!/)<)  +<1>-:-),16/);0*7):,16)-):616/6)4A<1+;-60)6+-,4)6/=)/-4-):616/-6>1:765-6<-..-+<;764-):616/*-0)>17:)6,
8-:.7:5)6+-#(%" # #!$('%&" ('#"  

553

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Effects of Audio and Tactile Biofeedback
Based on EEG Attention Levels on
University Students’ Relaxation
Shih-Jou YUa, Wai Ki Rebecca CHENGb, Yi-Hsuan CHENa & Jerry Chih-Yuan SUN a*a
Institute of Education, National Yang Ming Chiao Tung University, Taiwanb
University of the Arts London, UK* jerrysun@nycu.edu.tw
Abstract: This study aims to examine the effects of audio and tactile biofeedback
based on EEG attention levels in anti-phishing education on university students’relaxation. The study developed an attention feedback system to provide learners withaudio and tactile biofeedback by collecting learners’ EEG attention signals andconverting them to attention levels. The research method employed a quasiexperimental design. The participants were 90 university students who had no prioranti-phishing learning experiences. A random grouping was adopted to divide theparticipants into a non-immediate feedback group, an audio-immediate feedback
group, and a tactile-immediate feedback group. Each participant was required to wear
a portable EEG device that was connected to an attentional feedback system to collect
their EEG attention and relaxation signals during the learning activity. After the learning
activity, participants were asked to complete a post-activity feedback questionnaire.
The results showed that the tactile-immediate feedback group displayed a significantly
higher level of relaxation as compared to the audio-immediate feedback group. The
study suggests that instructors may consider using tactile-immediate biofeedback
based on learners’ attention levels to help regulate attention and improve relaxation in
online learning environments.
Keywords: Audio and tactile feedback, EEG biofeedback, relaxation, anti-phishing

1. Introduction
Remote online learning became a new norm in education after the COVID-19 pandemic in
March 2020. The impact has silently revolutionized the education system to embracetechnology-based cloud learning. However, this brings some challenges. According to thetrends report from the Anti-Phishing Working Group (APWG), there was a sudden increase inphishing attacks against videoconferencing service provider Zoom (APWG, 2020) where
schools worldwide were switching to video conferencing for their remote learning. Anothersurvey from the 2020 Cyber Threats Report (Netwrix, 2020) stated that 33% of educationalorganizations felt they were at greater cybersecurity risk than pre-pandemic, 89% of themfound new security gaps caused by the rapid transition to remote learning, and 50% of
educational organizations had experienced phishing attacks. Today, there is growing concern
about VPN exploitation and credential stuffing compared to pre-pandemic when malware orphishing was a major concern. Growing demand for cybersecurity awareness training will
become an important trend in K-12 schools and higher education institutions.
Previous studies showed that effective anti-phishing education is the key to preventing
phishing attacks (Sun et al., 2016). However, many of the anti-phishing materials highly relyon students’ self-learning. The complex digital worlds and other environmental distractions arecompeting with human’s limited attentional resources and increasing stress. To make selflearning effective, educational psychologists should accelerate the research on innovative554

approaches to assist students’ self-regulation of attention and motivation, as well as managing
stress in the midst of learning remotely and individually.
A branch of brain-computer interface (BCI) study has emerged that focuses on designing
attention-aware systems that assist users in effectively allocating their attentional focus to
optimize engagement (Vertegaal et al., 2006). Since the 1970s, BCI has been actively
employed as an assistive technology for clinical or therapeutic purposes, and has been applied
to “locked-in” patients or children with attention-deficit/hyperactive disorder (ADHD). In recent
decades, the development of Electroencephalography (EEG) and the advances in computing
power have provided a cost-efficient, safe, and portable approach which not only allows theuse of EEG data to understand users’ cognitive states, but also serves as immediate
communication with adaptive interfaces of visual, audio, or tactile feedback to influence andaugment cognitive functions (Tan & Nijholt, 2010; Vasiljevic & Miranda, 2023). Sun and Yeh(2017) used audio biofeedback based on EEG attention signals to improve learners’ attention,and showed that it was an effective strategy. The EEG biofeedback can help learners quickly
achieve deep relaxation (Rydzik et al., 2023). Xu and Zhong (2018) showed that whenlearners’ attention and relaxation levels are high, this state of mind can help them learn better.
Relaxation, also known as meditation (Xu & Zhong, 2018), can also reduce anxiety (Hardt,2012). Holmes (2019) suggested that understanding how to integrate neuroscience andmindfulness education to train self-regulation and match learners’ learning requirements is an
important issue. Therefore, the issue of how to balance attention and relaxation in the learning
process needs to be explored. Previous studies indicated that tactile feedback can improvenovices' relaxation in a digital environment, and does not affect attention (Kim et al., 2021) or
the performance of visual tasks (Alahakone & Senanayake, 2010). This study aimed todevelop an attentional feedback system that provides audio and tactile biofeedback based on
EEG attention levels to create a personalized learning environment. The study also examinedthe effect of audio and tactile biofeedback on university students' relaxation levels. The
following research question was addressed in this study and the research model is shown in
Figure 1.
Research question: Are there significant differences in relaxation among university students
using non-immediate, audio-immediate, and tactile-immediate feedback based on EEG
attention levels in online anti-phishing education?

Figure 1. The research model

2. Method
2.1 Participants
The study participants were 90 university students from Taiwan, excluding those who had priorlearning experience of anti-phishing. The effective sample comprised 90 (100%) students. The
participants were randomly divided into three groups: a non-immediate feedback group (n =30), an audio-immediate feedback group (n = 30), and a tactile-immediate feedback group (n= 30). There were 36 males (40%) and 54 females (60%), with an average age of 22.7 years

555and a standard deviation of 2.83. In terms of academic areas, 23 participants were from theCollege of Humanities and Social Sciences (25.56%), 13 were from the College of ElectricalEngineering (14.44%), 11 were from the College of Engineering (12.22%), 10 were from theCollege of Information Science and Technology (11.11%), 10 were from the College ofManagement (11.11%), nine were from the College of Science (10.00%), and 14 participants
were from other colleges (15.56%).

2.2 Experimental procedure
The informed consent process included the researcher explaining the content of the consent
form to the participants before the experiment. After the participants understood the content,
they voluntarily agreed and signed the consent form. Before starting the experiment,
participants needed to put on the portable EEG device for approximately one minute andconfirm that it did not make them uncomfortable. In the next step, researchers took the portable
EEG paired with the attentional feedback system and tested the biofeedback for approximately
two minutes. When the attentional feedback system was completely prepared, participants
could start the online anti-phishing learning activity. The learning activity lasted approximately
30 minutes. The EEG attention and relaxation signals were recorded during the learning
activity for the three groups. The non-immediate feedback group did not receive any sensory
feedback based on their mental attention states. The audio-immediate feedback group
received audio biofeedback when their attention level fell below the threshold of focused
attention state. The tactile-immediate feedback group received tactile biofeedback when theirattention level fell below the threshold of focused attention state. After the learning activity,
participants were asked to complete a post-activity feedback questionnaire, which tookapproximately 10 minutes. The experimental flowchart is shown in Figure 2.

Figure 2. The experimental flowchart556

2.3 Instrument
2.3.1 The attentional feedback system
The attentional feedback system was designed and planned by researchers and developedby a technology company. This system was a mobile application based on Android
development platforms, and the mobile device used in this study was a Samsung Galaxy S8smartphone. The attentional feedback system was connected to the portable EEG device byBluetooth to collect EEG attention and relaxation signals. It provided attention biofeedback
based on learners’ attention levels. The attention levels were calculated by the eSense
algorithm and the ThinkGear chip from NuroSky (2017). The ThinkGear chip enhanced theraw EEG signals and filtered noise and muscle movement interference. The eSense algorithm
converted the raw EEG values into relative attention levels ranging from 0 to 100 to describe
the range of brain activity. Besides 0 meaning the signals were unable to be used, other scores
were divided into five ranges to determine the attention levels: 1 to 20 indicated strongly
lowered levels, 20 to 40 indicated reduced levels, 40 to 60 indicated neutral levels that weresimilar to the baseline, 60-80 indicated slightly elevated levels, and 80 to 100 indicatedelevated levels. In this study, the biofeedback triggered audio or tactile feedback when theattention scores were below 40 and lasted for more than seven seconds. The system
controlled the interval between two instances of feedback for at least 30 seconds. This study
also used this system to collect relaxation signals and used the same method to estimate therelaxation levels. The user interface and design of audio and tactile biofeedback are shown in
Figure 3.

Figure 3. The design of audio and tactile biofeedback
2.3.2 The post-activity feedback questionnaire
The post-activity feedback questionnaire was modified from Sun and Yeh (2017). The
questionnaire includes three issues: use experience, timing and perception of receiving
attention feedback, and other perceptions. It aimed to understand learners’ perceptions ofusing the attentional feedback system, and collected their suggestions to improve the designof the learning activity and the biofeedback system in the future. The non-immediate feedback557

group had five items in their questionnaire, whereas both experimental groups, the audioimmediate feedback and the tactile-immediate feedback group, had eight items respectively.

3. Results
This study used the analysis of variance (ANOVA) to analyze the results by the IBM SPSS
Statistic 25 software. The sample conformed to the normal distributions based on the criterionof Kline (2011) and did not reject the hypothesis of the equality of variances by Levene’s test
(F(2,87) = 1.07, p = .35). According to Table 1, the ANOVA results indicated that the relaxationlevels were significantly different among the three groups (F = 3.70, p < .05, η2 = .08). Because
the sample sizes of each group were equal, we chose Tukey’s HSD test to analyze the posthoc comparisons. The results of Tukey’s HSD’s post-hoc comparison show that the relaxationlevel of the tactile immediate feedback group (M = 40.72, SD = 16.61) was significantly higher
than that of the audio immediate feedback group (M = 30.41, SD = 13.70), indicating thatlearners who used tactile biofeedback based on EEG attention levels could improve theirrelaxation levels compared to audio biofeedback. Table 1 shows the descriptive statistics forrelaxation and Table 2 shows a summary of ANOVA for relaxation for the three feedback
groups.
Table 1. Descriptive statistics for relaxationMSD

skewnesskurtosis

38.77

16.27

-0.03

-1.02

30.41

13.70

0.94

0.90

40.72

16.61

0.35

-0.54

Group
Non-immediate feedback group(n = 30)
Audio-immediate feedback group(n = 30)
Tactile-immediate feedback group(n = 30)
Table 2. A summary of ANOVA for relaxation
Source of variation

SS

df

MS

F

p

2
η

Post hoc

Between-group

1798.312

899.16

3.70

< .05

.08

(C) > (B)Within-group

21122.87

87

242.78

Total

22920.18

89

Note. (A) indicates the non-immediate feedback group; (B) indicates the audio-immediate
feedback group; (C) indicates the tactile-immediate feedback group

4. Discussion and conclusion
According to the results, the relaxation levels were significantly different among the threefeedback groups, with the tactile-immediate feedback group displaying higher levels of
relaxation as compared to the audio-immediate feedback group. Relaxation can reduce
anxiety (Hardt, 2012) and stress, and improve learners’ learning process (Holmes, 2019).
Therefore, tactile biofeedback was a helpful strategy to remind learners to focus their attention
levels without increasing their anxiety or stress, as a higher relaxation level resulted.
Additionally, tactile-immediate feedback helps learners regulate their behavior and avoid
competing with visual tasks (Alahakone & Senanayake, 2010). The result is also consistent
with Kim et al. (2021), who found that tactile feedback can help people adjust their breathing
to improve attention, while audio feedback interferes with attention. Participants’ written558

feedback aligned with the findings. Examples are: “When the smartphone vibrates, it reminds
me to focus on the content on the screen” (Tactile 27); “When the vibration occurs, I will recall
my status of attention” (Tactile 10); “The sound playing may startle and cause distraction”(Audio 29); and “The sound is very harsh” (Audio 28). Therefore, this study suggests that
instructors can use tactile feedback based on learners' EEG attention levels to enhance theirrelaxation levels. Future research can extend to examining learners’ preferences for volume
and melody, negative reinforcement issues, and different tactile patterns.ġ

Acknowledgements
This research was supported by the National Science and Technology Council (formerly
Ministry of Science and Technology) in Taiwan through Grant numbers MOST 111-2410-HA49-018-MY4, MOST 110-2511-H-A49-009-MY2, and MOST 107-2628-H-009-004-MY3.
We would like to thank NYCU’s ILTM (Interactive Learning Technology and Motivation, see:
http:// ILTM.lab.nycu.edu.tw) lab members and the students for helping conduct theexperiment.

References
Alahakone, A. U., & Senanayake, S. M. N. A. (2010). A real-time system with assistive feedback forpostural control in rehabilitation. IEEE/ASME Transactions on Mechatronics, 15(2), 226-233.
https://doi.org/10.1109/TMECH.2010.2041030
APWG. (2020). Phishing activity trends report Q1 2020.
https://docs.apwg.org/reports/apwg_trends_report_q1_2020.pdf
Hardt, J. V. (2012). Alpha brain-wave neurofeedback training reduces psychopathology in a cohort ofmale and female Canadian aboriginals. Advances in Mind-Body Medicine, 26(2), 8-12.
Holmes, K. (2019). Neuroscience, mindfulness and holistic wellness reflections on interconnectivity inteaching and learning. Interchange, 50(3), 445-460.
Kim, E., Heo, J., & Han, J. (2021). Attention to breathing in response to vibrational and verbal cues in
mindfulness meditation mediated by wearable devices. In M. Kurosu (Ed.), Human-computerinteraction. Interaction techniques and novel applications (Vol. 12763, pp. 415–431).
Springer. https://doi.org/10.1007/978-3-030-78465-2_31
Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). The Guilford
Press.
Netwrix. (2020). 2020 Cyber threats report. https://www.netwrix.com/2020_cyber_threats_report.html
NuroSky. (2017). ThinkGear Serial Stream Guide.
https://developer.neurosky.com/docs/doku.php?id=thinkgear_communications_protocol&s[]=thinkgearRydzik, Ł., Wąsacz, W., Ambroży, T., Javdaneh, N., Brydak, K., & Kopańska, M. (2023). The use of
neurofeedback in sports rraining: Systematic review. Brain Sciences, 13(4), Article 660.
https://doi.org/10.3390/brainsci13040660
Sun, J. C.-Y., & Yeh, K. P.-C. (2017). The effects of attention monitoring with EEG biofeedback on
university students' attention and self-efficacy: The case of anti-phishing instructional
materials. Computers & Education, 106, 73-82.
https://doi.org/10.1016/j.compedu.2016.12.003
Sun, J. C.-Y., Yu, S.-J., Lin, S. S. J., & Tseng, S.-S. (2016). The mediating effect of anti-phishing selfefficacy between college students' Internet self-efficacy and anti-phishing behavior andgender difference. Computers in Human Behavior, 59, 249-257.
http://dx.doi.org/10.1016/j.chb.2016.02.004
Tan, D. S., & Nijholt, A. (2010). Brain-computer interfaces: Applying our minds to human-computerinteraction. Springer.
Vasiljevic, G. A. M., & Miranda, L. C. d. (2023). The CoDIS taxonomy for brain-computer interface
games controlled by electroencephalography. International Journal of Human–Computer
Interaction. https://doi.org/10.1080/10447318.2023.2203006
Vertegaal, R., Shell, J. S., Chen, D., & Mamuji, A. (2006). Designing for augmented attention:
Towards a framework for attentive user interfaces. Computers in Human Behavior, 22(4),771-789. https://doi.org/10.1016/j.chb.2005.12.012
Xu, J., & Zhong, B. (2018). Review on portable EEG technology in educational research. Computersin Human Behavior, 81, 340-349.

559

Estimating Physical Interactions with
Neighboring Student for Detecting Active
Learners in the Computer Classroom
Takahiro YOSHINOa*, Shin UENOa & Hironori EGIa
Department of Informatics, The University of Electrical-communications, Japan
y2330128@edu.cc.uec.ac.jpaAbstract: In the classroom, some students may isolate themselves, hesitant to seek
help due to psychological barriers. These students need active addressing from
Teaching Assistants (TAs). Especially in computer classroom due to the types of study.
However, there are cases where some classes are arranged with only a few TAs for alarge number of students, and it may cause that students who should be approached
may be overlooked. In order to support isolated learners, this study estimates thephysical interaction with neighboring student based on the learner video in a computerclassroom. Analysis of learner video reveals the possibility of using a depth camera toseparate movements and estimating interactions. This result can be applied to the TA
support system which detects isolated learners.
Keywords: Student interaction, computer classroom, depth camera, learner video

1. Introduction
Teaching Assistants (TAs) are introduced to support classes in universities. One of the rolesof TAs in the classroom is to provide support to students. However, TAs do not have much
experience about education (Luo, Bellows, & Grady, 2000). They may overlook students whoshould be assisted. Some students are also hesitant to seek assistance due to psychologicalbarriers (Price et al., 2017). A proactive approach from TAs is needed for these students.
A more proactive approach is taken by implementing the TA supporting system(Imamura et al., 2020). Support from TAs is not necessary when students can resolve issues
among themselves. Detecting student isolation would allow for more appropriate support.
Many supporting systems in programming classes use learning results and programmingcode. It is hard to measure student behavior which do not show up in the code. This study
considers shooting from above the classroom. Additionally, a depth camera is used to capture
the student behavior. As a measure of student isolation, this study estimates physicalinteractions with neighboring student.

2. Estimating interaction with neighboring students
This study focuses on classes using computer classroom. In the computer classroom,a display is equipped in each student's seat. In order to avoid being behind the display, thisstudy take pictures from above the classroom without any obstacles. Figure 1 shows images
that can actually be taken in the computer classroom.

560



Figure 1. RGB image from above the classroom (left) and depth image (right)
By using a depth camera, the distance from the camera can be obtained as depth data.
When shooting from above, changes of depth data show vertical movement. Using this, thesystem can separate movements that take place directly above the desk, such as
keyboarding, from those that take place at eye level, such as pointing at a display or looking
into a neighbor’s display.

3. Analytics method
If changes of depth data can be captured, it may be possible to separate movements justabove the desk, movements at shoulder height, and movements where the head changes
position. In this study, estimating physical interactions is performed using the separation ofthese movements and the setting of the calculation area.
To capture the movement of students, the value of change in depth data is calculated
between each frame per pixel. To separate movements, it is counted the number of pixels inthree ranges of the obtained value of depth data changes. In order to capture the movement
just above the desk, the movement of the height near the desk, and the movement of thechanging head position, each range are defined and the value of change is counted as30mm to less than 140 mm for "low," 140 mm to less than 400 mm for "middle," and 400 mm
or more for "high”. The increase in the number of pixels in each range confirms theoccurrence of the movement.
As Figure 1 shows, the video data analyzed in this study shows two student side byside. Analyze one student by splitting the area into two parts, left and right. Physicalinteractions with neighboring student is estimated by some kind of change in the area between
students. For example, point to the next display or move body closer to the next seat. These
interactions are analyzed by calculating the space between displays.

4. Result
Learner videos ware collected in a programming class at a science and engineering university.
Each class were 90 minutes long. Each student can use the computers equipped in thecomputer classroom. The video captures the actual student behavior. Intel RealSense L515
was used to collect depth video. Data from 15 students were collected in six times of classes
in total. Then, analysis was performed on each video data. The analysis results are shown
based on the fourth experimental session in which physical interactions with neighboring
student were confirmed.

Figure 2. Calculation results561

Figure 2 shows the results of the calculation for one learner. The vertical axis shows thepercentage of pixels and the horizontal axis shows the time. In the graph, “low” is represented
by red, “middle” by yellow, and “high” by blue.
There is a notable area around 20m50s or 22m30s in the graph. The ratio of “high” does
not increase relative to the increase of the ratio of “low” in this range. In this moment, thestudent was reaching for the mouse. Another notable area is around 24m40s or 25m25s inthe graph. As the ratio of “low” increases, the ratio of “middle” and “high” increases. In thismoment, the student was moving all of body like a postural change. Moreover, the othernotable areas are the ranges from around 22m45s to 24m15s and from around 24m35s to
25m20s. The ratio of low is continuously changing in that range. In this time, the student was
keyboarding while his face and body turned in the direction of the display.
The calculation results for the other experimental sessions confirmed the same results.
At times when the ratio of “high” changed significantly, movements such as postural changes
or changing the position of the chair were confirmed. During time when the ratio of “low” waspresent in a continuously changing, movements such as keyboarding and checking textbookson the desk were confirmed.
About calculation results in the space between displays, there were four areas where
the “high” ratio intermittent changed. Two of these scenes had one student pointing at theother's display like figure1 (left). As for the other two parts, they were the scene of taking aseat at the beginning of the class and the scene of getting out of their seats at the end of theclass. When interactions were not confirmed, the ratio of “high” has not changed. These
range changed only the ratio of “middle” or "low".

5. Conclusion
This study analyzed actual student movement during class by using depth video from above
the classroom. The value of change is calculated in depth information between each frame
per pixel. Ratios of depth data change were calculated for each calculation area. Ratiochanges compared to actual student movement.
Analysis shows that depth data can be used to separate movements. Especially, it has
possibility to separate movements just above the desk, movements at shoulder height, andmovements where the head changes position. By setting the calculation area appropriately,
depth data may be used to estimate physical interactions with neighboring student. However,the data with interactions is few. In addition to that, it cannot be estimated in complex
situations. For example, it was observed that students were not analyzed because of going
through the calculation area.
In order to estimate the isolated learner, the system has to capture more clearly the
movement of individual students. For example, determine the head position and the bodyorientation by using depth data. We are planning to develop a system to guide TAs to active
support for isolated learner.

References
Imamura, R., Terui, Y., Yokoyama, Y., & Egi, H. (2020). Evaluation of the Learning Support Strategies
for Teaching Assistants. In 2020 IEEE International Conference on Teaching, Assessment, andLearning for Engineering, 120-126. IEEE.
Luo, J., Bellows, L., & Grady, M. (2000). Classroom management issues for teaching assistants.
Research in Higher Education, 41, 353-383.
Price, T. W., Liu, Z., Cateté, V., & Barnes, T. (2017). Factors influencing students' help-seeking behavior
while programming with human and computer tutors. In Proceedings of the 2017 ACM Conference
on international computing education research, 127-135.

562

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Effect of Active Breaks during e-Learning
and Mental Arithmetic Tasks
Masaki KODAIRAa*, Tatsuya HAMADAa & Hironori EGIa
Department of Informatics, The University of Electro-Communications, Japan*k2330044@edu.cc.uec.ac.jpaAbstract: In this study, we analyzed the effects of taking active breaks during mentalarithmetic and e-Learning tasks. Active breaks are defined as breaks that involve light
exercise. An experiment was conducted to analyze the effects of active breaks onrecovery from fatigue and the effects on the task results. Here, each subject’s fatigue
was analyzed using the leg movement measurement method. As a result, in the eLearning task experiment, recovery from fatigue was observed after active breaks.
However, improvement in learning effectiveness was not observed. In the mentalarithmetic task experiment, recovery from fatigue was also observed after active
breaks. In addition, it is possible that the recovery from fatigue enabled the subjects tomaintain a high rate of correct responses to the mental arithmetic task. It wassuggested that active breaks may be more effective in improving work efficiency when
the cognitive load of the task is greater.
Keywords: Active breaks, leg movement measurement, educational support system,
fatigue estimation

1. Introduction
In recent years, due to the influence of COVID-19, an increasing number of students arelearning at home. In this context, students are expected to take appropriate breaks on theirown, rather than at set times as in school. However, it is difficult to assess fatigue objectively
to determine weather it is appropriate to take a break. A previous study suggested appropriate
break timing based on students' learning performance (Ramachandran, Huang, & Scassellati,2017). and the results of that study demonstrated that individually determined break times
improved test scores compared to students who took breaks at predetermined times. The
study included various breaks, such as tic-tac-toe, stretch, and breathing, however, the effectsof the content of these breaks were unclear. Thus, we considered that task performance may
differ depending on the student’s behavior during a break. In this study, we focused on active
breaks, which involves performing light exercise, as a break behavior.
Fatigue is a common cause of poor learner performance. And it is important to consider
the degree to which fatigue is recovered by breaks. Most fatigue estimation methods utilize
facial expressions and eye blinking. These methods employ video cameras and may givepressure to be monitored to the learner. In contrast, leg movement measurements canestimate fatigue without giving pressure to be monitored to the learner because leg
movements are measured using a pyroelectric infrared sensor (Hamada, Terui, & Egi, 2022).
When using wearable devices, the discomfort caused by the device may affect concentrationand fatigue. However, leg movement measurement devices can be used to estimate fatigue
without contact; thus, the influence of the device is negligible.
In this study, we analyzed the effects of active breaks on task performance and fatigue
recovery. The target tasks included e-Learning and a mental arithmetic task, because suchtasks incur sufficient cognitive loads.

563

2. Methods
In this study, active breaks were performed by walking on a treadmill. Note that the load
incurred by active breaks is considered to vary for individuals. Thus, the Karvonen method
was used to set the exercise intensity, and active breaks were performed at a treadmill speed
that was appropriate for each learner. The Karvonen method can calculate the heart rate toachieve the target exercise intensity based on the learner's resting heart rate and age(Karvonen, 1957).
Here, the learner's fatigue was estimated by measuring leg movements, and thisinformation can be used to estimate the learner's mental condition without burdening on thelearner because it can realize contactless acquisition of biometric information. It has beenshown that there is a correlation between the learner’s leg movements and mental fatigue(Aikawa, Asai, & Egi, 2019).

3. Experiment and Results
Experiment I involved the e-Learning task, and Experiment II involved the mental arithmetictask.
To analyze the effect of active breaks on the learning effectiveness of e-Learning, we
focused on the rate of increase in scores on the pretest and post-test. Here, the rate ofincrease in scores was calculated by dividing the percentage of correct answers in the posttest by the percentage of correct answers in the pretest. The rate of increase in scores was
defined as the rate of correct responses on the post-test divided by the rate of correct
responses on the pretest. Results-AB (score) represents the results of the test in the range
studied after taking active breaks, and Results-PB (score) represents the results of the test
after taking passive breaks.
A paired-sample t-test between the active and passive breaks showed no significant
difference in the percentage increase in scores (p = 0.45, t = −0.77, degrees of freedom = 15).
However, the mean score of the test corresponding to learning after passive-breaks was
greater than that for the test corresponding to learning after taking active breaks. Thus, it ispossible that the passive-breaks were more effective in terms of improving the learning effect.

Figure 1. Experiment flow
In this study, the amount of leg movement was considered to analyze the effect of active
breaks on recovery from fatigue on learners during e-Learning. However, leg movements
involve many individual differences; thus, our analysis used the value obtained by dividing theamount of leg movements by the average of the individuals. A Wilcoxon signed-rank sum test
revealed that the amount of leg movements after active breaks was significantly lower (p <0.05). An example of low leg motion after active breaks is shown in Figure 1(upper right, leg
movements after active-breaks are in red box). Note that this example was taken from asubject in the PB-AB group.

564

The results of question indicate that many respondents were positive about the fatiguerelieving effects of the active breaks. There were several comments that could be read as"active breaks helped me feel better."
The score trends for each of the 100 MATH questions for the 10 subjects did not change
significantly for any of the subjects. By comparing the results of the mental arithmetic task
after breaks and active breaks, we found that all but three subjects showed a slight increasein scores after taking active breaks. These three subjects did not exhibit a significant decreaseof greater than 10 points of scores.
It is thought that as the experiment progresses, fatigue accumulated, concentration
declined, and the scores decreased. However, most subjects were able to maintain theirscores, which suggests that the active breaks had a positive effect on the MATH task.
Similar to the results of experiment I, here, we found that the amount of leg movement
after active breaks was low. A Wilcoxon signed-rank sum test revealed that the amount of leg
movements after taking active breaks was significantly lower than the amount of leg
movements after taking passive breaks (p < 0.05). The results of question indicate that many
respondents were positive about the fatigue-relieving effects of the active breaks. However,some subjects were negative about taking active breaks and one subject said “I lost myconcentration.”

4. Conclusion
In this paper, we analyzed the effects of taking active breaks while performing two types oftasks, i.e., e-Learning and mental arithmetic tasks. The experimental results demonstrated
that the active breaks helped recovery from fatigue in both tasks, but active breaks helped
improvement task performance in the only mental arithmetic task. Therefore, the difference inthe degree of cognitive load between the e-Learning and mental arithmetic tasks suggeststhat active breaks have a greater improve task performance effect on tasks with greater
cognitive load. However, there were several subjects who did not feel the effects of active
breaks; thus, it is necessary to determine which type of breaks is most appropriate for eachindividual learner.
In the future, we would like to develop a system that suggests appropriate break content
and timing based on the given task content and level of fatigue.

Acknowledgments
This work has been partly supported by the Grants-in-Aid for Scientific Research (NOs.
21K18496 and 21K02839) by Ministry of Education, Culture, Sports, Science and Technology(MEXT) in Japan.

References
Aikawa, D., Asai, Y., & Egi, H. (2019). Proposing an Estimation Method of Mental Fatigue by Measuring
Learner’s Leg Movement. HCII 2019: Learning and Collaboration Technologies. Designing
Learning Experiences, 227–236.
Hamada, T., Terui, Y., & Egi., H. (2022). Estimating Activity Conditions of Students in Class by
Measuring Leg Movement. 30th International Conference on Computers in Education, 656-658.
Karvonen, M. J. (1957). The effects of training on heart rate; a longitudinal study. Ann med exp biol
fenn, 35, 307-315.
Ramachandran, A., Huang, C., & Scassellati, B. (2017). Give Me a Break! Personalized Timing
Strategies to Promote Learning in Robot-Child Tutoring. 2017 ACM/IEEE International Conferenceon Human-Robot Interaction, 156-155.

565



" !   !"  " "   #" ! #" !"$  #" ! #"         
# #  &#   #%  $""  !     "!" 
    "!" 
3>@.34,D@ ?>?6D:?:@,.5;

  .*;7.;<6*B1*?.*>72:>.,1;878=B9.8/5.*;72701*+2=<=1*==1.B1*?.
=1.9;./.;;.-=26.8/-*B=8@8;4?.7=18>015.*;7.;*,=2?2=B.A=;*,=.-/;86=;*,.-*=*
,*79;8?2-.><./>5*7-27<201=/>527/8;6*=287*+8>==1.2;5.*;72701*+2=<=1.;.2<*5*,4
8/=;*,2701*+2=<27-*25B5.*;7270*=*<,18855.?.5/;865.*;7270580<%1.;./8;.@.
9;898<.=8>7-.;<=*7-<=>-.7=<E,1;878=B9.<8/5.*;72701*+2=<*==1. 5.?.5(.
27?.<=20*=.=1.9*==.;7<87.@..4*1.*-8/;.0>5*;=.<=<8?.;=1.B.*;><2705.*;7270
*7*5B=2,<=.,172:>.<8/,5><=.;270*7*5B<2<;86  -*25B580<8/=1.727=10;*-.;<27 @..45B 6*=1 5.*;7270 @. /27- ,5><=.;< 8/ 5.*;7270 9*==.;7< =1*= <>00.<= -2//.;.7=
,1;878=B9.<8/5.*;72701*+2=<%1./27-270<.7*+5.,87=.A=*@*;.;.,866.7-*=287</8;*68;.*>=1.7=2,5.*;7270.A9.;2.7,.@2=1*-*9=2?2=B*7-9.;<87*52C*=287@12,12<
98=.7=2*5/8;.71*7,2709.-*0802,*59;*,=2,.<2768+25.,87=.A=>*52C.-*7->+2:>2=8><
5.*;7270.7?2;876.7=<27/>=>;.;.<.*;,1  .*;7270 *+2=< &+2:>2=8>< .*;7270 %;*,. *=* .*;7270 7*5B=2,<-*9=2?..*;7270

 

+3094?.:80>?:?30;=010==0/?480:1/,D?:B:=670,=90=>8,D3,A0,@94<@0.3=:9:?D;0>@.3 ,> 8:=949290>>0A0949290>> ;=010=09.0 )30= ,?,7,  ,K0A4L *30
.3=:9:?D;0:170,=94923,-4?>.,9-00,>47D,>>0>>0/-D<@0>?4:99,4=0>B4?3>;00/D,9>B0=>
:B0A0= ?30 =0>@7?> ?09/ ?: -0 ,=-4?=,=D ,9/ >@-50.?4A07D /0;09/09? :9 49/4A4/@,7
,>>@8;?4:9>$,>709946:A,(:?0774$:9=0,70>,9,;;=:,.3?:/0,7B4?3?30,-:A0;=:-70870,=90=,.?4A4?D0C?=,.?0/1=:8?=,.0/,?, .,9 ;=:A4/0 @>01@7 ,9/ 49>423?1@7 491:=8,?4:9 ,-:@? ?30 ,8:@9? ,9/ ?D;0 :1 ,.?4A4?D
:..@==492,>B077,>?30?480,9//@=,?4:9:1?3,?,.?4A4?D(4.60=":E4,=>64+,7?0=>

:B0A0=?30>?@/40>:970,=94923,-4?>,=0:1?0949,>071;,.0/70,=9492.:9?0C?7460,$&&
$,>709946:A,0?,7(4.60=0?,7*30=04>,7,.6:1@9/0=>?,9/492:1?=,.492
3,-4?> 49 /,47D 70,=9492 ,? , >.3::7 70A07 1=:8 70,=9492 7:2> *30=01:=0 0C?=,.?492 ?30
.3=:9:?D;0>:170,=94923,-4?>4>48;:=?,9??:@9/0=>?,9/>?@/09?>I70,=9492-03,A4:=>,??30
" 70A07
9?34>>?@/DB049A0>?42,?0?30;,??0=9>:90B006,30,/:1=02@7,=?0>?>:A0=?30D0,=
@>49270,=9492,9,7D?4.>?0.394<@0>:1.7@>?0=492,9,7D>4>*30:-50.?4A04>?:0C?=,.?70,=90=>I
.3=:9:?D;0>:170,=94923,-4?>1=:8?304=/,47D7:2>+0,9>B0=?30=0>0,=.3<@0>?4:9+3,?
.7@>?0=>:170,=9492;,??0=9>.,9-00C?=,.?0/1=:8?30/,47D7:2>:18,?370,=9492
   9 ?34> >?@/D B0 ?,=20? ?30 70,=9492 .:9?0C? :1 ;=,.?4.492 ,9/ ?0>?492 8,?3 0C0=.4>0> 49 ,
!,;,90>05@94:=3423>.3::742@=0 >3:B>?30B:=617:B:1?3070,=9492,.?4A4?40>49?30>?@/D
.:9?0C? &9 , ?,-70? .:8;@?0= 70,=90=> ,..0>> ,9 0-::6 =0,/0= ::6(:77 &2,?, 0? ,7,9/;=,.?4.00C0=.4>0>:1,8,?3.:9.0;?1:=,B006,9/?309?,60,?0>?:1?30>,80
.:9.0;?:9$:9/,D49?301:77:B492B0061?0=?30?0>?70,=90=>.30.6?30,9>B0=>,9/>.:=0

566

?30?0>?B4?3?304=;00=>-D0C.3,924920,.3:?30=I>?,-70?>49,77D70,=90=>=0.:=/?30>.:=0?30D20?,9/?301@77>.:=0:1?30?0>?49,:,7&=409?0/.?4A0#0,=9492>D>?08&##4
$,5@8/,=309&2,?,
*34>B:=617:BB,>48;70809?0/=0;0,?0/7D:9,B0067D
-,>4>1=:8;=47
?:0-=@,=D +:=617:B:1+0067D$,?3#0,=949249)?@/D:9?0C?:1!@94:=423).3::7
=:8?30>?@/D.:9?0C?70,=94927:2>.:770.?0/-D::6(:77,9/&#,..@8@7,?049
?30 #0,=9492 (0.:=/ )?:=0 #() +0 =02,=/
 949?3 2=,/0=> ,> ?30 ;,=?4.4;,9?> :1 ?34>
>?@/D  70,=94927:2>,=00C?=,.?0/1:=?30,9,7D>0> 9?0=8>:1?307:2>B0.,7.@7,?0
?3049/4.,?:=:1*480);09?1:=?30,9,7D>0>?:,9>B0=?30=0>0,=.3<@0>?4:949?34>>?@/D*30
49/4.,?:=49/4.,?0>?30849@?0>,70,=90=>;09/>:9,'1470B4?349,93:@=*:.,7.@7,?04?
B0,22=02,?0?30:;0=,?4:9>1=:8?30>,80'1470,9/>@8@;?30/411?48049849@?0>:9,9
3:@=7D-,>4>*30=01:=04?>A,7@0=,920>1=:8 ?:    *:,9>B0=?30=0>0,=.3<@0>?4:9B0149/.7@>?0=>1=:8B0067D;,??0=9>:1;=,.?4.492-01:=0,
?0>?*30B0067D;,??0=94>49/4.,?0/-D?30?480>;09?B4?349?30B006-01:=0,?0>?/,D4=>?
B00C?=,.??30*480);09?B4?3,>@82=0,?0=?3,9E0=:>49.0B0/:9:?.:9>4/0=?30B00649
B34.3,70,=90=/:0>9:?>;09/?480;=,.?4.492,90110.?4A0;,??0=9-01:=0?30?0>?)0.:9/
B0/4A4/0?303:@=>B4?349,/,D49?:1:@=?480>7:?>8:=9492    ,1?0=9::9    0A09492      ,9/ :A0=9423?      +0 >@8 @; ?30 *480 );09? 49
0,.3?480>7:?*34=/B08,60, 8,?=4CB4?3?30>?,9/,=/4E0/A,7@0>:1?30?480>;09?
 -D?3070,=90=49?301:@=?480>7:?>1=:8$:9/,D?:)@9/,D:1?30B006-01:=0?30?0>?
/,D4>?30E>.:=0:1?30*480);09?49/4.,?0>?30?480>7:?B347049/4.,?0>?30/,D:1
?30B00649/4.,?0>?308,?=4C:1?306B00649,77DB0.:9A0=??308,?=4C49?:,A0.?:=
B4?3?B09?D0423?070809?>
(02,=/492 ?30 .7@>?0=492 ,9,7D>4> B0 49;@? ?30   A0.?:=> ?3,? 48;7D ?30 B0067D
;,??0=9>+0,/:;??30A0=,20)473:@0??0$0?3:/?:149/?30:;?48,79@8-0=:1.7@>?0=>,9/
4/09?41D?09B4?3?302=0,?0>?>473:@0??0>.:=042@=0 >3:B>?30?09.7@>?0=>080=24921=:8
?30"80,9>7@>?0=9,7D>4>*30=0/7490A4>@,74E0>?30.09?0=B0067D;,??0=949/4.,?0/-D
?30A0.?:=:1?30>?,9/,=/4E0/A,7@0>:1?30*480);09?49?301:@=?480>7:?>,.=:>>?30B006
9?34>>?@/D?30.3=:9:?D;04>49/4.,?0/-D?308,C48@8A,7@049?30B0067D;,??0=9
,>0/:9?307,-07>:1?30?09.7@>?0=>,9/?304=/0>.=4;?4:9B02=:@;?30.7@>?0=>:1;,??0=9>
49?:1:@=.3=:9:?D;0>
$:=9492 :9 +006/,D> 9  2=0,?0>? *480 );09? 49 ?30 8:=9492 1=:8 $:9/,D ?:
*3@=>/,DA09492 :9 +006/,D> 9  2=0,?0>? *480 );09? 49 ?30 0A09492 1=:8 $:9/,D ?:
*3@=>/,D
1?0=9::9:9+00609/>9 2=0,?0>?*480);09?:9)@9/,D,1?0=9::9$4C0/9 *480);09?:9*3@=>/,D,1?0=9::9,78:>?0<@,7>?3,?:9)@9/,D0A09492



567

 7@>?0=>1=:8+0067D',??0=9>:1'=,.?4.492-01:=0*0>?>  

,>0/ :9 ?30 149/492> B0 .,9 24A0 ;=,.?4.,7 >@220>?4:9> >@.3 ,> ?30 ?48492 ?: >09/ ?309:?414.,?4:980>>,20>.:9>4/0=492?30.3=:9:?D;0>:1?3070,=94923,-4?>*3069:B70/20:1
?30?08;:=,7.:9?0C?:1?3070,=94923,-4?>.,9;=:A4/0A,7@,-7049>423?49?:?30G.:9?0C?@,7
;=:147492H80.3,94>8>49.04?B477.,;4?,74E0:9?3070,=90=I>>.30/@701:=092,249249/4110=09?
70,=9492=07,?0/,.?4A4?40>

:770.?4A07DB0149/?34>>?@/D8,60>.:9?=4-@?4:9>-D0C?=,.?492?30.3=:9:?D;0>:1
?30 70,=9492 3,-4?> A4, ?30 .7@>?0=492 80?3:/ ,9/ /0,7492 B4?3 499:A,?4A0 4/0,> ,-:@?
.:9?0C?@,7;=:147492*301:=80=4>9:A07494/09?41D492?30.3=:9:?D;0>1=:8,;0=>;0.?4A0:1
0/@.,?4:9,7/,?,849492B3470?307,??0=09,-70>.:9?0C?,B,=0=0.:8809/,?4:9>1:=,8:=0
,@?309?4. 70,=9492 0C;0=409.0 B4?3 ,/,;?4A4?D ,9/ ;0=>:9,74E,?4:9 *30=01:=0 ?34> >?@/D 4>
;:?09?4,7 1:= 093,9.492 ;0/,2:24.,7 ;=,.?4.0> 49 8:-470 .:9?0C?@,74E0/ ,9/ @-4<@4?:@>
70,=949209A4=:9809?>491@?@=0=0>0,=.3

*34>B:=64>;,=?4,77D1@9/0/-D!)')""% %&!'%' !'%'



2 *3>6-*;#1.7 #!0*=*  8*58;2.7=.-*,=2?.5.*;7270!<B<=.6
=8 9;868=. ;.*-270 .70*0.6.7= <.5/-2;.,=.- 5.*;7270 +.1*?28; *7- 68=2?*=287 27 .A=.7<2?.
;.*-270!"&%#$&%!   
*<5.77248?*#8=.552 87;.*5.  >5B'2<>*57*5B=2,</8;$.<<287+*<.-%26.
(27-8@<-.7=2/2,*=28727'2;=>*5.*;72707?2;876.7=<7  % %# %! ! #  !#%! $&$%! 99
 
!0*=*)27!2 !4>+8$126*-*8326*)*6*-*   *7>*;B884+*<.- 5.*;7270 *7*5B=2,< 27 >72?.;<2=B .->,*=287 7  %# %!  ! #  !  !"&%#  &%!  99   
#2,4.;8C2*;<42 (*5=.;<  $=>-.7=52,4<=;.*6*=*8.<%26.8/*B *==.;
!&# !  #  $#   
$1.;'*=*5* *G.?2H  (1.78.*;7.;<$=>-B77*5B<2<8/=1.%26.8/*B
*7- (..4-*B(..4.7- &<*0. "*==.;7< 8/ .*;7270 *7*0.6.7= $B<=.6< /;86 8+25. *7-869>=.;<275.7-.-.*;7270!&# !#   '%$    

568

EGG

569

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Relationship Between Students’ Minecraft
Re-engagement Metrics and STEM Interest
Maria Mercedes RODRIGO*, Jonathan DL. CASANO & Mikael FUENTESa
Ateneo de Manila University, Philippines*mrodrigo@ateneo.edu
Abstract: We determine the relationship between student STEM interest and four reengagement metrics within a set of Minecraft worlds: number of concurrent players
playing outside of class hours, number of observations made, number of worlds visited,and number of science tools used. With the exception of number of observations made,there were no significant relationships found between STEM interest and these reengagement metrics. Therefore, the data so far does not provide much evidence thatthese behaviors are indicative of interest in STEM. Ways forward include later
administration of the STEM interest questionnaire, hypothesizing other in-gamebehaviors, and identifying a more sensitive STEM interest instrument.
Keywords: WHIMC, Minecraft, re-engagement, Philippines

1. Introduction
The What-If Hypothetical Implementations in Minecraft (WHIMC;
https://whimcproject.web.illinois.edu/) are a set of Minecraft-based simulations that immerse
students in alternate versions of Earth. They pose questions such as “What if Earth wastilted on its axis?” or “What if Earth had two moons?”. The students then explore these
Earths in order to observe how climate, geography, and life forms would differ because of
these varying circumstances. A team from the Ateneo de Manila University (ADMU)
partnered with WHIMC’s originators at the University of Illinois Urbana Champaign (UIUC) to
deploy WHIMC in Philippine classrooms. The goal of the ADMU team was to use WHIMC tocultivate student interest in STEM.
In order to measure the impact of WHIMC on student interest, the work of Casano,
Fuentes, and Rodrigo (2023) investigated the concept of re-engagement. A learner is said tobe re-engaged if he/she interacts with a learning intervention without a teacher’s prompting.
Student engagement takes place when the students use the intervention because he/she isrequired to do so, during a time that the teacher designates. Casano et al. note that priorstudies of student use of Minecraft tends to focus on engagement and sustained
engagement, which usually span from the time Minecraft is introduced in a class to the time
the students are asked to answer post-intervention questionnaires. While there is somework on student re-engagement with Minecraft, it is not as extensively studied.
Casano et al. (2023) operationalized re-engagement in WHIMC as social play, free
exploration, observations made, and science tools used. In this paper, we determine therelationship between these metrics and STEM interest. A student’s social play score was
equal to the number of overlapping play sessions, i.e. the number of concurrent companions
playing WHIMC with the student outside of teacher-mandated hours. Free exploration was
operationalized as the number of worlds the student visited, both during and outside ofteacher-mandated hours. Number of observations referred to the count of the observations
logged by the student. Finally, the use of science tools referred to the number of times
students measured different characteristics of the worlds they visited. The goal of this paper
is to determine how these re-engagement metrics relate with student STEM interest.

570

2. Methodology
The data analyzed in this paper was from eighth-grade students from eight different sectionsof a private school in the southern part of the Philippines. Their teachers received training
on the use of WHIMC, after which they crafted their own modules. These modules
underwent a process of review in which curricular alignment and quality were checked. A full
discussion of the preparatory process is available in (Manahan & Rodrigo, 2022). After
revision, the teachers conducted these lessons with their students.
There were five types of data collected from the students. We collected demographic
data about the students’ age, gender, school, and school section. WHIMC automaticallylogged student interactions with the system. Data from these logs included the student’s
position in the world, what observations the student made, and what tools the students used.
The students also completed three types of questionnaires. A STEM interest questionnaire(SIQ) was given to students before and after the use of WHIMC. This questionnaire asks
students questions about their perceived ability in STEM, their perceived future use of
STEM, and their interest and value for STEM. A Game Experience Questionnaire (GEQ;
IJsselsteijn & De Kort, 2013) asked students how engaging the game experience was.
Finally, Knowledge questionnaires were created by the teachers as part of their lesson
planning. Both the GEQ and the Knowledge questionnaires are excluded from the scope of
this paper.
There were 175 students who participated in the WHIMC lessons. The dataset forthis analysis was composed of the following fields:z Demographic data: Gender, Agez Questionnaire data: SIQ Pre-test score, SIQ Post-test score, Difference between SIQ
Post-test and Pre-test
z WHIMC log file data: Used WHIMC outside of class (Yes/No), Number of concurrent
users during use outside of class, Number of observations made, Number of science
tools used, Number of worlds visited outside of class
We removed students with incomplete pre- and post-test data, leaving a total of 131
students for this analysis. Of these 131 students, 58 were male and 73 were female. Theirages ranged from 13 to 15 years old.

3. Analysis and Results
To understand how STEM interest changed and how it related with re-engagement, we first
performed a paired t-test to determine whether the SIQ scores of the students changed
significantly. While the post-test scores were slightly higher on average (3.6) than the pretest (3.5), the difference was not significant (t(254.68)=-1.02, p=0.3). This is consistent with
Tablatin, Casano, and Rodrigo’s (2023) analysis of a subset of this data, which also showedno significant difference. This implies that using WHIMC did not influence students’ interest
in STEM.
There were no statistically-detectable gender differences. Male and female students
did not differ significantly in terms of their pre-test scores (t(123.31)=0.63, p=0.5), post-testscores (t(127.14)=-0.56, p=0.58), or their post- and pre-test difference (t(115.38)=1.43,
p=0.16).
For the next analyses, we treated the population as a whole and only used the SIQ
post-test scores as the outcome variable.
We first attempted to correlate the SIQ post-test scores with the number of concurrent
companions, the number of observations, number of science tools used, and the number ofworlds visited. All of these correlations were negligible.
We then tried to divide the population into different terciles based on each of themetrics. That is, we first sorted the number of concurrent companions and then divided the
population into the top, middle, and bottom third. There should have been 43 to 44 students
per tercile. However, if a cluster of students at the border of two terciles had the same
scores, we assigned the cluster to either the higher or lower tier, depending on what would
result into as balanced a split as possible. We then performed an ANOVA and a Tukey

571

Honestly Sigificant Difference post-hoc analysis to determine if there were differences in SIQ
post-test scores among the different terciles.
There were no significant differences among terciles of for the number of concurrent
companions (f(2)=1.83, p=0.17), number of science tools used (f(1)=0.73,p=0.4), or numberof worlds visited (f(2)=0.36,p=0.72).
For the total observations, there were 75 students with no observations at all and 57
students with at least one observation. Instead of splitting this group into terciles, we split
the group into two—those with at least one observation and those with no observations. We
found that the students who made no observations had lower SIQ post-test scores thanthose who made at least one observation (t(128.42)=2.08, p=0.04).

4. Conclusion
The results of the analysis are disappointing. With the exception of number of observations
made, none of the re-engagement metrics had a statistically detectable relationship with SIQ
scores. The data did not provide evidence in favor of these measures of re-engagement.
One possible explanation was that the SIQ post-test was given before the students reengaged with WHIMC. It is also possible that the exposure to WHIMC was too brief toencourage or inspire greater STEM interest. In future research, we could ask students tocomplete the SIQ again, several weeks after they complete the WHIMC lesson. We could
also search for other in-game behaviors that might be indicative of STEM interest. Finally,
we could search for a more sensitive measure of STEM interest, other than the SIQ.

Acknowledgements
We thank H Chad Lane and Jeff Ginger for their enthusiastic collaboration. We thank ourpartner teachers and schools. We thank Ms. Rosario Madjos for her administrative support.
We thank our colleagues at the Ateneo Laboratory for the Learning Sciences. Finally, we
thank the Ateneo de Manila University for making this work possible through the grant
entitled, “The Impact of Technology Use on Learning Outcomes and the Learning
Experience”.

References
Casano, J. D., Fuentes, M., & Rodrigo, M. M. T. (2023, June). Quantifying Re-engagement in
Minecraft. In International Conference on Artificial Intelligence in Education (pp. 248-253). Cham:
Springer Nature Switzerland.Manahan, D. M. A. B., & Rodrigo, M. M. T. Support Structures andActivities for Teachers Preparing for Game-Based Learning. The 10 Workshop on TechnologyEnhanced STEM Education (TeSTEM). In Iyer, S. et al. (Eds.) (2022). Proceedings of the 30th
International Conference on Computers in Education. Asia-Pacific Society for Computers in
Education.
IJsselsteijn, W. A., De Kort, Y. A., & Poels, K. (2013). The game experience questionnaire.
Roller, S. A., & Lampley, S. A., & Dillihunt, M. L., & Benfield, M. P., & Turner, M. W. (2018,
June), Student Attitudes Toward STEM: A Revised Instrument of Social Cognitive Career Theory
Constructs (Fundamental) Paper presented at 2018 ASEE Annual Conference & Exposition ,Salt Lake City, Utah. 10.18260/1-2--31001
Tablatin, C. L. S., Casano, J. D., & Rodrigo, M. M. T. (2023, March). Using Minecraft to Cultivate
Student Interest in STEM. In Frontiers in Education, 8, 143. Frontiers.
th572

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Enhancing Learning Experience in
University Engineering Classes with
Kahoot! Quiz Games
Zilu LIANGa*
Ubiquitous and Personal Computing Lab, Faculty of Engineering, Kyoto University of
Advanced Science (KUAS), Japan*liang.zilu@kuas.ac.jpaAbstract: In this short paper we present a study examining students’ experience usinga gamification quiz platform, Kahoot!, in two engineering courses in a private Japanese
university. We ran Kahoot! quiz game regularly at the end of each lecture session
throughout a semester. Students were encouraged but not forced to engage indiscussions with peers during gameplay. We evaluated our approach using an originalquestionnaire designed to probe students’ opinions of the usability and usefulness ofthe quiz games as well as their preferred play mode. The results showed that overall
students had positive experience playing the Kahoot! quiz games in class, though manyof them had no prior experience using Kahoot! or any kind of online quiz platforms
before taking the courses we offered. Surprisingly, their interest in the quiz games did
not wear off, and they enjoyed the fun and competitive atmosphere throughout thesemester. Consistent with findings in prior studies, our analysis showed that many
students considered the quiz games useful in helping them to concentrate in class andto develop a deeper understanding into the technical content. To a less degree,
students found the quiz games motivating them to learn more about the subject.
Students’ preference of the play mode split, which suggests the importance ofdiversifying the play modes to meet the needs of different types of learners. We
reflected over the nature of our quiz questions using the revised version of the Bloom’s
taxonomy. Our current design was centered on the ‘Remember’, ‘Understand’, and‘Apply’ levels of factual and conceptual knowledge. Future studies are needed to
expand the quiz game design to support the development of higher-order cognitive
skills at the ‘Analyze’, ‘Evaluate’ and ‘Create’ levels, as well as adding additional
objective measures for the assessment.
Keywords: Kahoot!, gamification, quiz, active learning, engineering education,
computing education

1. Introduction
Quiz is widely used as a tool for pre- and post-class assessment (Cavadas et al., 2017;
Cook & Babon, 2017; Liang, 2019). In particular, online quiz was found to be effective inincentivizing students’ completion of preparatory work in flipped classroom (Egan et al.,2017), and in engaging and motivating students in remote learning (Parte & Mellado, 2022).
Quiz improves knowledge retention and is considered an effective strategy of active learning(Cook & Babon, 2017). However, traditionally quiz is often used as an asynchronous activity
where students answer the questions on their own and receive feedback with a delay.
Game-based quiz games such as Kahoot! were developed to address the limitations of
traditional quizzes and to enhance students learning experience. Kahoot! is a web
application that allows educators to create interactive quiz games. It does not require an
installation and can be accessed through an Internet browser on any digital device. It has
been widely used in classes from K-12 to higher education around the globe (Wang & Tahir,2020). Kahoot! provides user-friendly graphic interface for instructors to create their quiz
activities, and a separate interface for players to answer the quiz items. Kahoot! allows

573

instructors to add a variety of questions, including single- and multi-choice questions, true or
false questions, and puzzles. With a premium subscription, instructors will also be able toembed slides, videos, audios in a quiz game as well as collecting opinions with polls, word
cloud and open-ended questions. Allowing for real-time feedback is a main difference
between a Kahoot! quiz game and a traditional quiz (either paper-based or digital). For eachquestion, once all students have submitted their responses, the system automatically grades
all students and projects the correct answer as well as the distribution of their responses tothe whole class. It allows instructors to provide immediate feedback, to comment oncommon mistakes, and to ask follow-up questions as needed. Such instantaneousinteraction is often more educationally effective than when delivered after a delay (Thelwall,2000). Instructors can customize the time allocated to each question so that the timepressure can be adapted to students’ levels. In addition, by projecting an updated ranking ofstudents after each question, it creates a competitive atmosphere which helps boost
students’ motivation and engagement (Wang & Tahir, 2020; Licorish, et al. 2017). At the endof a game, players are ranked based on their final scores and response time. After playing aquiz game, the platform will generate a report summarizing the overall performance of thewhole class, and players’ overall scores and scores to each question.
While there is a large body of evidence that the Kahoot! quiz games enhanceclassroom dynamics and motivate students to engage in the class flow (Cameron & Bizo,
2019; Licorish, et al. 2017; Wang & Tahir, 2020), little is known in terms of whether the quiz
games motivate them to learn more about the subject and whether they appreciate thecollaborative opportunities enabled by the quiz games. In addition, Asian student populations
were under-represented in previous studies. In this paper, we share our experience of using
quizzes to enhance learning experience in an English-medium engineering faculty in a
private Japanese university. Leveraging the Kahoot! quiz platform, we combined
gamification and summative quizzes to create a fun and competitive atmosphere in class,with the main purpose of promoting learner-content interactions and little emphasis on
assessment. We evaluated our approach by collecting students’ feedback using aquestionnaire. By analyzing their responses, we aimed to generate preliminary answers to
the following questions:z
How did students like the Kahoot! quiz games? Did they enjoy competing with peers?
z
How often did they engage in discussions with others during gameplay?
z
Did students prefer playing individually or in groups?
z
Did the quiz games help students concentrate in class?
z
Did the quiz games deepen their understanding of the learning content?
z
Did the quiz games motivate them to explore more on the subject?

2. Method
We used the Kahoot! quiz platform in two courses. One is a mandatory Android app
development course (denoted as C1) designed for first year university students, and theother is an elective digital signal processing course (C2) for third year students. Bothcourses offered 15 weekly teaching sessions. A teaching session last 3 hours and consist of
a lecture or tutorial and some hands-on exercises. Details of the courses can be found in(Liang, 2022; Liang et al., 2021). In the 2022 academic year, 56 and 21 students wereenrolled in each course. We played Kahoot! live at the end of each lecture session andalternated between the two play modes: classic mode and group mode. In classic mode
students answered the questions with or without peer-interaction and they were ranked
individually. In group mode students formed teams and had a short period of group
discussion before answering each question. In this play mode students were ranked byteams. In Figure 1 the left pane shows the user interface on students’ device, and the rightpane shows the question and options that are projected to the whole class. To avoid
overtaxing students’ cognitive resources, our quiz games usually consist of 5 questions thatcan be completed in less than 10 minutes.

574

Figure 1. User interface of the Kahoot! quiz game platform.
We developed an original questionnaire to collect students’ opinions of the Kahoot!
quiz games. The questionnaire consists of 17 items as shown in Table 1 under the themesof usability, entertainment, competition, reinforcement, and motivation. Students rated each
item on a Likert scale ranging from “strongly agree” (=5) to “strongly disagree” (=1). The
question items were designed to probe students’ opinions of the usability and usefulness of
Kahoot! as well as their preference of collaboration with classmates during gameplay. The
questionnaire was implemented using the Microsoft Form.

3. Findings
Table 1. The median and mode of students’ responses for each item.
Item
ID
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Q11
Q12
Q13
Q14

Questionnaire Item

Android App
Development(C1)

It was easy to use the Kahoot! app on my device.
It was fun to play the Kahoot! quiz games.
It was fun to compete against other students.
I wished to score top 3 and always tried my best tocompete.
I was frustrated when I didn't score well in a Kahoot!
game.
I discussed with other students to come up with myanswers.
I prefer playing the group battle mode.
I was initially excited about Kahoot! in the firstclasses but gradually lost interest.
Playing Kahoot! helped me stay concentrated.
Playing Kahoot! deepened my understanding of thelearning contents.
Playing Kahoot! motivated me to learn more about
the subject.
I wished Kahoot! was used in other lectures.
Overall, the Kahoot! quizzes were helpful.
Overall, I like playing Kahoot! quiz.

575

5 (5)
5 (5)
4 (5)
4 (5)

DigitalSignal
Processing
(C2)
5 (5)
5 (5)
4 (4)
4 (4)

4 (4)

4 (4)

3 (3)

3 (3)

3 (3)
2 (2)

3 (3)
2 (1)

4 (3)
4 (4)

4 (4)
4 (4)

3 (3)

4 (3)

4 (5)
4 (4)
4 (5)

4 (4)
4 (4)
4 (4)

We received responses from 47 and 21 students, which translates to a response rate of
83.9% and 100%, in C1 and C2, respectively. As for prior experience with Kahoot!, morethan half of the students (N=25) in C1 had no prior experience using Kahoot! before taking
the course. Among those 25 students, some had used other quiz platforms such as Quizlet(N=2) and Everyone’s Quick Quiz/ Min'na de hayaoshikuizu (N=6). Similarly, the majority ofthe students in C2 (N=19) had no prior experience using Kahoot! but a few had used Quizlet(N=2) and Everyone’s Quick Quiz/ Min'na de hayaoshikuizu (N=1). This implies thatgamification is not yet widely used in educational institutions prior to university.

Figure 2. Distribution of responses to “Q6: I discussed with other students to come up
with my answers” in C1 (Left) and C2 (right).

Figure 3. Distribution of responses to “Q7: I prefer playing the group battle mode” in C1(Left) and C2 (right).
Given the ordinal nature of the data, we calculated the median and mode (instead of
average and standard deviation) of students’ responses for each item, as shown in Table 1.
Overall, students’ experience with Kahoot! was positive in both courses. Students
considered the quiz games easy to use (Q1), fun to play (Q2, Q14), and helpful (Q13), andmany of them wished that Kahoot! was used in other courses (Q12). While we hypothesizedthat students’ interest in the quiz games would wear out gradually, the result suggested
something opposite (Q8). Students in both courses perceived the competitive atmosphereas fun (Q3) and motivating (Q4), though they may feel frustrated if they did not score well(Q5). The opinions split when it comes to collaborating with classmates (Q6, Q7). As shownin Figure 2, 21% and 10% of the students never or seldom discussed with others during
gameplay despite of being encouraged to. At the other end of the spectrum, 41% and 47%
often or always engaged in discussions with others to come up with answers. As for the576

preference of play mode, both courses showed similar tendency. As shown in Figure 3,
33%-41% students preferred group mode over classic mode, 24%-29% students had the
opposite opinion, and the rest had no preference. This implies that instructors should
alternate between the two play modes to meet the needs of students with different
preferences. Initially we feared that the Kahoot! quiz game may be perceived primarily as anentertainment activity but with limited effect on learning because a quiz game was only abrief activity in each class. Nonetheless, in both courses many students agreed that the quiz
games deepened their understanding (Q10), and to a less degree, helped them stayconcentrated (Q9) and motivated them to explore more about the subject (Q11).

4. Discussions
We have presented our experience of using Kahoot! quiz games in class to engage andmotivate students. Our findings echo previous studies that Kahoot! quiz brings the benefit ofenjoyment and engagement, which is likely due to the competitive atmosphere created by
the point system and the background music (Wang, 2016; Licorish, et al. 2017; Cameron &Bizo, 2019). While previous studies have demonstrated the effect of Kahoot! in motivating
students to engage with current learning, our results demonstrate that Kahoot! may also
motivate students to explore more of the subject in the near and far future.

Figure 4. The design of our Kahoot! quiz games was centered on the ‘Remember’,‘Understand’, ‘Apply’ levels of factual and conceptual knowledge.
In what follows we discuss the role of the Kahoot! quiz games through the lens of therevised version of the Bloom’s taxonomy (Krathwohl, 2002). As shown in Figure 4, therevised taxonomy consists of four types of knowledge and six levels of cognitive processorganized in hierarchy. The design of our Kahoot! quiz games was centered on supporting
students’ learning on the ‘Remember’, ‘Understand’, and ‘Apply’ levels of factual andconceptual knowledge. In particular, the single-choice and true-or-false questions prompted
students to recollect the factual and conceptual knowledge covered in the lecture sessions toboost knowledge retention. As indicated by students’ responses to Q10, the quiz games
were perceived as effective in helping deepen their understanding of the learning contents,
hence supporting the ‘Understand’ level of cognitive process. Given the time pressure forstudents to answer the questions synchronously, it is challenging to design questions that
support the development of higher-order thinking skills at the ‘Analyze’, ‘Evaluate’, and‘Create’ levels, particularly those related to the procedural and metacognitive knowledge
dimensions. Acknowledging the limitations of our quiz game design, we incorporated otheractivities as a complement. For example, in both courses students were required to work on
a self-directed final project towards the end of the semester. The planning, execution,577

implementation, and monitoring of the project provided great opportunities for students toacquire procedural and metacognitive knowledge while developing higher order thinking
skills. There are several limitations of this work that should be addressed in future studies.
First, this study was conducted in a private university and thus may not generalize well toother types of educational institutions. Second, the findings relied on self-reported data fromthe questionnaire, which may subject to recall bias or social desirability. Objective measuresor additional assessment could be used in the future to strengthen the study. Third, we did
not delve deeper into the design opportunities for supporting higher-order cognitive skills andcollaborative learning using the Kahoot! quiz game platform.

5. Conclusion
We have shared our learning design with the Kahoot! quiz game platform in two engineeringcourses targeting first- and third-year university students. The quiz games created a fun andcompetitive atmosphere to engage and motivate students. The questionnaire results
demonstrated that students had positive experience with the quiz games. Students found thequiz games useful in helping them concentrate in class and deepening their understandingof the technical contents. Students’ preferences split when it came to whether to engage indiscussions with peers and the play mode, which implies the importance of diversifying theplay settings to meet the needs of different types of learners. According to the revised
version of the Bloom’s taxonomy, our current quiz game design was centered primarily on
supporting the development of lower-order cognitive processes including ‘Remember’,‘Understand’ and ‘Apply’ factual and conceptual knowledge. In our next step we will evaluate
our learning design with a large cohort using both subjective and objective measures. We
will also expand our quiz design to support the development of higher-order cognitive
processes at the ‘Analyze’, ‘Evaluate’ and ‘Create’ levels and collaborative learning.

References
Cameron, K. E., & Bizo, L. (2019). Use of the game-based learning platform KAHOOT! to facilitate
learner engagement in Animal Science students. Research in Learning Technology, 27, 2225.
Cavadas, C., Godinho, W., Machado, C., et al. (2017). Quizzes as an active learning strategy: a study
with students of pharmaceutical sciences, Proceedings of the 12th CISTI. IEEE.
Cook, B. R., & Babon, A. (2017). Active learning through online quizzes: better learning and less(busy) work. Journal of Geography in Higher Education, 41(1), 24-38.
Egan, R. W., Fjermestad, J., & Scharf, D. (2017). The use of mastery quizzes to enhance student
preparation. Journal of Higher Education Theory and Practice, 17(8), 36-42.
Krathwohl, D. (2002). A revision of bloom’s taxonomy: an overview. Theory Into Prac, 41(4), 212-218.
Liang, Z. (2019). Enhancing active learning in web development classes using pairwise pre-and-post
lecture quizzes. In Proceedings of IEEE TALE 2019, Yogyakarta, Indonesia. IEEE.
Liang, Z. (2022). Contextualizing introductory app development course for first-year engineering
students. In Proceedings of the TALE 2022, Hong Kong. IEEE.
Liang, Z., Nishi, M., & Kishida, I. (2021). Teaching android app development to first year
undergraduates: Textual programming or visual programming? In Proceedings of TALE 2021.
Licorish, S. A., George, J. L, Owen, H. E., Daniel, B. (2017). “Go Kahoot!” enriching classroom
engagement, motivation and learning experience with games. In Proceedings of the 25th
International Conference on Computers in Education.
Parte, L., & Mellado, L. (2022). Academic performance in distance education: quizzes as a moderator
variable and students’ perception and expectation through linguistic analysis. Online Learning,
26(2), 124-147.
Thelwall, M. (2000). Computer-based assessment: a versatile educational tool. Computers &Education, 34(1), 37-49.
Wang, A. I., & Lieberoth, A. (2016). The effect of points and audio on concentration, engagement,
enjoyment, learning, motivation, and classroom dynamics using Kahoot! In Proceedings of the10th European Conference on Game Based Learning (ECGBL 2016).
Wang, A. I., & Tahir, R. (2020). The effect of using Kahoot! for learning – A literature review.
Computers & Education, 149(103818).

578" !   !" " "   #" !
#" !"$  #" ! #"            
        &    !    %& #!
$() )*)%' ') +$%"% ($ +'( )-%%*)'$ " %'$  ,'$""% $$ +'( )-% " %'$  '!"- 
?H2?8:4E FD4 65F  @B74717/:7<B3::753<137A031=;7<5C07?C7B=CA7<=C@2/7:G:7D3A<;=@3/<2 ;=@3 473:2A  AGAB3;A /@3 B@/<A4=@;7<5 6=E 9<=E:3253 7A 1=<AB@C1B32
27A1=D3@73A/@3@3/:7H32/<26=EA=:CB7=<A/@323D3:=>32/<2B3AB32(63A316/<53A
6/D3 >@=4=C<2 7;>:71/B7=<A 4=@ B63 4CBC@3 E=@94=@13 /<2 17B7H3<@G -3B :3/@<7<5 
@3;/7<A / <7163 AC0831B :/@53:G @3A3@D32 4=@ /2D/<132 >=ABA31=<2/@G 32C1/B7=</:
1=<B3FBA+67:3B63@37A5@=E7<5/BB3<B7=<B=0@=/23<7<532C1/B7=</:=>>=@BC<7B73A
/<23A>317/::GB=>@=D727<5:3/@<7<53F>3@73<13A4=@G=C<53@ABC23<BA@3:/B7D3:G:7BB:3
7A 1C@@3<B:G 9<=E< /0=CB 6=E B= ;=AB 34431B7D3:G >@=D723  32C1/B7=< B=  
97<23@5/@B3<B6@=C56 B65@/23ABC23<BA<B67A>/>3@E327A1CAAB6323A75</<2>@3A3<B47<27<5A4@=;/<7;>:3;3<B/B7=<ABC2G=4/<32C1/B7=</:5/;34=@6756A16==:
32C1/B7=<1/::32&# @/E7<5=</<7<B35@/B32/</:GA7A=45/;3>:/G:=52/B/
>@3 >=AB9<=E:3253/<227A>=A7B7=<AC@D3GA4=@<3/@:G 
6756A16==:ABC23<BAE3
>@3A3<B47<27<5A=<B6334471/1G=4B6332C1/B7=</:5/;3/<27BA1=<AB7BC3<B/1B7D7B73A7</2D/<17<5  :3/@<7<5 5=/:A +3 3F>:=@3 >=AA70:3 7<B3@/1B7=<A 03BE33< :3/@<7<5=CB1=;3A7<1=;7<5;/B69<=E:3253>@7=@5/;7<53F>3@73<13/<2=B63@4/1B=@AB6/B
1/<7<4=@;4CBC@3:3/@<7<523A75</<2A632:756B=<E6/B1/<>=A7B7=<G=CB64=@AC113AA
7<5/;30/A32:3/@<7<53F>3@73<13A

  

32C1/B7=<G=CB632C1/B7=<32C1/B7=</:5/;3 $"!%$! CE:7:4:2="?E6==:86?46":D27@F?52E:@?2=E649?@=@8JA6C>62E:?86G6CJ2DA64E@7@FC52:=J
=:G6D *2A:525G2?46D:?E9656D:8?2?5:>A=6>6?E2E:@?@7"DJDE6>D92G6=65E@E966G6C
6IA2?5:?8C@=67@C":?D@4:6EJ "E:D2=D@AC@7@F?5=JEC2?D7@C>:?8@FCH@C<7@C462C@F?5E96
8=@36 /9:=6D@>6@7E@52JQDJ@FE9H:==364@>6E967FEFC6"H@C<7@C462?52>2;@C:EJ@7
E96>H:==;@:?2H@C<7@C46E92EFE:=:K6D"2==H:==364@>66?5 FD6CDDF492D4@?DF>6CD@7" 2C5?6C &4F?66E2=   "E:D4C:E:42=E96C67@C6E@AC6A2C67FEFC686?6C2E:@?DH:E9
32D:4<?@H=6586@7"?@E;FDEE9C@F899:896C65F42E:@?3FE368:??:?8H:E949:=59@@5
=62C?:?8 
/9:=6"QD:>A24E@?D@4:6EJ:D566A6?:?82?56IA2?5:?8:?>JC:25H2JD2?5:??@G2E:G6
65F42E:@?2=@AA@CEF?:E:6D2C636:?8C2A:5=J56G6=@A65E96C692D366?=:EE=6C6D62C49:?E@
9@HDEF56?ED6DA64:2==JAC6 4@==686DEF56?ED4@?DECF4E2?F?56CDE2?5:?8@72?582:?
AC24E:46H:E94@C6:562D:?E967:6=5 D2C6DF=EE96C6:DJ6E=:EE=6A@DD:3:=:EJ@78C@F?5:?8E96
56D:8?@7=62C?:?86IA6C:6?46D:?6G:56?46 32D65244@F?ED@79@HJ@FE9=62C?"4@?46AED
9@HF?56CDE2?5:?8AC@8C6DD6D24C@DD4@?46AED@CH92E4@?46AED2C6>@DE2AAC@AC:2E67@C
H92E286 =6G6=D ":D3F:=E@?27@F?52E:@?@7A9:=@D@A9JADJ49@=@8J2?5>2E96>2E:4D2?5:E46?E6CD2C@F?5FD:?82=8@C:E9>DE@D@=G6C62= H@C=5AC@3=6>D*FDD6==2?5'@CG:8  
,9:DAC@G:56D2E96@C6E:42=7@F?52E:@?E@4@??64E"=62C?:?8E@6I:DE:?8+4:6?46
,649?@=@8J?8:?66C:?82?5&2E96>2E:4D+,&DF3;64ED:?$ 4=2DDC@@>D  :G6?E96
A24<65D4965F=6@76I:DE:?84@FCD6D@7$ DEF56?ED:E364@>6D2>@C6C62=:DE:42AAC@249

579

E@6>365"65F42E:@?:?$ 4=2DDC@@>D +F492?2AAC@249E@":?DECF4E:@?@776CD2C:49
4@?E6IEE@=62C?D4:6?E:7:42?5>2E96>2E:42=4@?46AED2=C625JE2F89E:?$ /2?82?5
#@9?D@? 2?5E@2AA=JE96>E@AC@3=6>D@=G:?8 (?6E649?@=@8J 32D652AAC@249E@3C:?8"E@E96$ 4=2DDC@@>E92E92DD9@H?AC@>:D6:?
@E96C+,&5:D4:A=:?6D:D5:8:E2=82>6 32D65=62C?:?8 64256D@7C6D62C496G:56?46A@:?E
E@E96677:424J@782>6 32D65=62C?:?8:?AC@>@E:?8DEF56?E=62C?:?8)=2DD6E2=    !@H6G6CE96C6:D=:EE=6C6D62C49:?E@FD:?882>6 32D65=62C?:?87@C"65F42E:@?7@CJ@FE9
%666E2=   8:G6?E92EE96C6D62C497:6=5@7$ "65F42E:@?:DDE:==:?:ED:?72?4J "?
E9:DA2A6CH6H:==5:D4FDDE9656D:8?2?5:?:E:2=:>A=6>6?E2E:@?DEF5J@72?65F42E:@?2=
82>642==65*"'  7@CE6249:?89:89 D49@@=DEF56?ED23@FE" /64@?5F4E652?
6G2=F2E:@?DEF5J2E9:89D49@@=D:?E96-?:E65+E2E6D *6DF=ED:?5:42E6E96A@E6?E:2=@7*"' E@3F:=5"<?@H=65866DA64:2==J7@CDEF56?EDH9@92G6324<8C@F?5<?@H=6586H:E9E96
C6=6G2?E>2E96>2E:42=4@?46AEDEJA:42==JE2F89E2EE969:89D49@@==6G6= $!"
"65F42E:@?92D=@?8366?23D6?E7C@>$ 4=2DDC@@>D *646?E677@CED2C6368:??:?8E@
:?G6DE:82E6E96:?E68C2E:@?@7":?E@$ D49@@=D:?4=F5:?8567:?:?8"=:E6C24J%@?82?5
&286C<@  2?556G6=@A:?84FCC:4F=22?58F:56=:?6D 2C5?6C &4F?66E2=  &",
"5F42E:@?"?:E:2E:G6   *6D62C496CD:?J@FE9"65F42E:@?92G6366?6IA6C:>6?E:?8
H:E9E6249:?8":?4=F5:?8>249:?6=62C?:?8*@5CL8F6K 2C4L26E2=   19@F6E2=   2?56E9:4D@CDJE96E2=   H:E9:?E964@?E6IE@74@>AFE2E:@?2=E9:?<:?8*:EE6C6E
2=  E9C@F894@?G6CD2E:@?2=286?ED%:?6E2=   52?46)2J?66E2=   2?5
82>6 32D65=62C?:?8%666E2=    :D4FDD:@?D@?J@FE9"65F42E:@?2C6962E:?8FA:?
FC@A6$2?5=9@76C6E2=  "  9:?2)6E6CD@?6E2=   "DC26=+92>:C
2?5%6G:?  2?52C@F?5E96H@C=50@F;F?6E2=  0F<F?2?5,2?8  @C
6I2>A=6C6D62C496CD:?,92:=2?592G656D:8?652?28C:4F=EFC2= 32D65"492==6?86E@7@DE6C
>:55=6 D49@@=DEF56?EDQ=62C?:?8@7E96>249:?6=62C?:?8AC@46DD:?E967@C>@7282>6+2<F=<F62<F=DF<6E2=  H96C6DEF56?ED3F:=5>249:?6=62C?:?8>@56=DE@4=2DD:7J
C:A6@CF?C:A6>2?8@6D "?FDEC2=:2C6D62C496CD92G656D:8?652?5:>A=6>6?E65
4=2DDC@@>24E:G:E:6D7@CE6249:?87F?52>6?E2=4@?46AED@7"E@062CDEF56?EDE@56>JDE:7J
"E9C@F8924E:G:E:6DDF492D2?F?A=F886524E:G:EJ@?724:2=C64@8?:E:@?2?52D:>A=6C@3@E:4
6I6C4:D6E92E:?EC@5F46DE964@?46AE@7>249:?6=62C?:?8!@6E2=   ,96H@C<AC6D6?E6596C62:>DE@F?4@G6C9@HE@56D:8?2?65F42E:@?2=82>6E@>66EE96
492==6?86D@7E6249:?8"E@$ DEF56?ED ,9:DH@C<3F:=5DFA@?6IA=@C2E:@?D:?E@9@H$ 
DEF56?ED2AAC@249"4@?46AEDH92E@3DE24=6DE96J72462?59@HE@8F:56E96>E9C@F89
@3DE24=6D C66?H2=56E2=    ,9:DH@C<2=D@5C2HDFA@?AC6G:@FD:?G6DE:82E:@?D:?E@=:?<:?8"E@E96$ >2E94FCC:4F=F>E@:56?E:7J"4@?46AEDDF:E23=67@C9:89D49@@=
DEF56?ED/2?82?5#@9?D@? 2DH6==2DH@C<:?G6DE:82E:?8E96=62C?:?8@7
4@>AFE2E:@?2=E9:?<:?8%666E2= 2?5D6>:?2=C6D62C49:?E@4@>AC696?D:@?@7>2E96>2E:42=C6AC6D6?E2E:@?D6 8 FC4:@ C:6=6E2=     #"  &"!  $
*"'  :D2C@=6 A=2J:?882>656D:8?65E@E62499:89 D49@@=DEF56?ED"4@?46AED
AC@>AEE96>E@2AA=JE96:C>2E9<?@H=65862?556G6=@AE96:C"AC@3=6> D@=G:?8D<:==D "?E9682>6DEF56?EDA=2J2D2DA246 72C:?8D4:6?E:DEH9@92D4C2D9=2?565@?2?2=:6?A=2?6E
?2>65*"'  :8FC6  "?@C56CE@D276=JC6EFC?9@>6E96D4:6?E:DE368:?D6IA=@C:?8
E96A=2?6EE@82E96CC6D@FC46D?66565E@C6A2:CE963C@<6?D9:AH9:=6F?4@G6C:?8E96
>JDE6CJ@7E96A=2?6E ,9624E:G:E:6D7@CDFCG:G2=2?57@C6IA=@C2E:@?7@C>E9632D:D7@CE96
E2D<DE96DEF56?ED42CCJ@FE:?E9682>6 ,9682>64FCC6?E=J4@G6CDE9C664=2DD:42=D62C49
2=8@C:E9>D3C625E9 7:CDED62C49+56AE9 7:CDED62C49+2?58C665JD62C49 249E@A:44@?D:DED@7EH@>@5F=6D2EFE@C:2=>@5F=66 8 :8FC6 3@EE@>=67E2?52EC2?D76C
>@5F=66 8 :8FC6 3@EE@>C:89E >365565:?2==E96EFE@C:2=2?5EC2?D76C>@5F=6D2C6

580

BF:KK6DE92E96=ADEF56?EDA2FD62?5D6=7 2DD6DD:8FC6 E@AC:89E "? 82>6492==6?86D
DF492DD62C49:?87@C>:DD:?8DA246D9:AA2CED@C4C24<:?8A2DDH@C5DD6CG62D?2EFC2=
@AA@CEF?:E:6D7@CE96:?EC@5F4E:@?@7D62C492D2E@A:4 ,966DD6?E:2=4@?46AEDDF492DDA246
2?5E:>64@>A=6I:EJ2=D@=6?5@AA@CEF?:E:6DE@4@??64E>2E9<?@H=658672>:=:2CE@9:89
D49@@=DEF56?ED2?5E96D6"4@?46AEDE92E2C6FDF2==JE2F89E:?9:896C65F42E:@? ,96
:?E68C2E6565F42E:@?2=4@?E6?E:?*"'  =6G6C286DE9:D@AA@CEF?:EJ3JDFAA@CE:?8E96
DEF56?EDQ2AA=:42E:@?@7>2E9<?@H=6586E@E966G2=F2E:@?@762492=8@C:E9>2DE96JAC@8C6DD
E9C@F89E9682>6 "?255:E:@?2=E@E96=62C?:?8>@5F=6DDEF56?ED42?2=D@6IA=@C6E9682>6
6?G:C@?>6?E7@CO@77 E2D<P24E:G:E:6D+23@FC:?6E2= DF492D82E96C:?8>:?6C2=D
2C@F?5E96A=2?6E  *'+4C66?42AEFC67C@>*"'  ,@A =67E,96A=2J6C4C2D9=2?565@?27@C6:8?A=2?6E ,@A C:89E DEF56?E :D AC6D6?E65 H:E9 2 BF:K BF6DE:@? 23@FE 6DE:>2E:?8 E96 4@>A=6I:EJ @7
D62C492=8@C:E9>D @EE@>=67EDEF56?EE9:?< 2=@F5DE9C@F89E968C665JD62C492=8@C:E9> 
@EE@> C:89EE96DEF56?ED@=G6D2? AFKK=6FD:?8@?6@7E96D62C492=8@C:E9>DE@7:IE96:C
4@>A2?:@?C@3@EQD4:C4F:E3@2C5  ! % ! "  ,9656D:8?@7*"'  :D8F:5653J=6DD@?D=62C?657C@>A:=@EDEF5:6D@?DEF56?E"
AC@3=6> D@=G:?8 C66?H2=56E2=    /62=D@56G6=@A65255:E:@?2=56D:8?AC:?4:A=6D
32D65@?@FC@3D6CG2E:@?D@7E96492C24E6C:DE:4DF?:BF6E@"=62C?:?82?59@H=6DD@?D
=62C?657C@>C6=2E657:6=5DDF492D4@>AFE6CD4:6?4665F42E:@?%666E2= 4@F=536
FD65E@:?7@C>E9656D:8?96C6 ##"#!#"?24@8?:E:G6:?E6CG:6HDEF5JC6D62C496CD6IA=@C659@H$ 
DEF56?EDA2CE:4F=2C=JE9@D6:?9:89D49@@=2AAC@249"4@?46AEDH92E@3DE24=6DE96J7246
2?59@HE@8F:56E96>E9C@F89@3DE24=6D C66?H2=56E2=    "?E96DEF5JDEF56?ED
H6C6AC6D6?E652D6E@7"AC@3=6>D:?2H:56C2?86@7E@A:4DDF492DG2C:@FD>249:?6
=62C?:?82=8@C:E9>D +EF56?EDQE9:?< 2=@F5D2DE96J2EE6>AE65E@D@=G6E96"AC@3=6>DFD:?8A2A6C2?5A6?4:=D965=:89E@?E964C:E:42=DE6A>@DEDEF56?EDDECF88=65H:E9N
AC@3=6>7@C>F=2E:@?@CE96 ()') %$A92D6:?4@>AFE2E:@?2=E9:?<:?8: 6  ()') %$*)%#) %$2?5 $"-( (%666E2=  ,9:D:D2?:?:E:2=DE6AH96C6DEF56?ED7@C>F=2E62
AC@3=6>56D4C:365:??2EFC2==2?8F2866 8 7:?5E96D9@CE6DEA2E9:?E@@?6E92E42?36

581

D@=G653J24@>AFE6CDF492D4C62E:?8G2C:23=6D6 8 5:DE2?4656E6C>:?:?86?54@?5:E:@?D6 8 D62C496?5DH96?5:DE2?4642?QE36>:?:>:K65 +EC2E68:6D6>A=@J653JE966IA6CE
:?E6CG:6H6CAC@G:565232D:D7@CA6528@8:42=56D:8?:?*"'  (?6@7E9656D:8?564:D:@?D>256E@724:=:E2E63DEC24E:@?H2DE@5:DA=2JE96C62= H@C=5
AC@3=6>D2?5E9623DEC24EC6AC6D6?E2E:@?DD:56 3J D:562?5E@FA52E63@E9DJ?49C@?@FD=J6 8 E96=@H6C =67ED4C66?:?:8FC6  "?2?*"'  C@FE6 A=2??:?8AC@3=6>H96C6
DEF56?ED2C6E2D<65E@FD6D62C492=8@C:E9>DE@A=2?2C@FE6E@2H2E6C72==2>2A:DA=2465
@?@?6D:56@7E96D4C66?H9:=62D62C49EC66:D5:DA=2J65@G6C2?5?6IEE@E96>2A DDEF56?ED5:C64EE962=8@C:E9>3J6IA=@C:?8=@42E:@?D4@??64E65G:2C@25D@?E96>2AE96
D62C49EC66FA52E6D244@C5:?8=JDE6A 3J DE6A2?5:==FDEC2E6D9@H=@42E:@?D2C6C6AC6D6?E65
2D?@56D2?5C@25D2C6C6AC6D6?E652D6586D:?2EC6652E2DECF4EFC62?59@HC@FE6
A=2??:?8@?2A9JD:42=>2A:D4@>AFE2E:@?2==JD@=G652DE966IA2?D:@?@72D62C49EC66 /96?28@2=?@56:DC624965:?E96D62C49EC66:E:D9:89=:89E653@E9@?E96D62C49EC662D2
A2E924C@DD6586D7C@>C@@EE@E968@2=?@562?5@?E96>2A2D2C@FE6C6249:?8E96
H2E6C72==4@??64E65G:2C@25D !!"!"?65F42E:@?EC2?D76C@7=62C?:?8@44FCDH96?=62C?:?8:?@?64@?E6IE
6?92?46D@CF?56C>:?6D2C6=2E65A6C7@C>2?46:?2?@E96C4@?E6IE)6C<:?D6E2=   
FC:?8=62C?:?8EC2?D76CDEF56?ED2AA=J=62C?:?8:?@?65:D4:A=:?624C@DD>F=E:A=6D:EF2E:@?D ,C2?D76C@7=62C?:?8:DA2CE:4F=2C=J:>A@CE2?E7@C"65F42E:@?2D"42?364@?D:56C652D2
5:D4:A=:?6@7FD:?82=8@C:E9>DE@D@=G6C62= H@C=5AC@3=6>D /96?DEF56?ED=62C?9@H2?"
2=8@C:E9>42?36FD65E@D@=G6:==FDEC2E:G6AC@3=6>D:?@?65@>2:?:E:D4C:E:42=E@2=D@8F:56
E96>E9C@F89AC@3=6>D7C@>25:776C6?E4@?E6IEE@96=AE96>3F:=5E964@??64E:@?NE96
23DEC24EC6AC6D6?E2E:@?@7E962=8@C:E9>E92E42?362AA=:65E@7@C>F=2E6D@=FE:@?DE@
D66>:?8=J5:776C6?E2?5F?C6=2E65AC@3=6>D 

@C62492=8@C:E9>4@G6C65:?*"'  H656G6=@A652EFE@C:2=AC@3=6>2?52EC2?D76C
AC@3=6> ,96EH@D6ED@7AC@3=6>D2C65:776C6?E6?@F89E@2C8F23=J364@?D:56C652D72C
EC2?D76CD:?DE625@7?62CEC2?D76CD)6C<:?D6E2=   ,FE@C:2=AC@3=6>D2C649@D6?7C@>
5@>2:?D72>:=:2CE@E96DEF56?EDDF492D7:?5:?82C@FE67C@>A@:?EE@@?2>2A "?2
EJA:42=EFE@C:2=>@5F=6DEF56?ED2C6D4277@=565E9C@F89E9623DEC24E:@?2FE@>2E:@?2?5
2?2=JD:DAC@46DD6D%666E2= E9C@F89E96A=2J6C492C24E6CQDE9:?< 2=@F52?5E96:C
5:2=@8F6H:E9E964@>A2?:@?C@3@E "?E9623DEC24E:@?A92D6E96DEF56?ED2C68F:565E@
4C62E62?23DEC24EC6AC6D6?E2E:@?@7E96AC24E:42=AC@3=6> 7E6CDEF56?ED56>@?DEC2E6E96:C
F?56CDE2?5:?83J4@CC64E=J6IA2?5:?8E96EC667@CD6G6C2==6G6=DE96J2C6AC@G:565H:E9E96
@AE:@?E@2FE@>2E6E96AC@46DD "?E962FE@>2E:@?A92D6DEF56?ED42?H2E49E96D62C49
EC664@?E:?F6E@6IA2?52FE@>2E:42==J@?E96D2>6:?E6C7246NA9JD:42=>2A2?523DEC24E
D62C49EC66A=2465D:56 3J D:56 +EF56?ED42?2=D@A2FD62?5DE6AE9C@F89E96EC66
6IA2?D:@?@?6DE6A2E2E:>6E@6I2>:?6E96AC@46DD4=@D6=J ,962FE@>2E656IA2?D:@?
2?:>2E:@?96=AD:==FDEC2E6E96492C24E6C:DE:4D@7E96D62C492=8@C:E9>D6 8 6IA2?5:?8:?2
3C625E9 7:CDE@C56AE9 7:CDE72D9:@? "?E962?2=JD:DA92D6DEF56?ED2C68F:5653JE9682>6
?2CC2E:G6E@6I2>:?6E96D@=FE:@?6 8 E96C@FE67@F?52?5E@6G2=F2E6E96AC@46DDE9C@F89
H9:49E96D@=FE:@?H2D86?6C2E656 8 E:>62?5DA2464@>A=6I:EJ@7E96D62C492=8@C:E9> ,96DF3D6BF6?EEC2?D76CAC@3=6>>@5F=6AC6D6?E65DEF56?EDH:E925:776C6?EAC@3=6>DF49
2D4C24<:?82A2DDH@C5@CD@=G:?82? AFKK=6 +EF56?EDH6C68F:5653JD:>:=2C3FE>F49
233C6G:2E65D4277@=5DE9C@F89E96 ()') %$ *)%#) %$2?5 $"-( (AC@46DD6D:?E96
EC2?D76CA92D6  !#$% #":D9F>2?:562DC6AC6D6?E65>2E96>2E:42==J2?5C62=:K65
4@>AFE2E:@?2==J C@>=2DD:42=+62C49E@%@42=+62C497C@>)C@A@D:E:@?2=E@:CDE (C56C
%@8:47C@>64:D:@?,C66DE@ 6?6E:4=8@C:E9>D"2=8@C:E9>D3F:=5@?6249@E96C2?6H
2=8@C:E9>:D@7E6?4C62E653J>@5:7J:?82?6I:DE:?8@?6E@D@=G6AC@3=6>DE92EE966I:DE:?8
@?6H2D?@EDF:E65@C23=6E@D@=G6 ,9:D:?D:89E4C62E6D3@E9492==6?86D2?5@AA@CEF?:E:6D
7@C"65F42E:@? ,966G@=FE:@?2CJ492C24E6C:DE:4D@7"2=8@C:E9>DAC@G:56232D:D7@C
A6528@8JE92E=6G6C286DDEF56?EDQAC:@C<?@H=6586@72?2=8@C:E9>E96J2C62=C625J72>:=:2C

582

H:E9H9:=64@?DECF4E:?8E96?6H@?6D J5:C64E=J4@>A2C:?8E96?6H2?5@=52=8@C:E9>D7@C
6I2>A=6DEF56?ED?@E@?=J=62C?E96?6H3FE2=D@C6:?7@C46E96=62C?:?8@7E96@=5 +F49
4@>A2C:D@?D2C6?@E;FDE2=8@C:E9>:43FE2=D@E964@?E6IEF2=:?E6C>D@72AA=:42E:@? -?56CDE2?5:?8E96AC@D@7E96?6H2?54@?D@7E96@=5:?H92EAC@3=6>DE96J2C6@C2C6?@E
DF:E65E@255C6DD:D2<6JE@FD:?8"7@CAC@3=6>D@=G:?8 ,962AAC@249@7AC:@C<?@H=6586
24E:G2E:@?:D?@E?6H=G6C>2??6E2=  ?@C:DE96:DDF6@724E:G2E:?8:?244FC2E6AC:@C<?@H=6586G2?%@@?6E2=   ,966IA=:4:E2?55:C64E4@>A2C:D@?36EH66?E96?6H2?5@=5H96?5:D4FDD:?8E96?6H>2J@776C2?@AA@CEF?:EJ7@CDEF56?EDE@C66I2>:?6E96:C
>:D4@?46AE:@?D@7E96@=5 "?*"'  82>6>@5F=6D2C6@C82?:K653J=62C?:?8E@A:4DDF492D+2?5+ 7E6C
D4277@=5:?8DEF56?EDE9C@F89E967:CDE"2=8@C:E9>DF492D+6249?6H"2=8@C:E9>
6 8 +:D:?EC@5F465E9C@F892?6I2>A=6AC@3=6>E92EE96AC6G:@FD2=8@C:E9>D72:=E@
D@=G66 8 4@>AFE6CCF?D@FE@7>6>@CJH96?FD:?8+7@CC@FE6A=2??:?8 ,96DEF56?ED
2C6E96?8F:565E9C@F89E96?2=JD:DA92D6E@F?4@G6CH9JE96AC6G:@FD2=8@C:E9>72:=65
6 8 DE@C:?8E@@>2?J?@56D:?4@>AFE6C>6>@CJ2?59@HE@>@5:7J:EE@255C6DD:ED
H62<?6DD6 8 AC:@C:E:K:?86IA2?5:?849:=5?@56D:?DE625@7D:3=:?8?@56D:?E96D62C49
EC66 +F49>@5:7:42E:@?E9FDC6DF=ED:?E963:CE9@7E96?6H2=8@C:E9>6 8 + ,965:C64E
4@>A2C:D@?D2C6?@E@?=JC62=:K65:?E966IA=2?2E:@?DE9C@F89E9682>6?2CC2E:G63FE2=D@
:==FDEC2E65@?E96FD6C:?E6C7246D24C@DDE96=62C?:?8@75:776C6?E2=8@C:E9>D $!# !$##

,H6?EJ E9C66>2E9D4:6?462?54@>AFE6CD4:6?46E62496CD7C@>2D49@@=5:DEC:4E:?2>2;@C
>6EC@A@=:E2?2C62:?E96-?:E65+E2E6DA2CE:4:A2E65:?E96DEF5J  9:89D49@@= 2865J@FE9
7C@>4=2DD6DE2F89E3JA2CE:4:A2E:?8E62496CDH6C6C64CF:E657@CE96DEF5J  !$!
)2CE:4:A2E:?8E62496CDH6C6AC@G:5652?@G6CG:6H@7E9682>6=62C?:?88@2=D2?5DEF5J
AC@465FC6276H>@?E9D367@C6E96DEF5J3682? 76HH66<DAC:@CE@E96DEF5JDEF56?ED
H6C68:G6?2?@?=:?6A2C6?E2=4@?D6?E7@C>2?52J@FE92DD6?E7@C> (?=JDEF56?EDH9@
4@?D6?E65A2CE:4:A2E65:?E96DEF5J ,96DEF5JH2D42CC:65@FE@G6C4=2DDD6DD:@?D6249
=2DE:?8 >:?FE6D=@?8H:E92E=62DE4=2DDD6DD:@?D565:42E65E@:?5:G:5F2=82>6A=2J
7@CDEF56?ED FC:?8E967:CDED6DD:@?DEF56?EDH6C67:CDE2DD:8?65"DE@AC@E64EE96:C:56?E:EJ
E9C@F89@FEE96DEF5J2?5E96?4@>A=6E65E96AC6 DFCG6J@?=:?6 EE966?5@7E967:CDE
D6DD:@?DEF56?ED=@8865:?E@E96*"'  82>6@?=:?6G:22H633C@HD6C ?JE649?:42=
5:77:4F=E:6D6?4@F?E6C65H6C6255C6DD655FC:?8E967:CDED6DD:@?G:2DFAA@CE7C@>E96
C6D62C49E62> FC:?8E96D64@?52?5E9:C5D6DD:@?DDEF56?ED4@?E:?F65E@:?E6C24EH:E9
*"'  2EE96:C@H?A246  2>6AC@8C6DD:@?A=2JE:>62?52?DH6CDE@:? 82>6
BF6DE:@?DH6C6C64@C5657@C6249A2CE:4:A2?E FC:?8E967@FCE9D6DD:@?DEF56?ED4@>A=6E65
E96A@DE DFCG6J@?=:?6 /:E9C6DEC:4E652446DDE@D49@@=42>AFD6D5F6E@(." E96DEF5JH2D42CC:65@FE
6?E:C6=J3JE96A2CE:4:A2E:?8E62496CD ,96C6D62C49E62>5:5?@EA2CE:4:A2E6:?E9652E2
4@==64E:@? 55:E:@?2==J3642FD6DEF56?EDH6C6?@EC6BF:C65E@2?DH6C2==E96BF6DE:@?D@?
E96AC6 2?5A@DE DFCG6JDE96C62C6>:DD:?852E22EE96:E6>=6G6=7@CD@>6DEF56?ED  "$!",96AC6 DFCG6J4@?D:DE65@7:E6>D23@FEDEF56?EDQ56>@8C2A9:4324<8C@F?5"-D6,JA6
"?E6C6DE:?""$?@H=6586 BF6DE:@?D&2E9+6=7 677:424J%:F2?5$@:C2=2 2?5&2E9$?@H=6586 ==D42=6D6I46AEE96&2E9+6=7 677:424JH6C656G6=@A653JE96C6D62C49

583

E62> ,96"-D6,JA6:?4=F565:E6>DDF492DO/96?"E9:?<23@FE9@H"Q5=:<6E@:?E6C24E
H:E9":?E967FEFC6"6IA64EE92E"H:==FD6"DJDE6>D:?>J6G6CJ52J=:762D24@?DF>6C
2?5"6IA64EE@-+"DJDE6>D2D2A2CE@7>J;@3 P,96"?E6C6DE:?"D42=6:?4=F565
BF6DE:@?DDF492DO(FED:56@7D49@@="ECJE@=62C?2=@E23@FE" P,962DD6DD>6?E@7"<?@H=65862?5>2E9<?@H=6586DA64:7:42==J7@4FD65@?E964@?E6?E4@G6C65:?*"'  :?
E967@C>2E@7>F=E:A=6 49@:46BF6DE:@?D ,96"BF6DE:@?DH6C6D6E:?E964@?E6IE@7D@=G:?8
"AC@3=6>DD:>:=2CE@E9@D66?4@F?E6C65:?E9682>6 ,96BF6DE:@?D2DD6DD65DEF56?EDQF?56CDE2?5:?8@77@C6I2>A=6AC@D2?54@?D@7E96D62C492=8@C:E9>DD62C492=8@C:E9>D>@DE2AA=:423=6E@DA64:7:4EJA6D@7AC@3=6>D6E4 "?E96A@DE DFCG6JE96D2>6:E6>D@?
:?E6C6DE:?"2?5"<?@H=65867C@>E96AC6 DFCG6JH6C6:?4=F565 "?255:E:@?E@E96
DFCG6JD82>6=@8D7C@>*"'  H6C64@==64E65 ,96=@8D:?4=F565E96:? 82>64=:4<
DEC62>52E22?5C6DA@?D6DE@:? 82>6BF:KK6D #%$#
(7E96 A2CE:4:A2E:?8DEF56?ED4@>A=6E65E96A@DE DFCG6J ,96C6D62C49E62>H2D
23=6E@>2E49AC6 A@DE DFCG6JD2?582>6=@8D7@CDEF56?ED (E96CE92??@C>2=
2EEC:E:@?6 8 DEF56?ED23D6?E2E6:E96CAC6A@DE25>:?:DEC2E:@?@C82>6A=2J4=2DD
255:E:@?2=52E2=@DDH2DAC:>2C:=J5F6E@6CC@CD:?DEF56?E"6?EC:6D@?E96DFCG6JA=2E7@C>
H9:49C6DF=E65:?>:D>2E496D@7DEF56?E"D36EH66?3@E9DFCG6JD2?582>6=@8D /6
4@?5F4E65'(.2?2=JD6DE@6?DFC6E967:?2=D2>A=6@7DEF56?EDH2D?@ED:8?:7:42?E=J
5:776C6?E7C@>E967F==A2CE:4:A2?ED2>A=6:?E6C>D@7324<8C@F?5DF492D86?56CC246 6E9?:4:EJ2?5AC:@C>2E96>2E:42=<?@H=6586 ,96A2CE:4:A2?EDQ2G6C286286H2D H:E9  E98C256CD  E98C256CD E9
8C256CD2?5E98C256CD E@E2=@7@7E96DEF56?ED:56?E:7:652D>2=6
:56?E:7:652D76>2=62?5:56?E:7:652D@E96C42E68@C:6D@CAC676CC65?@EE@5:D4=@D6 
@7E96DEF56?EDDA62<?8=:D92E9@>6DA62<3@E9?8=:D92?52D64@?5=2?8F2862E
9@>62?5DA62<@?=J2=2?8F286@E96CE92??8=:D92E9@>6 +A2?:D9:DC6A@CE652D
E96?@? ?8=:D9=2?8F2867@CE9@D6DEF56?ED "?E6C6DE:?8=J6G6?E9@F89*"'  2?5E96
DFCG6JD2C6@776C65:?3@E9?8=:D92?5+A2?:D92?5E96E62496CDH6C63C:676523@FEE96
=2?8F28649@:46AC:@CE@E96DEF5J2==E96DEF56?ED49@D6E@FD6E96?8=:D9G6CD:@?@7E96
DFCG6JD2?5E9682>6  !/64@?5F4E652A2:C65D2>A=6E E6DE@?E96"<?@H=6586D42=67C@>AC6 2?5A@DE DFCG6JD
E@6I2>:?6:7A=2J:?8E9682>6C6DF=E65:?82:?D:?"<?@H=6586 ,23=6 DF>>2C:K6DE96
AC6 A@DE492?86D:?"$?@H=65862?5:?DF3 4@?DECF4ED5:C64E=JC6=6G2?EE@>@5F=6D:?E96
82>6255:E:@?2="<?@H=6586:E6>D4@G6C65:?3C@256CE@A:4DDF492DD62C49EC66
C6AC6D6?E2E:@?D *6DF=EDD9@HE92EDEF56?EDH9@A2CE:4:A2E65:?E96DEF5J56>@?DEC2E65
DE2E:DE:42==JD:8?:7:42?E82:?:?"<?@H=6586H:E92>62?5:776C6?46@7 @?2 A@:?E
D42=6& ,96"<?@H=6586D42=6:?4=F56DF3 D42=6D7@C6249@7E96D62C49
2=8@C:E9>D4@G6C65:?E9682>6++2?5 C665JD62C49 55:E:@?2=A2:C65 D2>A=6E
E6DEDC6G62=652DE2E:DE:42==JD:8?:7:42?E82:?7@C+=62C?:?8>62?5:776C6?46@7  @?2?
A@:?ED42=6&
2D>2==6C2?5?@EDE2E:DE:42==JD:8?:7:42?E492?867@C+=62C?:?8
& 2?52?62C=J7=2E@FE4@>67@CE96:E6>D7@4FD65@?E96 C665JD62C492=8@C:E9> ,23=6  '(#&"))()'(*")(%$ "'$ $ $(&'&%()'! $%,$-+'""
"" )#( $)("  $'-'"'$ $(*("(&6EC:4
$!  
!&

)C6 ,6DE
 
 
 
 

)@DE ,6DE &2I +4@C6 , DE2E:DE:4D A G2=F6 7764E+:K6
 

 


 

 


 

 

 
 





584





 #$# !$ !"?E96AC6 DFCG6JH682E96C6552E2@?DEF56?EDQ56>@8C2A9:4324<8C@F?5DF492D86?56C
8C256=6G6==2?8F286DA@<62E9@>62?5G:56@82>66IA6C:6?46D '(.E6DEDD9@HE92E
AC6 A@DE"=62C?:?882:?D5:5?@E5:776CD:8?:7:42?E=J36EH66?DEF56?ED@75:776C6?E86?56C
8C256 =6G6=2?5=2?8F286DA@<6?2E9@>6 %62C?:?882:?D5:776C659@H6G6C36EH66?
DEF56?EDH:E95:776C6?EAC:@C82>:?86IA6C:6?46:8FC6 ,96A2CE:4:A2?EDC6A@CE652H:56
C2?86@782>:?86IA6C:6?46D  :G6?E9656E2:=6542E68@C:K2E:@?@782>:?86IA6C:6?46DH6
8C@FA65E96DEF56?ED:?E@EH@8C@FADE9@D6H9@5@?QEA=2JG:56@82>6D@CA=2J 9@FCD
A6CH66< @7E96D2>A=62?5E9@D6H9@A=2J@C>@C69@FCDA6CH66<  
+EF56?EDH9@A=2JG:56@82>6D=6DDE92?9@FCDA6CH66<925D:8?:7:42?E=J=@H6C82:?@?
@G6C2=="<?@H=6586&  &  & :?4=F5:?8DF3 D42=6D@?+
=62C?:?8&  &  A  C665J=62C?:?8& S &  &3FE?@E+=62C?:?8&  &  & 4@>A2C65E@DEF56?EDH9@A=2J
G:56@82>6D@C>@C69@FCDA6CH66< ,96AC6 DFCG6J2=D@:?4=F56D:E6>DE92E>62DFC6&2E9+6=7 77:424JC6=6G2?E&2E9
$?@H=65862?5"?E6C6DE:?" /64@?5F4E652D6C:6D@7C68C6DD:@?2?2=JD6DE@:?G6DE:82E6
E96D6E9C66DEF56?E=6G6=724E@CDE92E>2J36AC65:4E:G6@7@3D6CG65=62C?:?882:?D /6
7@F?5E92EE96AC:@C&2E9$?@H=65862D56>@?DEC2E65@?E96AC6 DFCG6J:E6>D6EAC65:4E65
@3D6CG65"=62C?:?882:?D* &  ,9:DDF886DEDE92EC6=6G2?E>2E9<?@H=6586:D
H62<=J3FED:8?:7:42?E=JC6=2E65E@9:896C"<?@H=658682:?65E9C@F89*"'  

 *' %67E)6C46?E286@7DEF56?EDH:E95:776C6?EH66<=J82>:?86IA6C:6?46 *:89E
"$?@H=658682:?3C@<6?5@H?3JH66<=J82>:?86IA6C:6?46  !!"" !
,96A6528@8:42=56D:8?@7*"'  :D32D65@?E969JA@E96D:DE92E"2=8@C:E9>D3F:=5@?
6249@E96C =8@C:E9>DDF492D+:?EC@5F465=2E6C:?E9682>62C65:D4FDD65:?
4@>A2C:D@?E@AC6G:@FD=J:?EC@5F4652=8@C:E9>DDF492D+ /9:=6DEF56?ED42?;F>A
E9C@F895:776C6?E>@5F=6D3J8@:?8E9C@F89E96>6?FD6=64E:@?D4C66?:?E9682>6@G6C2==
DEF56?EDE@@<2C6=2E:G6=J=:?62CA2E9E9C@F89E9682>63J8@:?8E9C@F89++E96?
C665J82>6>@5F=6D ,9FD2DDEF56?EDAC@8C6DDE9C@F89E9682>6>2DE6C:?8AC6G:@FD=J
5:D4FDD652=8@C:E9>D9@F=596=ADEF56?EDQ=62C?:?8@7E96?6H@?6DH9:=6=62C?:?8E96?6H
2=8@C:E9>96=ADDEF56?EC6:?7@C46E96=62C?:?8@7E96@=56C@?6D /62?2=JK659@HC6249:?8
>:=6DE@?6D:?E9682>6DF492D4@>A=6E:?8E96+>@5F=63@E9EFE@C:2=2?5EC2?D76C
AC@3=6>>@5F=6D:>A24ED@G6C2=="=62C?:?82?5E96=62C?:?8@7:?5:G:5F2=2=8@C:E9>D "?56A6?56?ED2>A=6E E6DED9@HDE96C6:D?@D:8?:7:42?E5:776C6?46:?"<?@H=658682:?
36EH66?DEF56?EDH9@4@>A=6E652==>@5F=6D@7E9682>62?5DEF56?EDH9@5:5?@E'' ,4@>A=6E:?8?@E4@>A=6E:?8&  +EF56?EDH9@4@>A=6E65E96+
>@5F=65:5?@E82:?D:8?:7:42?E=J>@C6"<?@H=6586@G6C2== & @C+<?@H=6586 & E92?E9@D6H9@DE2CE653FE5:5?QE4@>A=6E6+>@5F=6D !@H6G6C4@>A=6E:?8+5:596=ADEF56?ED82:?>@C6<?@H=6586@?+ &585

2?58C665JD62C49  &  @>A=6E:?8E96+@C8C665J>@5F=6D5:5
?@E92G62D:8?:7:42?E:>A24E@?@G6C2==@C:?5:G:5F2="2=8@C:E9>=62C?:?8  :G6?E92E2==
DEF56?ED2C68:G6?E96D2>62>@F?E@7E:>6E@A=2JE9682>6:?E964=2DDC@@>H65:5?@E
2?2=JK69@HE:>6:?82>6:>A24E65"=62C?:?8 (G6C2==DEF56?EDDA6?E36EH66?D64@?5D
2?5>:?FE6D:?E9682>6H:E92>62? >65:2?82>6A=2JE:>6@7 >:?FE6D ,96
@FE=:6C@76IEC6>6=J=@?882>6 A=2JE:>6:D=:<6=J5F6E@DEF56?ED7@C86EE:?8E@=@8@FE@7E96
82>62EE966?5@7E964=2DD  #%##! ,9:DDEF5J56>@?DEC2E6DE92E2C6=2E:G6=J3C:67:? D49@@=6IA@DFC6E@"=62C?:?86IA6C:6?46D
G:22?65F42E:@?2=82>642?C6DF=E:?=62C?:?882:?D7@C"4@?E6?EH:E9AC6 4@==6862865
J@FE9 I2>:?2E:@?@7E96DF3D42=6D4@C6D7@CE96"$?@H=65862DD6DD>6?E:?5:42E65E92E
E96=62C?:?8E92EE@@<A=246H2D4@?46?EC2E65@?E96+2=8@C:E9>H:E9D>2==6C82:?D7@C
+2?5?@82:?D7@C C665J C2H:?8@?E96=62C?:?82?52DD6DD>6?E56D:8?4@?46AE@72
=62C?:?8AC@8C6DD:@?/:=D@? F?42?2?5!>6=@ +:=G6C @FC4@?;64EFC6:DE92E
E9656D:8?@782>6A=2J:?H9:49DEF56?ED7:CDE6?4@F?E6C65+E96?4@>A2C65:EE@6249@7
E96?6IEEH@2=8@C:E9>D2DE9682>6AC@8C6DD65+E96? C665J=65E@4@?D@=:52E:@?@7
F?56CDE2?5:?8C6=2E65E@+2?5E9FD2566A6C@AA@CEF?:EJE@=62C?E92E4@?E6?E !@H6G6C
H65:5?@E@3D6CG62DE2E:DE:42==JD:8?:7:42?E:>A24E@74@>A=6E:?8E96++@C C665J
>@5F=6D@?+=62C?:?882:?D @>A=6E:?8E96+>@5F=6D9@H6G6C5:54@?EC:3FE6E@
=62C?:?8+2?5 C665JD62C49 ,9:DDF886DEDE96677:424J@7E96AC@8C6DD:G6C@==@FE@7
4@?E6?E:?@FC56D:8?H96C66249?6H4@?E6?E2C62:D6IA=:4:E=JC6=2E65E@AC:@C4@?E6?E %2E6C56D:8?:E6C2E:@?DH:===@@<E@6IE6?5@AA@CEF?:E:6D7@CDEF56?EDE@4@??64E2?5
4@?D@=:52E6E96:C6>6C8:?8F?56CDE2?5:?8@74@?E6?E6?4@F?E6C65=2E6C:?E9682>66 8 
E9C@F89255:E:@?2=:?E68C2E:G624E:G:E:6D ,96D:8?:7:42?46@7AC:@C>2E96>2E:42=<?@H=65867@CAC65:4E:?8@3D6CG65"=62C?:?882:?D
DF886DED2?65F42E:@?2=82>6E92E:D@AE:>:K657@CJ@FE9H9@2=C625J6?E6CH:E92DEC@?8
>2E96>2E:42=7@F?52E:@? ,9:DH@F=5492==6?86677@CED2EFD:?8E964FCC6?E:E6C2E:@?@7E96
82>67@C23C@259:89 D49@@=A@AF=2E:@?H:E92H:56C2?86@7AC:@C>2E94@>A6E6?4:6D 
FEFC656D:8?:E6C2E:@?DH:===@@<E@DFAA@CEDEF56?EDH:E9G2CJ:?8=6G6=D@7AC:@C>2E96>2E:42=
<?@H=65866:E96CE9C@F897@4FD65EFE@C:2=D7@CC6=2E65>2E94@?E6?E2?5 @CE9C@F89
:>AC@G6582>656D:8?E92E36EE6CC6D@?2E6DH:E9DEF56?EDH9@92G6?@EJ6EE2<6?25G2?465>2E94@FCD6D /62=D@D66AC@>:D67@CE9:D82>6 32D65:?DECF4E:@?2=>@56=:?E96762D:3:=:EJ@7:ED
:>A=6>6?E2E:@? :CDEE96J@FE9H9@6?82865H:E9E9682>65:5D@=2C86=J:?56A6?56?E=J@7
2E62496C ,96>:?:>2=?6657@C@FED:566IA6CE:D6>62?DE92E:>A=6>6?E2E:@?:D=:<6=JE@36
=6DD56A6?56?E@?92G:?865F42E@CDH:E9"2?54@>AFE6CD4:6?466IA6CE:D62?:>A@CE2?E
4@?D:56C2E:@?8:G6?H:56=JC6A@CE65D9@CE286D@79:89 D49@@=E62496CDH:E9DF496IA6CE:D6 
=D@C6=2E65E@762D:3:=:EJ@7:>A=6>6?E2E:@?4@>A=6E:?8E9682>6E@@<C@F89=J4=2DD
A6C:@5D@?2G6C286H9:49>:?:>:K6DE96E:>6:E>2J5C2H2H2J7C@>6I:DE:?8D4@A62?5
D6BF6?46 ,9FDE9665F42E:@?2=82>6:DH6== A@D:E:@?65E@36:?E68C2E65:?E@2H:56C2?86@74@FCD6D2?5:?DECF4E:@?2=4@?E6IED (?E96@E96C92?5E96"<?@H=65868C@HE9H2D
C6=2E:G6=JD>2==67764ED:K6 :?5:42E:?8E92EDA6?5:?84=2DD6DA=2J:?82?" E96>65
G:56@82>6:DF?=:<6=JE@4@?EC:3FE6E@=62C?:?882:?DE92E>:89E366IA64E657C@>27F==6C
:?DECF4E:@?2=D6BF6?46@C565:42E654@FCD6 "?E6C6DE:?8=J5FC:?8A@DE :>A=6>6?E2E:@?
4@?G6CD2E:@?DE96A2CE:4:A2E:?8E62496CD6IAC6DD65DEC@?8:?E6C6DEDE@:?E68C2E64=2DDC@@>
5:D4FDD:@?DH:E982>6 32D65=62C?:?82?5DF886DE655:G:5:?8E964=2DDC@@>E:>6:?E@
:?56A6?56?E82>6A=2J2?5A@DE 82>6A=2JH9@=6 4=2DD5:D4FDD:@?H96C6E62496CD@C82?:K6
5:D4FDD:@?DE@96=ADEF56?EDC67=64EFA@?H92EQD=62C?65E9C@F89E9682>6 +F492?
:?E68C2E652AAC@24992DE96A@E6?E:2=E@7FCE96C6?92?46E96677:424J@7E9665F42E:@?2=
82>6 
586

,96DEF5JH2D56A6?56?E@?2C6D62C496C 56G6=@A65>62DFC6@7"<?@H=6586H:E9=:>:E65
6G:56?462G2:=23=6@7:EDG2=:5:EJH:E9E96A@AF=2E:@?D2>A=65 ,9:DDA62<DE@E964FCC6?E
562CE9@7"<?@H=6586>62DFC6D56G6=@A657@CAC64@==686 2865J@FE92492==6?86E92E@FC
C6D62C49E62>2?5@E96CD2C6H@C<:?8E@255C6DDE9C@F89@?8@:?8C6D62C492?5
>62DFC6>6?E56G6=@A>6?E "?E9:DDA64:7:442D6H6?@E6E92EE962DD6DD>6?EH2D=:<6=JE@@
5:77:4F=E7@CE96D2>A=65A@AF=2E:@?7@C6I2>A=6E96>62?D4@C6@?E96A@DE :?E6CG6?E:@?
25>:?:DEC2E:@?H2D  @FE@72A@DD:3=6A@:?ED=:>:E:?8:EDA@E6?E:2=7@C56>@?DEC2E:?8
E96=62C?:?8@79:89 D49@@= 2865J@FE9 55:E:@?2==JE96>62DFC6:?4=F5655:776C6?E?F>36CD
@7:E6>D7@C6249@7E96D6DF3D42=6DH:E976H6C:E6>D7@C+2?5 C665J4@>A2C65H:E9+ ,9:D4@?DEC:4E65E962G2:=23=6C2?867@C>@G6>6?E@?E9@D64@?DECF4EDA@E6?E:2==J
:>A24E:?8@3D6CG23:=:EJ@7492?86DE92E>2J92G6E2<6?A=246 (FCE62>:D4@?5F4E:?8
ADJ49@>6EC:4E6DED4=2DD:42=2?5"*,>6E9@5D2?5EC:2?8F=2E:?8E92EH:E9:? 82>6
@AA@CEF?:E:6DE@56>@?DEC2E6F?56CDE2?5:?8E@:?7@C>C6G:D:@?DE@@FC:?DECF>6?E7@CE9:D
A@AF=2E:@?  !' $#

,9:DC6D62C49H2DDFAA@CE653JE96-?:E65+E2E6D'2E:@?2=+4:6?46@F?52E:@?F?56C C2?E  ?J@A:?:@?D7:?5:?8D2?54@?4=FD:@?D6IAC6DD65:?E9:D>2E6C:2=2C6E9@D6@7
E962FE9@CD2?55@?@E?646DD2C:=JC67=64EE96G:6HD@7E96'2E:@?2=+4:6?46@F?52E:@? " #
3D3:=>7<5/</@B74717/:7<B3::753<131C@@71C:C;/2/>B32B=C@=>3/<6756A16==:A
6BB>A /7>:CAC213A 
:D3@;/<<';7B6!/<2&3/23<13 %@7=@9<=E:3253/1B7D/B7=</<2B63
1=;>@363<A7=<=41=;>/B70:3/<27<1=;>/B70:3B3FB($',0*(5($4&+7$46(4.9 P 
C@17=& =;>@363<A7=<=4;/B63;/B71/:@3:/B7=<A67>A3F>@3AA327<5@/>6A1740$.)14
(5($4&+,0$6+(/$6,&5'7&$6,10   P  
C<1/<&/<2;3:='7:D3@
!3/@<7<5>@=5@3AA7=<A:75<7<51C@@71C:C;7<AB@C1B7=</<2/AA3AA;3<B1740$.1)(5($4&+,0 &,(0&(!($&+,0*   P 
=@AGB6'/:B=<=AB3@+/:A6';7:/19/<2-36(;/57<3/;=@3
3B671/:)A7<5AB=@73AB=23D3:=>B33<AS/E/@3<3AA/<2C<23@AB/<27<5=4/<27BAA=173B/:
7;>/1BA10)(4(0&(41&((',0*51)(5($4&+,037,6$%.($0' 756$,0('$46,&,2$6,10,0
0*,0((4,0*1/276,0*$0'!(&+01.1*9 P 
@73:'#C@17=&/<2@756B+
"/97<5A3<A3=45@/>6A@7B71/:4/1B=@A
7<4:C3<17<51=;>@363<A7=</<27<AB@C1B7=</:7;>:71/B7=<A1740$.)14(5($4&+,0$6+(/$6,&5
'7&$6,10   P /@2<3@"1C<3(=C@3BH9G"/@B7</<2'336=@<
4=@ P "/97<5@==;4=@7< P '1C@@71C:/!(&+0,&$. 9/215,7/101/276(4 &,(0&('7&$6,10 P@33<E/:2!37B<3@"/<2+/<5#!3/@<7<5/@B74717/:7<B3::753<13<A756BA7<B=6=E
G=CB63<1=C<B3@/<20C7:2C<23@AB/<27<5=41=<13>BA41&((',0*51)6+(10)(4(0&(1046,),&,$.06(..,*(0&(   P  
=+'1/227<5" =<5'<2=<37AE/A=>>3/<2AC(:/AA@==;
/1B7D7B73A4=@B3/167<5/@B74717/:7<B3::753<13B=>@7;/@GA16==:ABC23<BA10)(4(0&(41&((',0*5
101/276$6,10$.!+,0-,0*'7&$6,10 P /<2:6=43@"'B37<0/C3@!/M<75%/C;/<<+%:=;3@'/::/57/<2:4=:27
</0:7<5B631@3/B7=<=47<B3::753<BB67<5A@7<57<5/@B74717/:7<B3::753<13/<2@=0=B71AB=
A16==:A4106,(45,0'7&$6,1010)(4(0&( P
!33"/@B7<3<<3@=C:B3@::/<+@719A=<"/:G<';7B6/<2+3@<3@!

=;>CB/B7=</:B67<97<54=@G=CB67<>@/1B713041$'5  P !33'"=BB$BB3<0@37B!34BE716'1@70<3@(/G:=@'%/@9 &=E3:/H3EA97 
;3:='7:D3@/<2!3AB3@
7<4CA321=::/0=@/B7D37<?C7@G7<C>>3@3:3;3<B/@G
A16==:5/;30/A32:3/@<7<5/>>@=/16 9/215,7/10'7&$6,10'8$0&(5,046,),&,$.
06(..,*(0&(   P 

587

!7<%*/<@C;;3:3<!C97<+7::7/;A&/<2@3/H3/:
.6=@/73A75<7<5/
1=<D3@A/B7=</:/53<B4=@167:2@3<B=3F>:=@3;/167<3:3/@<7<51=<13>BA41&((',0*51)6+(
10)(4(0&(1046,),&,$.06(..,*(0&(  P
!7C,/<2 =7@/:/
(6334431B=4;/B63;/B71AA3:434471/1G=<;/B63;/B71A/1673D3;3<B=4
6756A16==:ABC23<BA41&((',0*51)146+($56(40'7&$6,10$.(5($4&+551&,$6,10
10)(4(0&(!=<5/<2"/53@9=
+6/B7A:7B3@/1G=;>3B3<173A/<223A75<1=<A723@/B7=<A41&((',0*51)6+(&10)(4(0&(10+7/$0)$&6145,0&1/276,0*5956(/5 P "(2C1/B7=<<7B7/B7D3
+6/BSAB63!G1C@@71C:C;6BB>A @/7A3;7B32C 2/7:G 7<23F6B;:
%/G<3+3@5<3@-+3AB"6/@>'6/>7@=&'H/47@(/G:=@*/<2
3A%=@B3A 
2/<13$#C:BC@/::G@3A>=<A7D31@3/B7D31=;>CB7<541&((',0*51)6+(10)(4(0&(107/$0$&6145,01/276,0* 956(/5 P %3@97<A#'/:=;=<3B/:  (@/<A43@=4:3/@<7<506(40$6,10$.0&9&.12(',$1)
'7&$6,10  P
%3B3@A=<==23 /<236:6/CA
32C1/B7=<7<67<//<2B63)<7B32'B/B3A
1=;>/@/B7D3/AA3AA;3<B(316<71/:@3>=@B3<B3@4=@'31C@7BG/<2;3@57<5(316<=:=5G
%:/AA!"/G3@&/<2=;3@
$0'%11-1)*$/(%$5('.($40,0*"(%@3AA
&7BB3@'5:7='B3BH3@&+7:A=<#/DB/#/<2!7>>:
(3/167<5/A
1=;>CB/B7=</:B67<97<54=@;722:3A16==:ABC23<BA#14-5+1210 '7&$6,10$66+( 6+
06(40$6,10$.10)(4(0&(10,0'7&$6,10
&=2@K5C3H/@1K/"=@3<=!3L<&=;I<=<HI:3H"/<2&=0:3A
D/:C/B7=<=4/<=<:7<37<B3@D3<B7=<B=B3/16E7B6!3/@<7<5"!B= P G3/@=:2ABC23<BA 9/215,7/
101/276(4 &,(0&('7&$6,10 P  
&CAA3::'/<2#=@D75%
@B74717/:7<B3::753<13;=23@</>>@=/16($4510'7&$6,10
,/,6(''/0=C@7<&=E3%"=BB+/<2!3AB3@
+63<=44B/A97A=<B/A9(63/4431B7D3
@=:3=4=44B/A9036/D7=@7<</@@/B7D313<B3@32:3/@<7<53<D7@=<;3<BA41&((',0*51)
06(40$6,10$.10)(4(0&(1046,),&,$.06(..,*(0&(,0'7&$6,10 P '>@7<53@2C1/B7=</:
'/9C:9C3/9C:AC9+7B==<'#5/@;9/8=@<E7E/B%%/B/@/<CB/>=@<%'C@/@3C<516/7+
%/B/@/<CB/>=@<%/<2'C0A==<B=@<%
 72A;/97<5<B35@/B7<5;/167<3:3/@<7<5
5/;7471/B7=</<2A=17/:1=<B3FB7<'("32C1/B7=<,06(40$6,10$.&10)(4(0&(106($&+,0*
$55(55/(06$0'.($40,0*)14(0*,0((4,0*!
P
'6/;7@/<2!3D7<(@/<A4=@;/B7=<A=41=;>CB/B7=</:B67<97<5>@/1B713A7<3:3;3<B/@G
A16==:=<B630/A3=4/@B74717/:7<B3::753<13B316<=:=573A41&((',0*51)" 81.7/( P  
D/<!==<"23@C7<D/<=5(/<2D/<"3@@7J<0=3@
1B7D/B7=<=4
7</11C@/B3>@7=@9<=E:3253/4431BA>@7;/@GA16==:ABC23<BAS;3B/1=5<7B7D38C25;3<BA/<2
1/:70@/B7=<($40,0*$0'05647&6,10  P +/<5#/<2=6<A=<"32C1/B7=<4=@ P =<<31B7<51=<13>BAB=6756A16==:
;/B61C@@71C:C;41&((',0*51)6+(#14-5+1210'7&$6,10,046,),&,$.06(..,*(0&( +7:A=<""3/AC@7<5>@=5@3AA7=<AAA3AA;3<BAB@C1BC@3AC<23@:G7<5/:3/@<7<5>@=5@3AA7=<1740$.1)(5($4&+,0 &,(0&(!($&+,0*!+()),&,$.1740$.1)6+($6,10$.551&,$6,10)14
(5($4&+,0 &,(0&(!($&+,0*   P -=C8C<,7?7<5+/<27<A=<
(3FB0==9A3@73A=<@B74717/:<B3::753<134=@:3;3<B/@G/<2"722:3'16==:A$56+,0$14/$."0,8(45,694(55-C9C</<2(/<5,C<2/;3<B/:A=4@B74717/:<B3::753<134=@756'16==:A$56+,0$
14/$."0,8(45,694(55.6=C,(/<5/:3G"6;/2'/<2/7.
Q#=EE/<BB=B3/167B4=@@3/:R
<B@=2C17<5;/167<3:3/@<7<5/A/A173<B747127A1=D3@GB==:4=@ P B3/163@A41&((',0*51)
06(40$6,10$.10)(4(0&(1046,),&,$.06(..,*(0&(,0'7&$6,10P'>@7<53@

588

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Executive Functions Training-oriented
Digital Games: Effectiveness andExperience
Chi-Fang HUANGa* , Hui-Ling HSUa & Zhi-Hong CHENa, b
Graduate Institute of Information and Computer Education, National Taiwan Normal
University, TaiwanbInstitute for Research Excellence in Learning Sciences, National Taiwan Normal
University, Taiwan*81008001e@ntnu.edu.twaAbstract: Executive function governs important cognitive abilities such as cognitive
resource allocation, which is related to daily life and can affect learning performance.
However, current executive function training mostly relies on self-administered tests,
which leads to a dull process, and low interest, thereby influencing training
effectiveness. To assist the affective domain during training, previous researchattempted to gamify the training. However, prior studies indicated that the training
intention might be influenced by the game-based design, and the game scenario also
impacts the training process. Therefore, we developed a training-oriented digital gamethat closely simulates real-life situations to train executive function based on classic
training, and discussed its effectiveness, and user experience. 38 college students
were recruited to experience the training-oriented digital game and classic games andfound that the training-oriented digital game effectively improved inhibition ability in
executive function. Additionally, the game developed in this study showed significantly
higher confidence and satisfaction than the classic game, while there was no significant
difference in flow, positive affect, and behavioral intention. This indicates that the gamedeveloped in this study could be used to train executive function while maintaining
support for the affective domain of gaming. We discuss further implications in the study.
Keywords: Executive functions, Digital games, Situational, Training-oriented game

1. Introduction
In the field of cognitive development, the development of executive function (EF) andrelated research have received increasing attention. EF refers to the cognitive ability toregulate and allocate cognitive resources, which significantly impacts reading, problemsolving, planning, decision-making, and other life-related skills (Miyake et al., 2000; Baggetta& Alexander, 2016). In comparison to Intelligence Quotient (IQ), which is commonly used inintelligence testing and future performance prediction, EF is a better predictor of life quality,
mental and physical health, and future achievement (Ahmed et al., 2019; Follmer, 2018; Moffitt
et al., 2011). This shows the importance and recent trend of studies related to EF. Miyake etal. (2000) identified three core cognitive abilities of EF: updating, inhibition, and shifting.
Among these three, inhibition refers to the ability to resist interference from irrelevant
information or impulses during cognitive processing, emphasizing the control of cognitive
resources. As human attention is limited, inhibition has a certain degree of influence on otherabilities and performances. This ability not only requires control over cognitive resources, butalso affects other cognitive abilities or performances related to everyday situations, directly or
indirectly affecting learning outcomes, work, and life. Therefore, it is particularly important anddeserves special attention.
Studies have found that EF is not only an innate ability but can also be improved
through training (Diamond, 2013; Kassai et al., 2019). And digitalized training has been
particularly effective in enhancing EF (Jak et al., 2013). However, previous studies often use
existing EF tests with minor adjustments and repeat them to achieve training effects, such as589

n-back (Soveri et al., 2017; Lo et al., 2021), Go-NoGo (Tschuemperlin et al., 2019; Najberg
et al., 2021). Therefore, these training systems are often composed of simple materials andcontinuous repeat of training. This leads to boredom and reduced intention for training (Prins
et al., 2013). Moreover, the training effects may decrease over time without continued training,
resulting in a need for longer training periods or repeated training sessions to maintain training
effects (Anguera et al., 2021; Söderqvist et al., 2012). Therefore, to maintain and sustain the
training effects of EF and to affect daily life positively, it is crucial to consider users' affective
experiences and intention of training.
To improve user experience and training intention, digital games are often used in the
training process. For example, McCord et al. (2020) asked elderly people to play Star Wars
Battlefront and show significant enhancement on EF after three weeks. Both short-term andlong-term training is effective in achieving training goals and positively impacting affective
aspects, optimizing the gaming experience, and enhancing training intention (Anastasiadis etal., 2018; Stanmore et al., 2017; Bediou et al., 2018; Nagle et al., 2018). Martincevic andVranic (2020) also used multiple games for EF training. However, no significant improvement
was shown and this may be because the casual video games are not designed for EF training.
Despite the effectiveness of enhancement on EF divided in prior studies, the positive effect ofgame-based design is still undeniable. Digital games provide adaptive information and training
content based on users' needs, levels, and performance (Bennis & Amali, 2019). The levelsettings in games not only allow users to choose suitable levels for practice, but also challenge
users to establish goals, enhance achievement, stimulate learning motivation, and even affectmetacognition and performance (Sun-Lin & Chiou, 2017).
In recent years, there have been studies attempting to develop game-based training toespecially aim at improving EF. Homer et al. (2019) focused on training inhibition ability innearly a hundred adolescents, using digital games to improve EF. They found that not only
the training significantly improved EF, but the design of game elements also impacted users'
preferences and interests. In other studies, Ober et al. (2021) developed a digital EF training
game called Gwakkamole, which required participants to click on regular avocados on thescreen and avoid clicking on avocados wearing a helmet. The empirical results indicated that
this game effectively improves EF. Another study found that the effectiveness of different subabilities of EF varies in the same training. Moreover, the design of challenging game elementsin the game also has an impact on user experience (Wells et al., 2021). In addition, there is aseries of studies that developed EF training games with aliens, including The Alien Game
(Homer et al., 2018) and All You Can ET (Mayer et al., 2019). The empirical studies using
these games to train EF, show significant training effectiveness of EF. Aside from theeffectiveness, the impact of game design on learners is also discussed. In addition toestablishing goals and achievements in challenging levels, which can help improve training
effectiveness, real-time feedback in the game also has certain assist effectiveness for users(Parong et al., 2020).
Notably, most scenario designs are set in virtual worlds, far from real-life situations.
However, Gray et al. (2019) have suggested that the effectiveness of EF training is influenced
by the degree to which the training process mimics real-life experiences and actions, andwhether the effects can be maintained. Therefore, this study aims to develop a trainingoriented digital game that is closer to real-life situations to explore whether it can achieve thegoal of improving EF and to further discuss the gaming experience when training. The
research questions are as follows: 1. Does the training-oriented digital game enhance
inhibition ability more effectively than classic games? 2. Does the training-oriented digital
game produce a more positive gaming experience than classic games?

2. System design
This study aims to develop a series of 4 EF training games based on real-life scenarios,with inhibition as the training target, as shown in Figure 1. Based on the commonly used Simontask (Simon et al., 1967) and Stroop test (Stroop, 1935) for measuring inhibition ability, thegame system includes two types of games: button selection and voice recognition. The button
selection games are mainly adapted from the Simon task and are divided into three stages
with increasing difficulty. In the first stage, users need to click on the left or right according to590

the required rule after viewing an image. After getting familiar with the rules, the second stageof the game introduces interference from the left and right movements and flips to train users'
ability to inhibit habitual responses. The third stage adds rule changes, requiring users to makechoices opposite to their familiar operations in specific situations, in order to train ruleswitching, inhibition of previous habits, and increase game challenges. On the other hand, thevoice recognition game is based on the Stroop test, and in multiple consecutive stages, users
are required to speak out the content according to the rule requirements. During this period,
rules are gradually stacked, and users need to inhibit interference from other types of icons
and speak out corresponding content based on the rules.

Figure 1. Samples of screens of training-oriented game system
The system was developed using Construct 3 with the aim of training EF skills. To
enhance the user gaming experience and encourage intention to keep on training, gameelements are added, which include real-life scenarios, sound effects, animations, levels, time
limits, and scores. Real-life scenarios are the main characteristics of the game design to makethe training process more realistic, and we selected sports, food, and pet interaction as thetheme of the game series. Different training scenarios, such as games, restaurants, androoms, were presented in rich colors to enhance the sense of immersion. Aside from thescenarios, sound effects and animations were provided to offer users instant feedback duringtransitions, enabling them to know immediately whether their answer was correct or incorrect.
This helped users to stay focused on the training objectives and remain immersed in theprocess. Then, the levels were designed to gradually increase in difficulty. This adds achallenging element to the game. Also, time limits were incorporated into the game to gain
challenges. A countdown timer is displayed on the screen to remind users to respond quickly
and accurately, with 10 seconds for each response, and faster responses result in higherscores. Last, the scoring system is also important. A scoring board or heart icons that closely
matched the packaging scenarios, providing users with a quantifiable and intuitive sense ofachievement.

3. Method
3.1 Research design
The research design focused on examining the effectiveness and experience of
training-oriented digital games developed in this study. The game usage was set as theindependent variable, with the participants' original EF ability as the control variable, and thetraining effectiveness and experience as the dependent variables. To address the researchquestions, a pretest was conducted before the experiment to assess the participants' initial EF
ability. After each intervention using the game, a post-test was conducted to examine thetraining effectiveness, and a questionnaire was administered to understand the game
experience, as shown in Figure 2.

Figure 2. Research process and instruments591

For the game intervention, the participants were divided into an experimental group anda control group, which used the training-oriented digital game and classic game, respectively.
The study design referenced previous research designs (Homer et al., 2019; Plass et al.,2019), and the participants were instructed to operate the digital game for 15 minutes within
an effective training period that did not induce fatigue.

3.2 ParticipantsThis study recruited 38 college students without any cognitive-related disorders as
participants, aged between 20 to 24 years old, including 10 males and 28 females. To avoid
the influence of gaming experience on the training process and experience, the weekly gaming
duration of the participants was investigated. 18 participants played digital games for morethan half of the days in a week, while 20 participants played games for less than half of thetime. This shows an evenly distributed pattern and is expected not to affect the research
results.

3.3 Research instrument
The research instruments used in this study include not only the training-oriented digital
games mentioned above but also classic games used in the control group as a comparison.
To assess the effectiveness of the intervention, both EF assessment and questionnaires wereused to examine both cognitive and affective aspects, which are described in detail below.
To parallel the training-oriented digital games, 4 classic games were selected. The
selected classic games were common ones, including Pong, FreeCell, Pinball, andMinesweeper. These games were chosen to familiarize participants with the game rules. And
to examine the effectiveness of the training, Trail making test was used in the pretest and posttest to assess participants' original inhibition ability and training outcomes. The test isfrequently used in previous studies related to EF training (Sousa et al., 2020; Sosa & Lagana,2019), Comprises a series of circle-linking tasks, and participants have to inhibit the effects ofdifferent types of circles, while others require participants to link circles in a specific order.
After the intervention, a five-point Likert scale questionnaire was used to understandparticipants' gaming experience. The gaming experience questionnaire includes three
dimensions, namely flow, positive affect, and behavioral intention (Law et al., 2018; Hwang etal., 2013). The flow and positive affect dimensions consist of 4 questions each, while
behavioral intention has 5 questions.

3.4 Data analysis
To answer the research question, this study utilized IBM SPSS for data analysis.
Specifically, an Analysis of Covariance (ANCOVA) was conducted using the differences ingames as the between-group variable, pretest scores as the covariate, and post-test scoresas the dependent variable to examine whether there were significant differences in
performance between groups after intervention. In addition, an Analysis of Variance (ANOVA)
was employed to investigate the user experience, with game intervention differences as theindependent variable and participants' experiences as the dependent variable. The
questionnaire on experience was used for comparative analysis.

4. Result and discussion
4.1 Training effectivenessAccording to the research results, there is a significant difference between pre-test andpost-tests of inhibition ability, as shown in Table 1. Through ANCOVA, it shows no significant
difference in pre-test scores between the two groups. Moreover, after using the training
games, the inhibition ability of the training game group was significantly improved comparedto the classical game group, with an average increase from 0.37 to 0.51, and an F-value of
5.34. It can be concluded that participants were able to enhance their inhibition ability more
effectively after using training-oriented digital games than classical games.

592

Table 1. The result of training effectiveness on inhibition
Group

Inhibition

Pre-test

Post-test

Mean

SD

Mean

SD

Training game (n=19)

0.37

.29

0.51

.24

Classic game (n=19)

0.69

.29

0.39

.22

F

eta2

5.34*

.132

*p < .05烊**p < .01
The results of the present study are consistent with previous research, which showedthat EF training-oriented games can effectively achieve the goal of enhancing EF abilities.
Homer et al. (2019) conducted inhibition ability training on adolescents using a digital gameand found significant improvement in training effectiveness. Similarly, Wells et al. (2021)
developed a digital EF training game, Gwakkamole, and not only indicates that this game
could effectively train EF but also mentions that the design of challenging game elements in
the game would also have an impact on the training process. Another series of studies of EF
training games take aliens as the game theme, including The Alien Game (Homer et al., 2018)
and All You Can ET (Mayer et al., 2019). Empirical research similarly found significant
improvements in inhibition ability and noted the impact of game design on learners.
Establishing goals and achievements through the challenges of the game can help improvetraining effectiveness, and real-time feedback in the game can also assist users (Parong etal., 2020). These findings are consistent with the present study, as the design of challenging
elements, such as levels and scores, as well as real-time feedback, such as sound effects andanimations, has become an essential part of the game process to assist learners in achieving
better training results.
However, the above research scenarios differ from real-life situations. The present
study takes activities that are closer to real-life situations as the game scenario, based on the
gap identified in previous research (Gray et al., 2019). The results of the study also showedthat designs that are closer to real-life situations can effectively improve inhibition ability.

4.2 Game experience
In addition to exploring the training effectiveness for training-oriented games, this study
further investigates the game experience. Firstly, the reliability of the questionnaire wasconfirmed with a Cronbach's alpha value of 0.95 for the training games group and 0.93 for theclassic games group, demonstrating the trustworthiness of the questionnaire results.
Subsequently, an ANOVA was conducted to compare participants' perceptions of training
games and classic games, including three dimensions: flow, positive affect, and behavioral
intention. The results showed no significant differences in these dimensions between the twogroups, indicating that games designed for training purposes and classic games can bothachieve a positive user experience. The results are presented in Table 2.
Table 2. Comparison of game experience between groups
F

eta2

1.04

1.96

2.25

3.49

0.77

0.61

0.53

3.32

0.81

0.10

0.09

Training games (n=19)

Classic games (n=19)

Mean

SD

Mean

SD

Flow

3.64

1.10

3.16

Positive affect

3.72

1.07

Behavioral intention

3.22

1.00

593

Homer et al., (2019) conducted an empirical study on digital training games and foundthat the game design has an impact on user preference and long-term usage intentions. In
other empirical studies, Wells et al. (2021) compared the performance of EF training gameswith a control group and found that EF training-oriented designs have lower user affect andlong-term usage intentions. The study also further emphasizes the impact of game elements,
including challenges, on user experience. As the prior study has pointed out, enhancing EF
requires continuous training, and users' affective experience and long-term usage intentions
should be given more attention. Therefore, this study aims to design EF training-oriented
games that not only focus on training effectiveness but also emphasize the affective domain.
The training process incorporates gaming elements such as level design, time limits, andscores, to gain challenges. Also, we use other game elements such as scenarios and realtime feedback in sound and animation to achieve a positive user experience in trainingoriented games. This study aims to bridge the research gap and guide users toward continuing
their training to achieve long-term goals in improving EF.

5. Conclusion
The executive function (EF) controls important cognitive abilities and can impact
learning performance and daily life. Although prior studies developed digital training andtraining games to enhance EF, the affective domain is yet to be improved (Wells et al., 2021).
Moreover, game scenarios of most EF training games are often far from real-life, which also
influences the training process (Gray et al., 2019). Therefore, this study developed a trainingoriented digital game with real-life scenarios to address this research gap. Through empiricalresearch, this study answers the following research questions: (1) Does the training-oriented
digital game enhance inhibition ability more effectively than classic games? ANCOVA results
show that inhibition ability significantly improved through training with the training-oriented
digital game compared to the classic game, demonstrating the effectiveness of this trainingoriented digital game design. (2) Does the training-oriented digital game produce a morepositive gaming experience than classic games? The result of analysis through questionnaires
shows no significant differences in the flow, positive affect, and behavioral intention
dimensions between the training-oriented game and the classic game. This indicates that bothtypes of games can achieve the goal of a positive gaming experience and the training-oriented
game can guide users to continue training to enhance EF. This provides insights into thedesign of EF training games, the game elements and situation should be emphasized.
The present study has some limitations and suggestions for future research. Due to theconstraints of time, the sample size of this study was limited. Therefore, future studies should
focus on larger sample sizes to improve the appropriateness of research inference. In addition,
the training time was only 15 minutes. Although the design was based on previous research
and was found to be effective, it is recommended to increase the training duration or
frequency. Also, as the effectiveness of enhancing inhibition has been proven, the training aimof other EF sub-skills should be considered in future game development and studies. Finally,
the training game was designed with scenarios close to real life, and the study mainly
discussed its effectiveness and gaming experience. Research further comparing real-life andvirtual situations is expected to establish the importance of situation setting.

References
Homer, B. D., Ober, T. M., Rose, M. C., MacNamara, A., Mayer, R. E., & Plass, J. L. (2019). Speed
versus accuracy: Implications of adolescents' neurocognitive developments in a digital game to
train
executivefunctions.
Mind,
Brain,
and
Education,
13(1),
41-52.
https://doi.org/10.1111/mbe.12189
Ober, T. M., Brenner, C. J., Olsen, A., Homer, B. D., & Plass, J. L. (2021). Detecting patterns ofengagement in a digital cognitive skills training game. Computers & Education, 165, 104144.
https://doi.org/10.1016/j.compedu.2021.104144
Wells, A., Parong, J., & Mayer, R. E. (2021). Limits on training inhibitory control with a focused video
game. Journal of Cognitive Enhancement, 5(1), 83-98. https://doi.org/10.1007/s41465-020-00184-2

594

Incorporating tangible rewards into
gamification increases students’ identified
regulation in fully online learning
Ya XIAOa*, Khe Foon HEWa
Faculty of Education, The University of Hong Kong, Hong Kong SAR, China*yaxiao@connect.hku.hkaAbstract: Intangible rewards (such as virtual points and virtual badges) have often
been used in gamified learning contexts to motivate learners. However, such intangible
rewards are not stimulating to all learners. Studies have reported that learners express
their desire to redeem intangible rewards for some utilitarian resources or benefits.
Considering that empirical evidence regarding how tangible rewards impact students’motivation is still lacking, the present study applied a randomized controlled trial
approach to explore the effects of tangible rewards on students’ identified regulation (atype of autonomous motivation) and external regulation (a type of controlled
motivation). The study was conducted in a fully online gamified flipped class. Individual
students were randomly assigned either to the tangible rewards group (EG = 28) or theintangible rewards group (CG = 29). Students in EG reported significantly higher
identified regulation than those in CG while no significant difference was found in termsof external regulation. The results theoretically supported the standpoint that tangible
rewards can help promote autonomous/self-determined motivation and gave practicalguidance for educators who are interested in using tangible rewards in online gamified
courses.
Keywords: Gamification, redeem, tangible rewards, motivation

1. Introduction
Gamification has been used to enhance greater student engagement, especially in the
online learning environment (Zainuddin et al., 2022). Gamification rewards (Nicholson,2015), usually take two forms — intangible and tangible. Intangible rewards refer to virtual
badges/points and verbal praise that do not contain any utilitarian benefits to recipients(Meder et al., 2018). Tangible rewards refer to those that involve material goods (e.g.,
money) or some utilitarian benefits, (e.g., opportunities to participate in preferred activities)(Cameron et al., 2001; Kappen & Orji, 2017).
In most previous gamified learning studies, intangible rewards are more commonlyused (Bai et al., 2021; Meder et al., 2018). Although non-utilitarian rewards can bring about
enjoyable experiences in the gamified learning setting (Landers et al., 2015), not all learners
favor these intangible factors in the long run. Instead, they tend to value the option of
exchanging such rewards for more utilitarian or material resources (e.g., Huang & Hew,
2018).

2. Related Literature
2.1 Theoretical Controversy
The use of tangible rewards in education has been controversial and remains up for debate(Kappen & Orji, 2017). The controversy mainly revolves around how tangible rewards impact
students’ autonomous motivations.
The self-determination theory makes a distinction between autonomous andcontrolled motivations based on their underlying regulatory processes and their associated595

levels of self-determination. Autonomous motivation is a type of motivation in which
individuals engage in a behavior because they find it inherently interesting, enjoyable, ormeaningful. Controlled motivation, on the contrary, involves feeling pressured or coerced toengage in a behavior or activity due to external factors. Identified regulation, is a type ofautonomous motivation that involves people identifying with the personal value andimportance of the behavior for themselves and thus accepting it as their own. External
regulation is a type of controlled motivation that involves people engaging in a behavior or
activity solely to obtain a specific external outcome or reward (Ryan & Deci, 2000).
Specifically, some theorists criticize the detrimental effect of tangible rewards
because tangible rewards tend to be perceived as controlling (i.e., being forced to complete
designated tasks because of the reward). The perceived locus of causality changes fromoneself to external rewards (Deci et al., 1975). They may begin to view the engagement in
the designated tasks as a means to an end rather than as an inherently meaningful pursuit.
Namely, they tend to feel that their efforts are only worthwhile if they are rewarded, rather
than because they are improving their abilities. In this case, giving tangible rewards can leadto a decrease in identified regulation and an increase in external regulation.
However, other theorists argue that tangible rewards can be helpful if usedappropriately. According to the social learning theory, when the rewards are tied to specificlevels of performance (e.g., exceeding a certain score or surpassing other people’s scores),an individual’s perceived competence or self-efficacy will be enhanced (Bandura, 1986). The
tangible rewards act as a social validation of one’s competence, whereby a person’s talents
and abilities (internal factors) are perceived as the reasons for the rewards, rather than the
rewards being perceived as the reasons for high-level performance (Cameron & Pierce,2002). As a result, individuals' identification with the learning activity is strengthened insteadof being harmed.

2.2 Previous Studies
Only a limited number of studies that have directly compared the use of tangible andintangible rewards in gamified educational contexts. For example, Ortega-Arranz et al.
(2019) conducted a study that compared the behavioral engagement of students in a
Massive Open Online Courses (MOOC) course among three groups: a “gamified group”(awarded with virtual badges only), a “tangible reward gamified group” (awarded with virtual
badges redeemable for certain learning benefits), and a control group (without rewards). No
significant difference was found between the two gamified groups although both of themoutperformed the control group.
More recently, Bai et al. (2021) explored the effects of tangible rewards on students’behavioral engagement and learning performance in a fully online university course,
comparing students who were given virtual points to students who were given virtual points
redeemable for high-quality learning materials. While tangible rewards encouraged students
to create and reply to more posts, no significant difference was observed in their learning
performance.
In general, there is still a lack of research that directly compares the effect of tangible
and intangible rewards in fully online learning setting. In addition, the current empiricalstudies on gamification have not explored the effects of tangible rewards on students’motivation.

3. Research Questions
The present study aims to fill the research gap by examining two research questions:
RQ1: What is the effect of tangible rewards on students’ identified regulation in online
learning?
RQ2: What is the effect of tangible rewards on students’ external regulation in online
learning?

596

4. Methodology
4.1 Context and Participants
The present study was conducted in the higher education context, in a fully online
International Business course. The course comprised of eight sessions, each lasting 3.5hours, and was aimed at preparing undergraduate students for a postgraduate entrance
examination. Students who planned to take the examination and enter an international
business master’s degree program voluntarily signed up for the course.
Altogether 57 participants with an average age of 21 were enrolled in this course. The
participants were individually randomly assigned to either the control group (CG) or theexperimental group (EG). The only difference between the two groups was that the EG
students could redeem exclusive learning materials with their virtual points while the CG
students could not. All the redeemable rewards were later given to the CG students after theexperiment to avoid ethical issues.

4.2 Gamified Flipped Class Design and Redemption Scheme
In a typical flipped classroom, students engage in (1) pre-class computer-basedlearning (such as video lectures) and (2) in-class interactive learning. Students are expected
to familiarize themselves with pre-class learning materials and focus on higher-levelcognitive activities, such as peer learning, and problem-solving (Bishop & Verleger, 2013;
Sointu et al., 2023).
To increase student engagement in the pre-and in-class learning activities, points,
badges, and leaderboards (known as the PBL triad) were used for both CG and EG. The
PBL triad is one of the most commonly used gamification elements (Leitão et al., 2022).
Please see Table 1 for a detailed gamification design.
Table 1. Gamification design in the flipped class
PointsDescription

5

5 points for downloading pre-class learning materials (only the first
download will be counted)

10

10 points for answering each pre-class MCQ question correctly (5MCQ questions in total)

5

5 points for completing all pre-class learning activities, including
downloading the pre-class learning activities and fulfilling pre-class
tests

10

10 points for answering each in-class MCQ question correctly (10MCQ questions in total)

10

10 points for voluntarily answering or posing questions in class (onlythe first three attempts will be counted)

BadgesDescription

Fully PreparedThis badge rewards students whose accuracy rates in the pre-classquizzes are greater than or equal to 80%

TaskCompletion

This badge rewards students who complete all pre-class learning
activities, including downloading the pre-class learning activities andfulfilling pre-class tests

Task Master

This badge rewards students who complete the ten items in the inclass quizzes

597Quiz WhizThis badge rewards students whose accuracy rates in the in-classquizzes are greater than or equal to 80%

Leaderboards

Description

Leaderboard bysessionThis leaderboard announces students’ rankings and accumulated
points by each weekly session.

Overall
Leaderboard

This leaderboard announces students’ overall rankings andaccumulated points throughout the whole course

Tangiblerewards

Description

Exclusive
learning
materials(EG only) The rewards could be redeemed weekly for students who
accumulated 160 virtual points or above that week.

In this study, the course was conducted through a synchronous videoconferencing tool.
All the pre- and in-class learning activities were conducted using a self-designed gamified
learning platform.

4.3 Data Collection and Analysis
Students identified regulation and external regulation was measured by scales adapted fromthe Situational Motivation Scale (SIMS, Guay et al, 2000). The Shapiro–Wilk statistics
showed a non-normal distribution of student responses in the CG (W = 0.924, p = .038) andthe EG (W = .846, p = .001) in terms of identified regulation. Similarly, in terms of external
regulation, student responses in the CG (W = .917, p = .025) and EG (W = .915, p = .026)
were also non-normally distributed. In this case, students’ responses to these two scales
were examined using the Mann–Whitney U test.

5. Results
5.1 Effects on Students’ Identified Regulation
The questionnaire results indicated good reliability (Cronbach’s alpha = 0.947). As shown in
Table 3, the students in the EG (Mdn = 6.167, SD = 1.044) reported a significantly higher
level of identified regulation than those in the CG (Mdn = 4.667, SD = 1.107; U = 630.000, p< .05, r = 0.478).

5.2 Effects on Students’ External Regulation
The questionnaire results indicated relatively acceptable reliability (Cronbach’s alpha =0.617) (Hajjar, 2018; Wim et al., 2008). The students in the EG (Mdn = 3.333, SD = 1.373)
reported no significant difference in terms of external regulation with those in the CG (Mdn =4.000, SD = 1.119; U = 330.000, p = .222) (see Table 3).
Table 3. Effects of tangible rewards on students identified regulation and external regulation
Measures
Identified regulation

Comparison group
EG
CG
External regulation
EG
CG
Note. * significant using p < .05598

Mdn (SD)
6.167 (1.044)
4.667 (1.107)
3.333 (1.373)
4.000 (1.119)

p-value (effect size)
p = .000*, r = 0.478
p = .222

6. Discussions and Conclusions
This study applied a randomized controlled trial approach to examine the effects of tangible
rewards redeemed through intangible rewards on students’ identified regulation and external
regulation in a fully online gamified flipped class. The results showed that the use of tangible
rewards significantly increased students’ identified regulation while indicating no significant
influence on students’ external regulation.

6.1 Implications
The findings shed more light on the theoretical controversy over the effects of tangible
rewards. Specifically, the increase in identified regulation indicates that tangible rewards
facilitate the internalization of motivation — the process by which an individual adopts andintegrates an external regulation or value into their sense of self (Deci & Ryan, 2015). Boththe reward scheme and the reward type are crucial to the effectiveness of tangible rewards
in promoting internalization.
In this study, the redemption of tangible rewards is associated with a performancestandard (i.e., earning over 160 virtual points, which is roughly 80% of the full virtual points
for a week). This criterion provides valuable information that can positively reinforce
students' competence (Bandura, 1986), thereby mitigating the potential negative effects of
the reward being a controlling factor (Ryan & Deci, 2000). Additionally, according to theextended attribution theory, tangible rewards tied to performance levels enhance perceived
competence and encourage individuals to attribute their success to their own abilities(Lepper et al., 1996).
The type of tangible rewards used in this study is compatible with learning purposes,i.e., exclusive learning materials that may boost students’ scores in the postgraduate
entrance examination. In other words, the redeemed tangible rewards are personally
meaningful/relevant to students as they are closely related to their valued future goals. Suchpersonal relevance is crucial to identified regulation (Vansteenkiste et al., 2018). That
explains why in the redeeming procedure, students tend to identify learning activities withpersonal meaning, endorsing that learning is good for themselves.
To summarize, theoretically, this study provides evidence supporting the theoreticalstandpoint that tangible rewards can be helpful in promoting autonomous/self-determined
motivation if used appropriately. Practically, this study provides two important implications for
applying tangible rewards in the gamified learning context, (1) linking tangible rewards to a
standard of performance; (2) linking tangible rewards to students’ valued learning goals.

6.2 Limitations
Three primary limitations need to be acknowledged in this study. Firstly, the research
focused solely on a fully online class that utilized synchronous videoconferencing
technology. Therefore, the outcomes of this investigation may not be applicable to otherlearning contexts, such as asynchronous learning in MOOCs or face-to-face learning.
Secondly, the tangible rewards implemented in this study may only be effective for students
who are preparing for the entrance examination. The use of this kind of reward may not be
generalizable to other situations, highlighting the importance of providing rewards that matchthe specific needs of the target population. To address this issue, future research could
include a pre-survey to gather information about students’ requirements and preferences.
Thirdly, the relatively small ample size may limit the generalizability of the findings.

Acknowledgements
The research was supported by a grant from the Research Grants Council of Hong Kong
(Project Reference No: 17616020).

599

References
Bai, S., Gonda, D. E., & Hew, K. F. (2021, December). Effects of tangible rewards on student learning
performance, knowledge construction, and perception in fully online gamified learning. 2021
IEEE International Conference on Engineering, Technology & Education, Wuhan, China, 899–
904.
Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Upper Saddle
River, NJ: Prentice Hall.
Bishop, J., & Verleger, M. A. (2013, June). The flipped classroom: A survey of the research. 2013ASEE Annual Conference & Exposition, Atlanta, Georgia, 23.1200.1–23.1200.18.
Cameron, J. D., Banko, K. M., & Pierce, W. D. (2001). Pervasive negative effects of rewards on
intrinsic motivation: The myth continues. The Behavior Analyst, 24(1), 1–44.
Cameron, J. D., & Pierce, W. (2002). Rewards and intrinsic motivation: Resolving the controversy.
London: Bergin & Garvey.
Deci, E. L., & Ryan, R. M. (2015). Self-determination theory. International Encyclopedia of the Social& Behavioral Sciences, 21, 486-491.
Deci, E. L., Cascio, W. F., & Krusell, J. (1975). Cognitive evaluation theory and some comments on
the Calder and Staw critique. Journal of Personality and Social Psychology, 31(1), 81-85.
Guay, F., Vallerand, R. J., & Blanchard, C. (2000). On the assessment of situational intrinsic andextrinsic motivation: The Situational Motivation Scale (SIMS). Motivation and Emotion, 24(3),175-213.
Hajjar, S. (2018). Statistical analysis: internal-consistency reliability and construct validity.
International Journal of Quantitative and Qualitative Research Methods, 6(1), 46-57.
Huang, B. Y., & Hew, K. F. (2018). Implementing a theory-driven gamification model in higher
education flipped courses: Effects on out-of-class activity completion and quality of artifacts.
Computers & Education, 125, 254–272.
Kappen, D. L., & Orji, R. (2017). Gamified and persuasive systems as behavior change agents forhealth and wellness. XRDS: Crossroads, The ACM Magazine for Students, 24(1), 52–55.
Landers, R. N., Bauer, K. N., Callan, R. C., & Armstrong, M. B. (2015). Psychological theory and thegamification of learning. In T. Reiners & L. C. Wood (Eds.), Gamification in education andbusiness (pp. 165–186). Springer International Publishing.
Leitão, R., Maguire, M., Turner, S., & Guimarães, L. (2022). A systematic evaluation of gameelements effects on students’ motivation. Education and Information Technologies, 27(1), 10811103.
Lepper, M. R., Keavney, M., & Drake, M. (1996). Intrinsic motivation and extrinsic rewards: A
commentary on Cameron and Pierce’s meta-analysis. Review of Educational Research, 66(1), 532.
Meder, M., Plumbaum, T., Raczkowski, A., Jain, B., & Albayrak, S. (2018). Gamification in ecommerce: Tangible vs. intangible rewards. Proceedings of the 22nd International Academic
Mindtrek Conference, Finland, 11–19.
Nicholson, S. (2015). A recipe for meaningful gamification. In Gamification in education and business(pp. 1–20). Springer, Cham.
Ortega-Arranz, A., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., Martínez-Monés, A., Gómez-Sánchez,E., & Dimitriadis, Y. (2019). To reward and beyond: Analyzing the effect of reward-based
strategies in a MOOC. Computers & Education, 142, 103639.
Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: classic definitions and new
directions. Contemporary Educational Psychology, 25(1), 54e67.
Sointu, E., Hyypiä, M., Lambert, M. C., Hirsto, L., Saarelainen, M., & Valtonen, T. (2023). Preliminary
evidence of key factors in successful flipping: Predicting positive student experiences in flippedclassrooms. Higher Education, 85, 503-520.
Wim, J., Katrien W., Patrick D. P., & Patrick V. K. (2008). Marketing Research with SPSS. PrenticeHall; Pearson Education.
Vansteenkiste, M., Aelterman, N., De Muynck, G., Haerens, L., Patall, E., & Reeve, J. (2018)
Fostering personal meaning and self-relevance: A self-determination theory perspective oninternalization. The Journal of Experimental Education, 86(1), 30-49.
Zainuddin, Z., Farida, R., Keumala, C. M., Kurniawan, R., & Iskandar, H. (2022). Synchronous online
flip learning with formative gamification quiz: Instruction during COVID-19. Interactive
Technology and Smart Education, 19(2), 236–259. https://doi.org/10.1108/ITSE-01-2021-0002

600

" !   !" " "   #" !
#" !"$  #" ! #"







                                  '*)%%"% *)%$!$$+'().  $$'( %"" %"*#$+'().,%'! #'
>>G<5B;D?I 98I 7B   449>7 5=2?4954 C3166?<49>7 D? D85 D51389>7 @B?35CC 31> 25 56653D9F5 9>
9=@B?F9>7 CDE45>DC C@1D91< 129<9D95C  ?G5F5B 65G CDE495C 81F5 255> 3?>4E3D54 D?
5H@<?B5 D85 56653DC ?6 49665B5>D 5=2?4954 C3166?<49>7C ?> CDE45>DC C@1D91< 129<9D95C *85B56?B5D85@EB@?C5?6D89CCDE4IG1CD?9>F5CD971D5D8556653DC?649665B5>D5=2?4954
C3166?<49>7C?>CDE45>DCC@1D91<<51B>9>7 *8B55DI@5C?65=2?4954C3166?<49>7CG5B5
45C97>543?>DB?<<12<51>9=1D9?>C3166?<49>79>CDBE3D9?>1<75CDEB5@<EC1>9=1D9?>
C3166?<49>71>4@8IC931<?2:53DC3166?<49>7&% 1C54?>D89CG53?>4E3D541
AE1C95H@5B9=5>D1< CDE4I 9> G8938  5<5=5>D1BI C38??< CDE45>DC G5B5 B1>4?=<I
49F9454 9>D? D8B55 7B?E@C D? <51B> 75?=5DBI 9> 1 71=521C54 <51B>9>7 3?>D5HD 1>4
=51CEB54 D859B ;>?G<5475 <51B>9>7 C@1D91< 129<9DI 1>4 6<?G 5H@5B95>35  *85 B5CE<DC
C8?G54D81D1<D8?E78D85B5G5B5>?C97>96931>D49665B5>35C9>C@1D91<129<9DI1=?>7D85
D8B55 7B?E@C CDE45>DC 9> D85 &% 7B?E@ @5B6?B=54 C97>96931>D<I 25DD5B 9> ;>?G<5475
DB1>C65BD81>D85?D85BDG?7B?E@CG89<5CDE45>DC9>D85&%7B?E@1<C?814D85<?G5CD
<5F5< ?6 6<?G 5H@5B95>35  > 1449D9?> G5 9>F5CD971D54 D85 9>6<E5>35 ?6 CDE45>DCO @B9?B
@B?69395>3I?>D8556653DC?649665B5>D5=2?4954C3166?<49>7C *85B5CE<DCC8?G54D81DCDE45>DC9>D85<?G@B?69395>3I7B?E@@5B6?B=5425DD5B9>D85&%3?>49D9?>1>4D85
3?>49D9?>D81>D8?C59>D858978@B?69395>3I7B?E@  )@1D91<129<9DI1=521C54<51B>9>7=2?49543?7>9D9?>)3166?<49>7  

-D5H=5@56=@=HM75B6989:=B985G5<IA5BQG75D57=HMHCIB89FGH5B8F95GCBCJ9FF975@@5B8
A5B=DI@5H9H<9GD5H=5@F9@5H=CBG5ACB;C6>97HGCF=BGD5797CBG=GH=B;C:H<F99:57HCFGA9BH5@
FCH5H=CBGD5H=5@D9F79DH=CB5B8GD5H=5@J=GI5@=N5H=CB&=BB*9H9FG9B  GCB9C:H<9
65G=7<IA5B7C;B=H=J956=@=H=9GGD5H=5@56=@=HM=G=ADCFH5BH:CFD9CD@9HCF97C;B=N9H<9=FCKB
9BJ=FCBA9BH5B8GC@J9GD5H=5@DFC6@9AGI::M-CF6MCK9!5F8B9F  -D5H=5@
56=@=HM@9J9@GBCHCB@M8=F97H@M5::97H@95FB9FGIB89FGH5B8=B;F9DF9G9BH5H=CB5B8GC@IH=CBC:
G7=9B79 H97<BC@C;M 9B;=B99F=B; 5B8 A5H<9A5H=7G -.' DFC6@9AG "9 9H 5@ "C8;?=GG9H5@ 6IH5@GCDF98=7H@95FB9FG75F99F57<=9J9A9BH5B875F99F7<C=79G=B-.':=9@8G3CCB'5BB
 #B H<9 7IFF9BH D985;C;=75@ 7CAAIB=HM ;5A965G98 @95FB=B; =G K=89@M 58CDH98 6M
A5BM 98I75HCFG  (IA9FCIG GHI8=9G <5J9 G<CKB H<5H 98I75H=CB5@ ;5A9 =G 9::97H=J9 =B
9B<5B7=B;@95FB9FG@95FB=B;D9F:CFA5B799FF59H5@ G9@:9::=757M%INB9H7CJ59H
5@ 
5B8ACH=J5H=CB 58859H5@  #B588=H=CB98I75H=CB5@;5A9G75BDFCJ=89
GHI89BHGK=H<J=GI5@F9DF9G9BH5H=CBGC:C6>97HG=B5H<F998=A9BG=CB5@:CFA5B8GHI89BHG75B
A5GH9FGD5H=5@F9@5H=CBG=BH<9DFC79GGC:=BH9F57H=B;K=H<;5A9GK<=7<5@GC<5G5DCG=H=J9
=AD57H CB GD5H=5@ 56=@=HM /HH5@  .<9F9:CF9 GCA9 F9G95F7<9FG <5J9 HF=98 HC =BH9;F5H9
8=;=H5@;5A965G98@95FB=B;=BHCH<9H957<=B;5B8@95FB=B;DFC79GGC:GD5H=5@56=@=HM5B8<5J9
7CB:=FA98H<99::97H=J9B9GGC:98I75H=CB5@;5A9G=B=ADFCJ=B;GHI89BHGGD5H=5@56=@=HM"CI
9H5@ <5=9H5@ &=B<9B
 

601

"CK9J9F7IFF9BH98I75H=CB5@;5A9G:CF89J9@CD=B;GHI89BHGGD5H=5@56=@=HMD5M@9GG
5HH9BH=CBHCH<99GG9BH=5@:95HIF9GC:GD5H=5@7C;B=H=CB5B8B9;@97HH<9=ADCFH5B79C:9A6C8=98
7C;B=H=CB:CFGD5H=5@56=@=HM J=89B79:FCAG9J9F5@GHI8=9GGI;;9GHGH<5HH<9F9=G5@=B?69HK99B
H<9 6C8M 57H=CB 5B8 GD5H=5@ 7C;B=H=CB 5B8 H<5H D<MG=75@ ACJ9A9BH D@5MG 5B 57H=J9 FC@9 =B
GD5H=5@ 7C;B=H=CB &CN5BC "5F8  .J9FG?M 'CFG9@@5  %F5IGG  M C::@C58=B;
A9BH5@ DFC79GG=B; HC D<MG=75@ 57H=CBG =H A5M <9@D HC =ADFCJ9 GHI89BHG IB89FGH5B8=B; C:
GD5H=5@ F9@5H=CBG<=DG  -CA9 GHI8=9G <5J9 699B 7CB8I7H98 HC 89G=;B H957<=B; 57H=J=H=9G :CFGD5H=5@56=@=H=9G65G98CBH<9D9FGD97H=J9C:9A6C8=987C;B=H=CB5B8H<9F9GI@HG<5J9G<CKB
H<5HGHI89BHGK<CIG99A6C8=98G75::C@8=B;57<=9J969HH9F@95FB=B;CIH7CA9G%KCB9H5@ ,565HHI9H5@ IFH99H5@  .<9F9:CF9;=J9BH<9J5@I9C:G75::C@8=B;:CF
GHI89BH @95FB=B; 98I75HCFG A5M 7CBG=89F =BHFC8I7=B; 9A6C8=98 G75::C@8=B; HC GIDDCFH
GHI89BHGGD5H=5@89J9@CDA9BH IFF9BH@M:9K9L=GH=B;GHI8=9G<5J99LD@CF98H<969GHDF57H=79C:89G=;B=B;9A6C8=98
G75::C@8=B;HC9B<5B79GD5H=5@56=@=HM=B8=;=H5@;5A965G98@95FB=B; .<9F9:CF9H<=GGHI8M5=AG
HC =BJ9GH=;5H9 5B8 7CAD5F9 H<9 9::97HG C: H<F99 8=::9F9BH HMD9G C: 9A6C8=98 G75::C@8=B; CB
GHI89BHGGD5H=5@@95FB=B;K=H<:IFH<9F7CBG=89F5H=CBC:H<9=B:@I9B7=B;:57HCFGHI89BHG=B=H=5@
DFC:=7=9B7M   !"  !  !!$
.<9 H<9CFM C: 9A6C8=98 7C;B=H=CB GH5H9G H<5H H<9 DFC79GG C: ?BCK@98;9 7CBGHFI7H=CB =G
=BG9D5F56@9:FCAH<9D<MG=75@=BH9F57H=CBC:H<9@95FB9FK=H<H<9@95FB=B;9BJ=FCBA9BH#C5BBCI
#C5BBCI M9LD5B8=B;@95FB=B;:FCAJ=GI5@7C;B=H=J957H=J=H=9GHCD<MG=75@ACJ9A9BH
H<9F9G95F7<9FG69@=9J9H<5HH<=G5@@CKGGHI89BHGHC9B;5;9=BAI@H=AC85@=BH9F57H=CBGH<5H<9@D
GHI89BHG DFC79GG @95FB=B; 7CBH9BH 5H 5 899D9F @9J9@ B5GHCDCI@CI 9H 5@  5B8
IB89FGH5B8 56GHF57H G7=9BH=:=7 7CB79DHG  #B 588=H=CB G9J9F5@ GHI8=9G <5J9 G<CKB H<5H
9A6C8=98@95FB=B;9B<5B79GGHI89BHGGD5H=5@H<=B?=B;G?=@@GIFH99H5@ 5B8H<5HH<9
9A6C8=989@9A9BH=GDCG=H=J9@M5GGC7=5H98K=H<A5H<9A5H=75@DFCC:D9F:CFA5B79=BG=;<H5B8
=BHI=H=CB(5H<5B9H5@ 5G98CBH<=G5;FCK=B;BIA69FC:F9G95F7<9FG<5J95HH9ADH98HC5DD@M9A6C8=98
7C;B=H=CB H<9CF=9G HC =BGHFI7H=CB5@ 89G=;B  -CA9 GHI8=9G 9B7CIF5;9 GHI89BHG HC =B7F95G9
D<MG=75@ ACJ9A9BH 5G 5B 5=8 HC 7C;B=H=J9 DFC79GG9G ,C@@=B89 9H 5@  K<=@9 CH<9FG
=B7CFDCF5H9H<9=895C:9A6C8=987C;B=H=CB=BHCG75::C@8=B;HCGIDDCFHH<9=F@95FB=B;,5<=A=
9H 5@   .<=G G75::C@8=B; K<=7< =G DFCJ=898 8IF=B; =BGHFI7H=CB HC <9@D GHI89BHG C::@C58
7C;B=H=CBCBHCH<96C8MCF9LH9FB5@C6>97HG=GF9:9FF98HC5G9A6C8=98G75::C@8=B; A6C8=98
G75::C@8=B;HF5BG:CFAGG9BGCFM9LD9F=9B79=BHC7C;B=H=CB5B8F98I79G7C;B=H=J9@C58H<FCI;<
D<MG=75@ACJ9A9BH4<5B;9H5@ HC<9@D@95FB9FG=BH9;F5H956GHF57H7CB79DHGK=H<H<9@95FB=B;9BJ=FCBA9BH5B857<=9J969HH9F@95FB=B;CIH7CA9G,5<=A=9H5@ A6C8=98 G75::C@8=B; 75B 69 F9DF9G9BH98 =B A5BM :CFAG 5B8 H<9F9 5F9 5@GC
8=::9F9B79G =B H<9 ACHCF B9FJ9G H<5H 75B 69 AC6=@=N98 6M J5F=CIG 9A6C8=98 G75::C@8=B;G CBHFC@@56@9 5B=A5H=CB =G CB9 C: H<9 ACGH 7CAACB 9A6C8=98 G75::C@8=B;G  CBHFC@@56@9
5B=A5H=CBG75::C@8=B;=G5@9GG9A6C8=98:CFAC:@95FB=B;GIDDCFH=BK<=7<@95FB9FG75BIG9
:=B;9FHF57?=B;HC7CBHFC@H<95B=A5H=CBK=H<HCC@GGI7<5GJ=FHI5@G@=89FGHC57<=9J99A6C8=98
7C;B=H=CB  CF9L5AD@9$C<BGCB!@9B69F;9H5@ 89G=;B985B=A5H=CBGH<5H75B7CBHFC@
H<9 GD998 C: 5 FCIB8 65@@ H<FCI;< 5 J=FHI5@ G@=89F HC 5GG=GH GHI89BHG =B @95FB=B; D<MG=7G  #B
588=H=CBGCA9F9G95F7<9FG<5J9DC=BH98CIHH<5H;9GHIF9G75B57H5G57FCGGAC85@DF=A9
"CGH9HH9F  @=65@=
 DFCJ=8=B; @95FB9FG K=H< 588=H=CB5@ A9ACFM 7C89G H<9F96M
GHF9B;H<9B=B; A9ACFM F9DF9G9BH5H=CB =B7F95G=B; F9HF=9J5@ 7I9G $C<BGCB!@9B69F; '9;CK5B,CA5BCK=7N
5B8F98I7=B;H<97C;B=H=J9@C58C:@95FB9FG!C@8=B'958CK .<9F9:CF9GCA9F9G95F7<<5G5@GC5HH9ADH98HC7CA6=B9=BGHFI7H=CB5@;9GHIF9GK=H<
5B=A5H=CBHCAC6=@=G9ACF9ACHCFB9FJ9GK<=@9DFCACH=B;GHI89BHG9A6C8=987C;B=H=CB  CF
9L5AD@9 =B '9F?CIF=G 9H 5@   G GHI8M GHI89BHG F9ACH9@M A5B=DI@5H98 DFC;F5AA98
FC6CHGK=H<H<9<9@DC:=BGHFI7H=CB5@;9GHIF9G75::C@8=B;5B85B=A5H=CBG75::C@8=B; 9G=89G

602

D<MG=75@C6>97HG75::C@8=B;=G5<=;<@M9A6C8=98@95FB=B;GIDDCFH=BK<=7<@95FB9FG75B;5=B5
899D9FH57H=@99LD9F=9B796M=BH9F57H=B;K=H<H957<=B;5=8G>CMGH=7?G:CF79:998657?89J=79G
9H7 5B8C::@C58H<9=F7C;B=H=CBHCH<9D<MG=75@C6>97H  CF9L5AD@94C<5F9H5@ IG98
5<5DH=789J=79H<5H5DD@=98:CF79:998657?HC<9@DGHI89BHG@95FB56CIH7<9A=75@6CB8=B; #B GIAA5FM F9G95F7<9FG <5J9 89G=;B98 F=7< 9A6C8=98 G75::C@8=B; =B 8=::9F9BH
8=G7=D@=B9GHCGIDDCFHGHI89BHG9A6C8=98@95FB=B; "CK9J9F:9KGHI8=9G<5J95HH9ADH98HC
=BJ9GH=;5H99A6C8=98G75::C@8=B;:CFGD5H=5@56=@=HM@95FB=B;@95J=B;579FH5=B;5D=BH<=G5F95 #B 588=H=CB ACGH C: H<9 9L=GH=B; F9G95F7< <5G 699B G=HI5H98 K=H<=B HF58=H=CB5@ H957<=B;
5DDFC57<9G 5B8 BC F9G95F7< <5G 5HH9ADH98 HC 9LD@CF9 H<9 9::97H C: 8=::9F9BH 9A6C8=98
G75::C@8=B;G=BH<97CBH9LHC:;5A965G98@95FB=B;  !!$

1<9B89G=;B=B;9A6C8=98=BH9FJ9BH=CBG=H=GB979GG5FMHC:I@@M7CBG=89FH<9F9@9J5BH:57HCFG
C:H<9@95FB9FGH<9AG9@J9GGI7<5GDF=CF?BCK@98;9CF56=@=HMCB@9M9H5@ *CGH9H5@  <=5B8!@5G9F BCH98H<5HH<9@9J9@C:9LD9FH=G9C:H<9@95FB9F=G5?9M:57HCF=B
89H9FA=B=B;K<5H=B:CFA5H=CB=GF9@9J5BHHCH<9@95FB9F5B8K<5H=B:CFA5H=CBHC:C7IGCB G@95FB9FG =B7F95G9 =B DFC:=7=9B7M H<9=F 9A6C8=98 @95FB=B; GHM@9G A5M 7<5B;9 GCA9K<5H (CJ=79G:CF9L5AD@9C:H9BIG9H<9=F:=B;9FGK<9B9B;5;=B;=B56GHF57HBIA69FH5G?G6IH5G
GHI89BHG9LD9FH=G9=B7F95G9GA9BH5@F9DF9G9BH5H=CBG697CA9ACF956GHF57H5B8G=AD@=:=98
*CIK 9H 5@  GC 9LD9FHG DF9:9F HC GC@J9 DFC6@9AG =B 5B =BH5B;=6@9 K5M  .<9 LD9FH
,9J9FG5@::97H5@GCDC=BHGCIHH<5H=BGHFI7H=CB5@;I=85B79H<5H=G9GG9BH=5@:CFBCJ=79GA5M
B9;5H=J9@M 5::97H ACF9 9LD9F=9B798 @95FB9FG A95B=B; H<5H @95FB9FG K=H< @9GG 8CA5=B
?BCK@98;95F9ACF9@=?9@MHC69B9:=H:FCA5B9BJ=FCBA9BHK=H<5@5F;9BIA69FC:GCIF79GC:
=B:CFA5H=CB5B8=BGHFI7H=CB5@GIDDCFHK<=@9H<9CDDCG=H9=GHFI9:CF@95FB9FGK=H<<=;<9F8CA5=B
?BCK@98;9%5@MI;59H5@  .<9F9=G9AD=F=75@9J=89B79C:H<99::97HGC:=B=H=5@DFC:=7=9B7MCB9A6C8=98@95FB=B; *CIK 9H 5@   :CIB8 H<5H 7<=@8F9B K=H< @CK9F A5H< G?=@@G 69B9:=H98 ACF9 :FCA IG=B;
5I;A9BH=B; =BGHFI7H=CB5@ 5B=A5H=CBG K=H< 5 6C8M 5B5@C;M  7CAD5F98 HC 7<=@8F9B K=H<<=;<9FA5H<G?=@@G -=A=@5F9J=89B79K5G:CIB8=BGH5H=GH=75@8=G7=D@=B9GCB@9M9H5@ 
5B87CADIH5H=CB5@H<=B?=B;'9F?CIF=G<CF=5BCDCI@CGH<5HD5FH=7=D5BHGK=H<@CKDF=CF?BCK@98;9A5M69B9:=HACGH:FCA<=;<9F@9J9@GC:9A6C8=989LD9F=9B79K<=@9D5FH=7=D5BHG
K=H<<=;<DF=CF?BCK@98;9DF9:9F8=G9A6C8=985B856GHF57H@95FB=B;7CBH9BH "CK9J9FH<9F9
5F9GHI8=9GH<5H<5J97CA9HCH<9CDDCG=H97CB7@IG=CB -K5FH9H5@ :CIB8H<5HGHI89BHG
K=H<@CK9F=B=H=5@DFC:=7=9B7M69B9:=H98ACF9:FCAD@5M=B;H<989=7H=7;9GHIF9DC=BH=B;J9FG=CB
C: H<9 ;5A9 K<=@9 GHI89BHG K=H< <=;<9F =B=H=5@ DFC:=7=9B7M 69B9:=H98 ACF9 :FCA H<9 =7CB=7
;9GHIF9 A9H5D<CF=75@ 9B57H=J9 GMA6C@=7 J9FG=CB C: H<9 ;5A9  .<9F9:CF9 GHI89BHGQ
DFC:=7=9B7M5@GCB998HC697CBG=89F98K<9B89G=;B=B;9A6C8=98G75::C@8=B;G   !!"$.<956CJ9GHI8=9G<5J98=G7IGG98H<99::97HGC:9A6C8=98G75::C@8=B;GCB8=;=H5@;5A965G98
GD5H=5@56=@=HM@95FB=B; "CK9J9F:9KGHI8=9G<5J9:C7IG98CBH<99::97HGC:8=::9F9BH9A6C8=98
G75::C@8=B;GCB=ADFCJ=B;GD5H=5@56=@=HM #B588=H=CB@=HH@9CFBCF9G95F7<<5G7CBG=89F98H<9=B:@I9B79 C: @95FB9FG DF=CF DFC:=7=9B7M  CF H<=G F95GCB 5 EI5G=9LD9F=A9BH5@ GHI8M K5G7CB8I7H98HC5GG9GGH<99::97HGC:8=::9F9BH9A6C8=98G75::C@8=B;GCBGHI89BHGGD5H=5@56=@=HM
@95FB=B;5B8HC9L5A=B9H<99::97HGC:@95FB9FGDF=CFDFC:=7=9B7MCB9A6C8=98@95FB=B; .<F99
7CAACBHMD9GC:9A6C8=98G75::C@8=B;GK9F9G9@97H98:CFH<=GGHI8M7CBHFC@@56@95B=A5H=CB=BGHFI7H=CB5@;9GHIF9D@IG5B=A5H=CB!5B8D<MG=75@C6>97H*) .<9:C@@CK=B;HKC
F9G95F7<EI9GH=CBGK9F99LD@CF98
1<5H=GH<99::97HC:8=::9F9BH9A6C8=98G75::C@8=B;GCBGHI89BHG@95FB=B;C:GD5H=5@
56=@=HM C GHI89BHG K=H< J5F=CIG DFC:=7=9B7M D9F:CFA 8=::9F9BH@M K=H< 957< 9A6C8=98
G75::C@8=B;GIDDCFH



603

 !! 
.<9D5FH=7=D5BHGK9F9 :CIFH<;F589FG
:9A5@9G'
- :FCAH<F99
7@5GG9G=BCB9DI6@=7DF=A5FMG7<CC@=B-=7<I5B<=B5 @@GHI89BHG<58BCH699B9LDCG98HC
H<9@95FB=B;7CBH9BH5B8H<9()05B5@MG=GC:H<F997@5GG9GQA5H<:=B5@9L5A;F589GC:@5GH
57589A=7G9A9GH9FG<CK98BCG=;B=:=75BH8=::9F9B79     D S  
*F9J=CIG GHI8=9G <5J9 7CB:=FA98 H<9 7CFF9@5H=CB 69HK99B GD5H=5@ 56=@=HM 5B8 A5H<9A5H=75@
56=@=HM!99F9H5@ GCK97CB7@I898H<5HH<9=B=H=5@GD5H=5@56=@=HMC:H<9GHI89BHG=BH<9
H<F99 7@5GG9G K5G G=A=@5F  .<9 7@5GG9G K9F9 F5B8CA@M 5GG=;B98 HC IG9 H<F99 9A6C8=98
G75::C@8=B;G7CB8=H=CB :9A5@9GA5@9GHC@95FBIG=B;H<97CBHFC@@56@9J=FHI5@H<F99
8=A9BG=CB5@G=AI@5H=CB!7CB8=H=CB:9A5@9G A5@9GHC@95FBIG=B;H<9G=AI@5H=CBD@IG
=BGHFI7H=CB5@ ;9GHIF9G 5B8 *) 7CB8=H=CB  :9A5@9G  A5@9G HC @95FB IG=B; D<MG=75@
C6>97HG  57< 7@5GG K5G H5I;<H 6M H<9 G5A9 HKC =BGHFI7HCFG K=H< 9LH9BG=J9 H957<=B;
9LD9F=9B79  ".<99LD9F=A9BHK5G7CB8I7H985G5B 85M9LD9F=A9BH @@GHI89BHGHCC?5H<F9985M@9GGCB A=BD9F85M56CIHA9BH5@:C@8=B;:FCA85M H9BA=BIH965G=77CB79DH=BGHFI7H=CBK5G
;=J9B=BH<9:=FGH7@5GG5B8H<9BGHI89BHGIG98H<98=;=H5@;5A99BJ=FCBA9BHCBH56@9HHC@95FB
6MH<9AG9@J9GK=H<=BGHFI7HCFGDFCJ=8=B;@95FB=B;GIDDCFHG .<=FHMHKCA=BIH9DF9DCGHH9GHG
K9F958A=B=GH9F98HCGHI89BHGCBH<9:=FGH5B8H<9@5GH85MC:H<99LD9F=A9BHF9GD97H=J9@M 
IFH<9FACF9GHI89BHG7CAD@9H98H<9:@CKEI9GH=CBB5=F95:H9FH<9@5GH@9GGCB  ! .<9 8=;=H5@ 98I75H=CB5@ ;5A9 K5G 75@@98 * "#$)%$ K<=7< :C7IG98 CB GD5H=5@
J=GI5@=N5H=CB 56=@=HM HF5=B=B; 5B8 A5H< ;9CA9HF=7 7CBH9BH @95FB=B; C: H<9 7I69 B9HG :C@8 5B8
IB:C@8  .<9 * "#$)%$ ;5A9 K5G 89G=;B98 HC <9@D GHI89BHG :CFA GD5H=5@
F9DF9G9BH5H=CBG 6I=@8 5B IB89FGH5B8=B; C: H<9 7I69 5B8 B9H 7CB79DHG 5B8 7CBGHFI7H H<9
5GGC7=5H=CB69HK99BH<F995B8HKC8=A9BG=CB5@C6>97HG -D97=:=75@@MGHI89BHGK9F9F9EI=F98
HC 7CAD@9H9 H<F99 A5=B 7<5@@9B;9G DF9G9BH98 =B H<9 ;5A9  !=J9B 5 B9H K=H< H<9 G5A9
D5HH9FBCB957<G=895B8:=B8H<9CDDCG=H9G=89GC:H<9G<5D987I69 !=J9B5B9HA5F?98
K=H<6CHHCA HCDGIF:5795B8:=B8H<9:FCBH5B8HCD 6CHHCAGIF:579C:H<9:CFA987I69!=J9B
57I69:=B8H<97CFF9GDCB8=B;B9H:FCA5:=;IF9 (CB9C:H<9GHI89BHG<58DF=CF9LD9F=9B79
K=H< * "#$)%$ .<9 9A6C8=98 @95FB=B; F9GCIF79G J5FM=B; =B G75::C@8=B; K9F9 5G :C@@CKG  #B H<9 
7CB8=H=CBH<9;5A9DFCJ=898H<9GHI89BHK=H<5H<F998=A9BG=CB5@7I69H<5H75B69FCH5H98
5B8J=9K985GK9@@5G58MB5A=75B=A5H=CBC:H<9HF5BG=H=CB69HK99BH<97I695B8H<9B9H .<9GHI89BH75B7CBHFC@H<9GD998DFC;F9GG5B8GCCBC:H<95B=A5H=CBG99 =;IF9 5 #B
H<9 ! 7CB8=H=CB GHI89BHG K9F9 BCH CB@M 9LDCG98 HC H<9 H<F998=A9BG=CB5@ 7I69G 5B8
8MB5A=75B=A5H=CBG=BH<9;5A96IHK9F95@GC;I=898HCIG9;9GHIF9GH<5HF9DF9G9BH98H<9
HF5BG=H=CBDFC79GG G99 =;IF9 6  #BH<9 *)7CB8=H=CBH<9GHI89BHKCI@8 ;9H H<9 D<MG=75@
A5;B9H=7GEI5F9D=979GG99 =;IF9 7 604*'  7CB8=H=CB 5! 7CB8=H=CB65B8*) 7CB8=H=CB7 "

.<9DF9H9GHG5B8DCGHH9GHG=B7@I898EI9GH=CBG5GG9GG=B;HKC75H9;CF=9GC:J5F=56@9GGD5H=5@
56=@=HMA9BH5@:C@8=B;5B8A9BH5@FCH5H=CB 5B8?BCK@98;9@95FB=B;F9H9BH=CB5B8HF5BG:9F -HI89BHG?BCK@98;9@95FB=B;GH5HIG75B F9:@97HH<9=FIB89FGH5B8=B;C:C6>97H=J9?BCK@98;9=B
H<9DFC79GGC:GD5H=5@56=@=HM7I@H=J5H=CBGC=HK5G5@GCA95GIF98=BH<=GGHI8M 'CF9CJ9FH<9
:@CKEI9GH=CBB5=F97CBH5=B98=H9AG7C@@97H=B;85H5CB GHI89BHGQ@95FB=B;9LD9F=9B79G
 &)"").
.<9GD5H=5@56=@=HMH9GH=B7@I898 A9BH5@:C@8=B;H9GHK<=7<K5GH<956=@=HMH<5HGHI89BHGK9F9
HF5=B98 8=F97H@M =B H<9 ;5A9 5B8 A9BH5@ FCH5H=CB H9GH K<=7< K5G H<9 HF5BG:9F 56=@=HM H<5H
GHI89BHGK9F9BCHHF5=B988=F97H@M  CFH<9A9BH5@:C@8=B;H9GHH<9*5D9F C@8=B;.9GH:FCAH<9
%=HC: 57HCF,9:9F9B798C;B=H=J9.9GHG?GHFCA9H5@  K5GIG98 7CBH5=B=B; =H9AG #B H<=G H9GH GHI89BHG K9F9 ;=J9B 5 GEI5F9 D5D9F H<5H <58 699B :C@898 5B8 DIB7HIF98 =B 5
G9F=9G 5G K9@@ 5G :=J9 :=;IF9G G<CK=B; H<9 DCG=H=CB C: H<9 <C@9G K<9B H<9 D5D9F K5G :I@@M
IB:C@898  .<9M K9F9 5G?98 HC :=B8 H<9 7CFF97H BIA69F :FCA 5ACB; H<9 :=J9 :=;IF9G  57<7CFF97H5BGK9FK5GKCFH<CB9DC=BHK<=@9957<=B7CFF97HCFBC5BGK9FK5GKCFH< C:5
DC=BHCB5G75@9:FCAHC
.<9 '9BH5@ ,CH5H=CB .9GH K5G 58A=B=GH9F98 IG=B; H<9 05B89B69F;  %IG9 '9BH5@
,CH5H=CB.9GH*9H9FG9H5@  5B87CBH5=B98 =H9AG 57<H9GH7CBH5=B98 5H5F;9H:=;IF9
5GK9@@5G:CIFGH=AI@IG :=;IF9G5B8GHI89BHGK9F95G?98HCG9@97HH<9FCH5H98J9FG=CBC:H<9
GH=AI@IG:=;IF9 H<5HA5H7<98 H<9H5F;9H:=;IF9 #:H<97CFF97HGH=AI@IG:=;IF9 K5G:CIB8CB9
DC=BHK5G;=J9B CB5G75@9:FCA HC  $%,""'$$
.<9?BCK@98;9@95FB=B;H9GH7CBH5=B98EI9GH=CBG:FCAH<9F9H9BH=CBH9GH5B8H<9HF5BG:9FH9GH
K=H<957<=B7@I8=B;H9BEI9GH=CBG5HCH5@C: DC=BHG .<9?BCK@98;9H9GH@5GH98 A=BIH9G .<9F9H9BH=CB H9GHA95GIF98 GHI89BHGIB89FGH5B8=B;C:7I69G5B8B9HG  CF9L5AD@9;=J9B
5BIA69F98B9HKCF?GHI89BHGK9F95G?98HC:=B8H<9CDDCG=H9G=89C:H<9BIA69FHKC .<9
HF5BG:9FH9GH9J5@I5H98 GHI89BHG5DD@=75H=CBC:?BCK@98;9=BB9KG=HI5H=CBGGI7<5G5G?=B;
GHI89BHGHC:=B8G<5D9GH<5H75BBCH69:C@898=BHC7I69G )B6CH<H<9 F9H9BH=CB H9GH5B8H<9
HF5BG:9F H9GH 957< 7CFF97H 5BGK9F F979=J98 CB9 DC=BH K<=@9 957< =B7CFF97H CF BC 5BGK9F
F979=J98 N9FC DC=BH  *F9H9GH 5B8 DCGHH9GH EI9GH=CBG K9F9 8=::9F9BH 6IH H<9 7CBH9BH 5B8
8=::=7I@HMK9F9G=A=@5F #B588=H=CBH<9?BCK@98;9H9GHK5GH9GH986MHKC9LD9F=9B798A5H<
H957<9FG5B85A5H<H957<=B;9LD9FHK=H<;CC89LD9FHJ5@=8=HM

605

"%,-&'$
.<9 -<CFH @CK -H5H9 -75@9  - --  89J9@CD98 6M $57?GCB 7?@IB8 5B8 '5FH=B
K5GIG985G5:@CK9LD9F=9B79EI9GH=CBB5=F9HCA95GIF9H<9=BH9BG=HMC::@CK9LD9F=9B79=B
GHI89BHG=B;5A965G989A6C8=98@95FB=B; .<9G75@97CBG=GH98C:B=B9=H9AG7CFF9GDCB8=B;
HCH<9B=B98=A9BG=CB5@7CB79DHI5@=N5H=CBC:7CBJ97H=CB6MG=?GN9BHA=<5@M=   - -- 
K5GF5H98CB5G=LDC=BH&=?9FHG75@9F5B;=B;:FCA GHFCB;@M8=G5;F99HCGHFCB;@M5;F99
K=H<<=;<9FG7CF9G=B8=75H=B;GHFCB;9F:99@=B;GC::@CK .<9F9@=56=@=HMC:- -- K5GR 
$57?GCB9H5@    !$  .<985H5K5G5B5@MN98=B-*--J #'CFD .C5BGK9F,+ 5B5@MG9GC:J5F=5B79
()0GK9F97CB8I7H98HC5GG9GG8=::9F9B79G69HK99B7CB8=H=CBG=BDCGHH9GHG7CF9G G
H<9F9GI@HC:5B()0G<CK98H<5HH<9F9K5G5G=;B=:=75BH8=::9F9B7969HK99B7CB8=H=CBG=B
H<9DF9H9GH:CFA9BH5@:C@8=B;56=@=HM    D S  5IB=J5F=5H9
5B5@MG=GC:7CJ5F=5B79()0K5G9AD@CM98HC577CIBH:CFA9BH5@:C@8=B;DF9H9GHG7CF9G 
@@ J5F=56@9G A9H H<9 5GGIADH=CB C: <CAC;9B9=HM C: J5F=5B79 ;=J9B H<9 F9GI@HG C: &9J9B9QG
H9GHG9L79DHH<9:@CK9LD9F=9B79 .<IG19@7<QG()0K5GIG98:CFH<95B5@MG=GC:H<9:@CK
J5F=56@9 .56@9 DF9G9BHGH<989G7F=DH=J9GH5H=GH=7G:CF5@@J5F=56@9G .C5BGK9F,+ 5A98=5BGD@=HK5G58CDH98HC5GG=;BGHI89BHG=B957<7CB8=H=CBHC
8=GH=B7H@9J9@GC:DF=CFDFC:=7=9B7MC69?.J9FG?M 5G98CBGHI89BHGQDF9:C@8=B;H9GH
G7CF9G H<9M K9F9 8=J=898 =BHC @CK 5B8 <=;<DFC:=7=9B7M ;FCIDG  .56@9  G9HG CIH H<9
89G7F=DH=J9GH5H=GH=7G:CF957<;FCID=BH<F997CB8=H=CBGCB5@@J5F=56@9G .<9F9K9F9G=;B=:=75BH
8=::9F9B79G69HK99BH<9@CK5B8<=;<DF=CFDFC:=7=9B7M;FCIDG=B957<7CB8=H=CB   DB  !    DS  *)  

D
S    .C7CAD5F9957<;FCIDQG=ADFCJ9A9BH:FCAH<9DF9H9GHHCH<9
DCGHH9GHK975@7I@5H98H<9@95FB=B;;5=B6MGI6HF57H=B;H<9DF9H9GHG7CF9:FCAH<9DCGHH9GH
G7CF9  ()0G K9F9 9AD@CM98 HC 9L5A=B9 8=::9F9B79G =B @95FB=B; ;5=BG 69HK99B ;FCIDGIB89F 957< 7CB8=H=CB  @@ J5F=56@9G A9H H<9 5GGIADH=CB C: <CAC;9B9=HM C: J5F=5B79 CF <58
9EI5@;FCIDG=N9G    

1=H<F9GD97HHCH<99::97HC:9A6C8=98G75::C@8=B;CB?BCK@98;9@95FB=B;5:H9F7@5GG9GH<9
H<F997CB8=H=CBG8=8BCHG=;B=:=75BH@M8=::9F=BH<9=FF9H9BH=CBG7CF9G    D  
S 
K<9F95GH<95B5@MG9GF9J95@985G=;B=:=75BH9::97HC:7CB8=H=CBCBHF5BG:9FG7CF9G
        D  S    .<9 :C@@CKID &- 7CAD5F=GCBG G<CK98 H<5H
GHI89BHG =B H<9 *) 7CB8=H=CB D9F:CFA98 69HH9F =B H<9 HF5BG:9F H9GH H<5B GHI89BHG =B H<9 
7CB8=H=CB' - D 5B8GHI89BHG=BH<9!7CB8=H=CB' -  D F9GD97H=J9@M ,9;5F8=B; H<9 9::97H C: 9A6C8=98 G75::C@8=B; CB GD5H=5@ 56=@=H=9G 5:H9F 7@5GG9G H<9
5B5@MG9G 8=8 BCH :=B8 5 G=;B=:=75BH 9::97H C: 7CB8=H=CB CB A9BH5@ :C@8=B; 56=@=HM     D S BCFCBA9BH5@FCH5H=CB56=@=HM    D S  
,9;5F8=B; H<9 9::97H C: 9A6C8=98 G75::C@8=B; CB GHI89BHGQ :@CK 9LD9F=9B79 8IF=B;
@95FB=B;H<95B5@MG9GF9J95@985G=;B=:=75BH9::97HC:7CB8=H=CBCB:@CKG7CF9G       DT   .<9F9GI@HGC::C@@CKID5B5@MG9G=B8=75H98H<5HGHI89BHG=B
H<9*)7CB8=H=CB<585@CK9F:99@=B;C::@CKH<5BH<CG9=BH<97CB8=H=CB' -
D
5B8GHI89BHG=BH<9!7CB8=H=CB'  - D  
.56@9  ('&)+())()(%'&'$&%())()(606

05F=56@9CB8=H=CB(
!CB8=H=CB( *)CB8=H=CB(
*F9
*CGH
*F9
*CGH
*F9
*CGH
'-
'-
'-
,9H9BH=CB                    .F5BG:9F                    
C@8=B;                   ,CH5H=CB                  @CK         
 $
1=H< F9GD97H HC H<9 DF9DCGH H9GH @95FB=B; ;5=B H<9 5B5@MG9G C: ?BCK@98;9 F9H9BH=CB 85H5
G<CK98H<5HH<9F9K5GBCG=;B=:=75BH@M;F95H9F;5=B69HK99BH<9@CKDFC:=7=9B7M;FCID5B8H<9<=;<DFC:=7=9B7M;FCID  
D  S !    D S *)  
D S  ,9;5F8=B;H<9?BCK@98;9
HF5BG:9FH<9F9GI@HGGH=@@:5=@98HCF957<GH5H=GH=75@G=;B=:=75B79  
D  
S 
!    D S *)    D  
S  
1=H<F9;5F8HCH<9A9BH5@:C@8=B;56=@=HMH<95B5@MG9GG<CK98H<5HH<9@CKDFC:=7=9B7M
;FCID57<=9J985G=;B=:=75BH@M;F95H9F;5=BH<5BH<9<=;<DFC:=7=9B7M;FCID=BH<97CB8=H=CB  DS   5GK9@@5G=BH<9*)7CB8=H=CB    DS   #BH<9!7CB8=H=CB5@H<CI;<H<9@CKDFC:=7=9B7M;FCID<585BIA9F=75@@M;F95H9F;5=BH<5BH<9<=;<DFC:=7=9B7M;FCIDH<9F9GI@HG:5=@98HC:=B85GH5H=GH=75@G=;B=:=75B79    D S  ,9;5F8=B;H<9A9BH5@FCH5H=CB56=@=HMF9GI@HGG<CK98
H<5H H<9F9 K5G BC G=;B=:=75BH ;F95H9F ;5=B 69HK99B H<9 @CK DFC:=7=9B7M ;FCID 5B8 H<9 <=;<
DFC:=7=9B7M;FCID   D  S !   D S 
*)   D  S   
.56@9  ('&)+())()(%'"%,$&'%$.'%*&($)'%$)%$(
05F=56@9

*FC:=7=9B7M (  (    !(  (    *)(  (   
*F9
*CGH
*F9
*CGH
*F9
*CGH
'-
'-
'-
,9H9BH=CB &CK                 "=;<                   .F5BG:9F &CK                     "=;<                   
C@8=B;
&CK                "=;<            
 
,CH5H=CB &CK                      "=;<                        !   !!$
.<=GGHI8M5GG9GG98GHI89BHGGD5H=5@56=@=HM?BCK@98;9@95FB=B;5B8:@CK9LD9F=9B795:H9F
;5A965G98=BGHFI7H=CB=BH<F997CB8=H=CBG #BH9FAGC:GD5H=5@56=@=HM@95FB=B;H<9A5=BF9GI@HGG<CK98 H<5H K<=@9 GHI89BHG GD5H=5@ 56=@=HM =ADFCJ98 G=;B=:=75BH@M =B 5@@ H<F99 9A6C8=98
G75::C@8=B;7CB8=H=CBGH<9F9K9F9BCG=;B=:=75BH8=::9F9B79G=BG7CF9GCBA9BH5@:C@8=B;5B8
A9BH5@ FCH5H=CB =B H<9 H<F99 7CB8=H=CBG  .<=G A5M J9F=:M H<9 =ADCFH5B79 C: =B7CFDCF5H=B;
9A6C8=98G75::C@8=B;=BH957<=B;GD5H=5@56=@=HM%KCB9H5@ ,565HHI9H5@ IFH9
9H5@ F9;5F8@9GGC:H<9:CFAC:G75::C@8=B; 607

CF ;9CA9HFM ?BCK@98;9 @95FB=B; H<9 A5=B F9GI@HG G<CK98 H<5H H<9F9 K5G BC
G=;B=:=75BH 8=::9F9B79 =B ?BCK@98;9 F9H9BH=CB G7CF9G 69HK99B H<9  7CB8=H=CB H<9 !
7CB8=H=CB5B8H<9*)7CB8=H=CBK<=@9GHI89BHG=BH<9*)7CB8=H=CBD9F:CFA98G=;B=:=75BH@M
69HH9F=BH<9HF5BG:9F5GG9GGA9BHC:;9CA9HFM?BCK@98;9 .<9F9GI@HGG<CK98H<5H5@@H<F99
HMD9GC:9A6C8=98G75::C@8=B;K9F956@9HCGIDDCFHGHI89BHG@95FB=B;C:;9CA9HFMK<=@95@GC
89ACBGHF5H=B;H<958J5BH5;9GC:C6>97H9A6C8=98G75::C@8=B;=BH9FAGC:?BCK@98;9HF5BG:9F .<=G=G7CBG=GH9BHK=H<DF9J=CIGGHI8=9GGI;;9GH=B;H<5H9B;5;=B;=B<=;<9F@9J9@GC:9A6C8=98
@95FB=B;57H=J5H9GG9BGCF=ACHCF7C89GH<5H9B<5B79A9ACFMHF579G5B8<9@D@95FB9FG@95FB7CBH9BH :5GH9F 5B8 =B ;F95H9F 89DH< $C<BGCB!@9B69F; 9H 5@   G D<MG=75@ C6>97H
G75::C@8=B; =G 5 F9@5H=J9@M <=;<9A6C8=A9BH G75::C@8=B; GHI89BHG KCI@8 ;5=B ACF9 8=J9FG9
A9ACFM 7I9G K<9B IG=B; =H K<=7< A5M <9@D H<9A HC 9LHF57H ?BCK@98;9 ACF9 EI=7?@M CB
HF5BG:9FH9GHG #BH9FAGC:GHI89BHG:@CK9LD9F=9B79H<9A5=BF9GI@HGG<CK98H<5HH<9G7CF9GC::@CK
9LD9F=9B79K9F9G=;B=:=75BH@M@CK9F=BH<9*);FCIDH<5B=BH<9CH<9FHKC;FCIDG .<9F95GCB
A5M69H<5HGHI89BHG=BH<9*);FCID<58HCGK=H7<:F9EI9BH@M69HK99B9@97HFCB=789J=79G5B8
A5;B9H=7 GEI5F9 D=979G 8IF=B; H<9 @95FB=B; DFC79GG 5B8 H<=G G<=:H =B 5HH9BH=CB A5M <5J9
5::97H98H<9GHI89BHG:@CK9LD9F=9B79   !"! #!!$ 

.C=BJ9GH=;5H9K<9H<9FGHI89BHGK=H<8=::9F9BHDFC:=7=9B7MKCI@8D9F:CFA8=::9F9BH@MK=H<957<
9A6C8=98G75::C@8=B;GIDDCFHH<=GGHI8M8=J=898GHI89BHG=BHC<=;<5B8@CKDFC:=7=9B7M;FCIDG
65G98CBH<9DF9:C@8=B;H9GHG7CF9G B5@MG=GC:H<9@95FB=B;D9F:CFA5B79C:H<9HKC;FCIDGF9J95@98H<5HH<9F9K5GBCG=;B=:=75BH8=::9F9B79=BH<9@95FB=B;;5=B=BH9FAGC:?BCK@98;9
F9H9BH=CB?BCK@98;9HF5BG:9F5B8A9BH5@FCH5H=CB56=@=HM69HK99BH<9HKC;FCIDGF9;5F8@9GG
C: H<9 9A6C8=98 G75::C@8=B; IG98  G :CF H<9 A9BH5@ :C@8=B; 56=@=HM H<9F9 K5G 5 G=;B=:=75BH
8=::9F9B79=BH<9@95FB=B;;5=B69HK99BGHI89BHG=BH<9<=;<DFC:=7=9B7M;FCID5B8GHI89BHG=B
H<9@CKDFC:=7=9B7M;FCID=B6CH<H<97CB8=H=CB5B8H<9*)7CB8=H=CBK=H<6CH<G<CK=B;
;F95H9FDFC;F9GG:CFGHI89BHG=BH<9@CKDFC:=7=9B7M;FCID .<9F95GCBA5M69H<5HGHI89BHG
=BH<9;FCID5B8H<9*);FCIDK9F956@9HC;9H7@95F:998657?:FCAH<9G75::C@8=B;8IF=B;
H<9@95FB=B;DFC79GGK<=@9GHI89BHG=BH<9!;FCIDK9F9IB56@9HC89H9FA=B9K<9H<9FH<9=F
ACJ9A9BHGK9F97CBG=GH9BHK=H<H<9F95@:C@8=B;DFC79GG )BH<9CH<9F<5B8=A5;=B=B;H<9
DFC79GGC::C@8=B;H<97I69CIHC:H<=B5=F5@GCH9GHGGHI89BHGGD5H=5@H<=B?=B;G?=@@GK<=7<A5M
BCH695DD@=756@9HCGHI89BHGK=H<@CK=B=H=5@GD5H=5@56=@=HM   
)IFGHI8M=BJ9GH=;5H98H<99::97HGC:8=::9F9BH9A6C8=98G75::C@8=B;GCBGHI89BHG@95FB=B;C:
GD5H=5@56=@=HM .<9F9GI@HGG<CK98H<5HGHI89BHG=BH<9;FCIDH<9!;FCID5B8H<9*)
;FCID K9F9 56@9 HC @95FB ;9CA9HFM 5B8 9B<5B79 H<9=F GD5H=5@ 56=@=H=9G =B8=75H=B; H<9
9::97H=J9B9GG C: =B7CFDCF5H=B; 9A6C8=98 G75::C@8=B; =B 9B<5B7=B; GD5H=5@ 56=@=H=9G  CF H<9
HF5BG:9F5GG9GGA9BHC:;9CA9HF=7?BCK@98;9H<9*);FCIDG7CF98G=;B=:=75BH@M<=;<9FH<5B
H<9CH<9FHKC;FCIDG5@H<CI;<H<=G;FCID5@GCG7CF98G=;B=:=75BH@M@CK9FH<5BH<9CH<9FHKC
;FCIDG:CF:@CK9LD9F=9B79 .<=GGI;;9GH98H<5HK9G<CI@8BCHCB@M9LD@CF9H<97C;B=H=J9J5@I9
C: C6>97H G75::C@8=B; 6IH 5@GC D5M 5HH9BH=CB HC =HG B9;5H=J9 =AD57H CB :@CK 9LD9F=9B79  #B
588=H=CB K9 :IFH<9F 5B5@MN98 H<9 DCGG=6@9 9::97HG C: GHI89BHG  DFC:=7=9B7M =B 8=::9F9BH
9A6C8=98G75::C@8=B;7CB8=H=CBG5B8G<CK98H<5HGHI89BHGK=H<@CK9FDFC:=7=9B7MK9F9ACF9
GI=H56@9:CFG75::C@8=B;5B8*)G75::C@8=B;K=H<7@95F9F:998657? 19<CD9H<5HCIFKCF?
K=@@ <9@D 98I75HCFG HC 69HH9F =B=H=5H9 H<9 H957<=B; C: GD5H=5@ 56=@=HM 5B8 5DD@M 5DDFCDF=5H9
9A6C8=98G75::C@8=B;GHC9B<5B79GHI89BHGGD5H=5@56=@=HM "


608

19KCI@8@=?9HC57?BCK@98;9,ICB5B"I4<=MCB;&=3I&=2=5CE=19=3IL=19="I5
.=5B-CB;>=5B;<9B3CB;;IC15B;35B>IB5C$=B; 9B;3=B;MCB;$=B$=B;4<5C
"5=@=B;.5B;$=B;&=:CFH<9=F<9@DK=H<9LD9F=A9BH7CB8I7H=B;5B885H57C@@97H=CB 
CIB85H=CB%9M*FC>97HC:.<99=>=B; H< =J9395F*@5B=B
:CF8I75H=CB-7=9B79
O,9G95F7<CBH<9I@H=J5H=CBC:&95FB=B;6=@=HM65G98CB&95FB=B;-7=9B79G5B8!5A9
65G98&95FB=B;P(C  






>1CD?@?E<?E )  )81B@<5C #   125B     > 5F1<E1D9?> ?6 =E<D9=?41< 9>D5B13D9?>C G9D8
D538>?<?7I G89<5 <51B>9>7 C395>35 3?>35@DC  0)2)1( .30-!+ .& $3#!2).-!+ %#(-.+.'7  M  
EBD5 1B4?>I " EDD?> *1I<?B   *89>; 4=@B?F9>7=1D85=1D93C<51B>9>7
D8B?E785=2?4954C@1D91<DB19>9>7 .'-)2)4%%1%!0#(0)-#)/+%1!-$,/+)#!2).-1  
819 "1E * &1>0 
 E>7BI31DNC5B9?EC71=56?B3?>F5I9>7C@1D91<9>6?B=1D9?>D?
D85F9CE1<<I9=@19B54 3+2),.$!+ %#(-.+.')%1!-$-2%0!#2).-  
89# *  <1C5B(   &B?2<5=C?<F9>7129<9DI >( )D5B>25B74 3,!-!")+)2)%1-
)-&.0,!2).-/0.#%11)-'!//0.!#(@@  M   )1>B1>39C3?B55=1> ?><5I' D;9>C?>( ! $7EI5> $5<C?>   #1>D1B1I("5F5B179>71E7=5>D54
B51<9DID?D5138@B?2129<9DI1>4C1=@<9>7 .,/32%01$3#!2).-  
C9;CJ5>D=981<I9#    +.5 $5G/?B;1B@5B(?G E66I )?B2I) ?G5  >9>F5CD971D9?>?6D85B?<5?6C@1D91<129<9DI9>B5@B5C5>D9>71>4
C?<F9>7 G?B4 @B?2<5=C 1=?>7 5>79>55B9>7 CDE45>DC  .30-!+ .& -')-%%0)-' $3#!2).-    
;CDB?=(  B5>38 - 1B=1>  5B=5>   !-3!+&.0*)2.&&!#2.00%&%0%-#%$#.'-)2)4%2%121 &B9>35D?>$ 4E31D9?>1<*5CD9>7)5BF935C 
>F5<?@5 >F5<?@5# >F5<?@5) >F5<?@50 >F5<?@5# >F5<?@5" 5D1< +C9>71=?29<5F9BDE1<B51<9DI1>43?=@ED5B71=5D?9=@B?F5F9CE?C@1D91<C5<65669313I9>=944<5
C38??<CDE45>DC .,/32%01$3#!2).-  
1441 &5<<57B9>9# ,9F1>5D 1<<5785B 0 
 6653DC?64979D1<71=5C?>CDE45>D=?D9F1D9?>9>=1D85=1D93C=5D11>1<IC9C9>! .30-!+.&.,/32%011)12%$%!0-)-'    
1B4>5B    B1=5C?6=9>4D85D85?BI?6=E<D9@<59>D5<<975>35C .30-!+.&.+)#7-!+71)1!-$
!-!'%,%-2   
55B     'E9>>  #   1><5I   #    (5<1D9?>C 25DG55> C@1D91< C;9<<C 1>4 =1D8
@5B6?B=1>35 9> 5<5=5>D1BI C38??< 389<4B5>  <?>79DE49>1< 9>F5CD971D9?>  %4%+./,%-2!+17#(.+.'7    
?<49>#514?G )    -945>9>7 D85 <5>C -81D D85 =1>E1< =?41<9DI B5F51<C 12?ED <51B>9>7
<1>7E1751>43?7>9D9?> &89<?C?@8931<*B1>C13D9?>C?6D85(?I1<)?395DI).+.')#!+#)%-#%1  
5. "9* *EB5<% !E1>7/ 5'  *859=@13D?6CD5=54E31D9?>?>=1D85=1D931<
45F5<?@=5>D9>389<4B5>1754I51BC -2%0-!2).-!+.30-!+.&$3#!2).-!+%1%!0#(  
?47;9CC 9<<971>!  *?<=95 ! *8?=1C# )  1BB1> !  )@1D91<3?7>9D9?>
1>4C395>3513895F5=5>DD853?>DB92ED9?>?69>DB9>C931>45HDB9>C93C@1D91<C;9<<C6B?=D? I51BC 0)2)1(.30-!+.&$3#!2).-!+17#(.+.'7  
?CD5DD5B      <921<9 #  -    ,9C92<5 5=2?49=5>D 5CDEB5C 1C C9=E<1D54 13D9?> 17#(.-.,)#3++%2)-%4)%5 M  ?E * 1>7/ ) *1>7 *  5C97>9>71>1<D5B>1D5B51<9DI2?1B471=5G9D81E7=5>D54
B51<9DI 1>4 =E<D949=5>C9?>1< C3166?<49>7 6?B @B?=?D9>7 C@1D91< 1>4 <?7931< 129<9DI  -2%0!#2)4%
%!0-)-'-4)0.-,%-21  
?1>>?E# ?1>>?E  *538>?<?7I5>81>3545=2?4954<51B>9>75C97>9>71>45F1<E1D9>7
1>5G3<1CCB??=5H@5B95>35 $3#!2).-!+ %#(-.+.'7.#)%27  M  "?>71>4C8?BD=51CEB5C?66<?G*853?>CDBE3D
13;C?>)  #1BD9>  ;<E>4(  
F1<949DI ?6 D85 ))  )  1>4 >5G 2B956 3?E>D5B@1BDC  .30-!+ .& /.02 !-$ 6%0#)1%
17#(.+.'7  ?8>C?><5>25B7 #    1BD?<?=51    !1<9>1     &<1D6?B= 9C >?D 45CD9>I =2?4954
<51B>9>7 56653DC 3?=@1B9>7  45C;D?@ D?  F9BDE1< B51<9DI )*# 5H@5B95>35C  .30-!+ .&
.,/32%011)12%$%!0-)-'     

609

?8>C?><5>25B7#  #57?G1>(?=1>?G93J  =2?4954C395>351>4=9H54B51<9DI
?G 75CDEB5 1>4 =?D9?> 31@DEB5 16653D @8IC93C 54E31D9?>  .'-)2)4% %1%!0#( 0)-#)/+%1 !-$,/+)#!2).-1    ?8>C?><5>25B7 #    #57?G1>(?=1>?G93J   9B38695<4      )1F9?(1=?C    6653DC?65=2?4954<51B>9>71>44979D1<@<1D6?B=?>D85B5D5>D9?>?6@8IC93C3?>D5>D5>DB9@5D1<
6?B35 0.-2)%01)-17#(.+.'7      !1<IE71) IB5C& 81>4<5B& )G5<<5B  *855H@5BD9C5B5F5BC1<56653D $3#!2).-!+17#(.+.')12    !G?> !  5?> #  08?E   !9= !   BEC8 *      =2?4954 <51B>9>7 6?B 3?=@ED1D9?>1<D89>;9>79>51B<I@B9=1BI54E31D9?> .30-!+.&%1%!0#(.- %#(-.+.'7)-$3#!2).-18514?6
@B9>D18514?6@B9>D  "9>      85>   #    5F5<?@9>7 C@1D91< F9CE1<9J1D9?> 1>4 =5>D1< B?D1D9?> G9D8 1 4979D1<
@EJJ<571=51D@B9=1BIC38??<<5F5< .,/32%01)-3,!-%(!4).0 1@B   "9>>#  &5D5BC5>    =5B75>351>4381B13D5B9J1D9?>?6C5H49665B5>35C9>C@1D91<
129<9DI1=5D11>1<IC9C ()+$%4%+./,%-2  M  "?J1>?)  1B4 # *F5BC;I  &EDD9>713D9?>9>@5BC@53D9F5 .'-)2).-  M #5B;?EB9C    8?B91>?@?E<?C !    &B?7B1==9>7 5=2?4954 9>D5B13D9?>C G9D8 1 B5=?D5<I
3?>DB?<<5454E31D9?>1<B?2?D  0!-1!#2).-1.-.,/32)-'$3#!2).-   #?BC5<<1    !B1ECC (  #    *85 B?<5 ?6 75CDEB5C 9> C@1D91< G?B;9>7 =5=?BI 1>4 C@5538 ,%0)#!-.30-!+.&17#(.+.'7   M  $1D81>#  )385>3;!  ,9>C?>81<5B( #93815<9C  -1<;9>7D?>  =2?4954
75?=5DB93B51C?>9>74I>1=9375CDEB5C4EB9>79>DE9D9?>9>C978D1>4@B??6 .30-!+.&$3#!2).-!+17#(.+.'7  &1>4?5BB1& 5B>K>45JP<F1B5J ECD?&1BB1 7<5C91C?B45B1&  6653DC?6EC9>7
71=521C54<51B>9>7D?9=@B?F5D8513145=93@5B6?B=1>351>4=?D9F1D9?>9>5>79>55B9>7CDE495C 
.30-!+.&$3#!2).-!+.,/32)-'%1%!0#(     &5D5BC# "15>7 "1D81=!  13;C?># 019I?E>1( (9381B4C?>   (54B1G>
,1>45>25B7  !EC5 #5>D1< (?D1D9?>C *5CD 9665B5>D ,5BC9?>C 1>4 13D?BC D81D 16653D
&5B6?B=1>35 0!)-!-$.'-)2).-    &?CD" ) ,1>?7* &11C 0G11>(   6653DC?6C9=E<D1>5?EC<I?2C5BF9>71>4
=1;9>775CDEB5CG89<5CDE4I9>77B1==1B1>9=1D9?>C?>3?7>9D9F5<?141>4<51B>9>7 .,/32%01)-3,!-%(!4).0  M  &?EG- F1>?7* 0G11>( &11C  E7=5>D9>79>CDBE3D9?>1<1>9=1D9?>CG9D812?4I
1>1<?7ID?85<@389<4B5><51B>12?ED@8IC931<CICD5=C 0.-2)%01)-17#(.+.'7   &?EG-9=*  " F1>?7* &11C  >5=2544541>45=2?49543?7>9D9?>B5F95G?6
9>CDBE3D9?>1<=1>9@E<1D9F5C $3#!2).-!+17#(.+.'7%4)%5    
(121DDE& 521B>?D+ ?I5;$   H@<?B9>7D859=@13D?69>D5B13D9F5=?F5=5>D21C54
1>1D?=I <51B>9>7 9> B51< 3<1CCB??= C5DD9>7 1=?>7 ;9>5C9?<?7I CDE45>DC  -!2.,)#!+ #)%-#%1$3#!2).-     
(189=9) )8ED5,  E<G945B 19>2B9475! !E21( /1>7. )=9D8 1;5B( ) #5<<? )  !    *9=9>7 ?6 <51B>9>7 CE@@?BDC 9> 54E31D9?>1< 71=5C 31> 9=@13D CDE45>DCO
?ED3?=5C .,/32%01$3#!2).-   )8?E<46B1=5C?6B565B5>35255>13D549>1CDB?>?=I(?<<9>45 531=@$ 5B>91EH 
9>CDBE3D9?>&8IC931<(5F95G (71)#1$3#!2).-%1%!0#(  )G1BD#  !?B>;1C5=) ?<?>3?CD1$ 138971> ,9D1<5 #  B?=2CDB13D
D??>3B5D5F945>356?B45C97>9>7<51B>9>7@<1D6?B=CD81D141@DD?EC5BC@B?69395>395C .'#)
+DD1<       )559>7 D85 297 @93DEB5 #1@ EC5 1>4 D85 45F5<?@=5>D ?6 C@1D91< 3?7>9D9?> %4%+./,%-2!+#)%-#%    /??>) / #1>> "  H@<?B9>7D85C@1D91<129<9DI?6E>45B7B14E1D5CDE45>DCCC?391D9?>
G9D875>45BCD5==1:?BC1>4 796D54@B?7B1==5=25BC89@  )&2%$()+$3!02%0+7 
M

081>7 / *E3;5B#  )D97<5B -  -1D389>7181>4C?>13D9F9DI9=@B?F5CCDE45>DCO
E>45BCD1>49>7?6B1>4?=>5CC .,/32%01$3#!2).-  
0?81B ( "5FI) *  B?=655<9>76?B35CD?E>45BCD1>49>76?B35C*859=@13D?62?49<I
5>7175=5>D?><51B>9>79>C395>35 .30-!+.&%1%!0#()-#)%-#% %!#()-' 



610

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring the Impact of Designing a Robot
as a Pet with Interdependence Theory
on Long-Term Relationships
and Learning Performance
Vando Gusti Al HAKIMab, Su-Hang YANGc, Jen-Hang WANGd*, Yu-Chen CHANGa,
Hung-Hsuan LINa & Gwo-Dong CHENaa
Department of Computer Science and Information Engineering, National Central Universityb
Department of Electrical Engineering Education, Yogyakarta State University, Indonesiac
Department of Hospitality Management, Chien Hsin University of Science and TechnologydResearch Center for Science and Technology for Learning, National Central University
acd
Taiwan*harry@cl.ncu.edu.tw
Abstract: Educational robots have shown promise in enhancing learning performance;
but, many of these robots function solely as companions or tutors and rely on novelty
to attract attention, making it challenging to maintain long-term relationships. Although
pet robots have been utilized to create relationships with users, their potential in
education remains underexplored. This study addresses this issue by designing a robot
as a pet, applying the interdependence theory to establish lasting relationships andimprove learning performance. The interdependence theory suggests that relationships
between individuals can be strengthened through mutual dependence and thefulfillment of each other's needs. In this study, learners were prompted to engage incontinuous care for their pet robots, receiving emotional reinforcement from themseamlessly across both virtual and tangible forms enabled by the ChatGPT API. To
culminate their learning process, learners presented their ultimate learning outcomesalongside their pet robots, enacting situational dramas for their classmates. To evaluate
the effectiveness of this approach, a quasi-experiment was conducted with 100
undergraduate learners enrolled in a Japanese for Hospitality and Tourism course in
Taiwan. Results indicate that robots designed as pets, leveraging interdependence
theory, significantly yield positive effects on learners, encompassing improved learning
outcomes, extended interaction rates, and heightened satisfaction with their study
journey compared to conventional robots. The study concludes by discussing research
limitations and offering suggestions for further enhancements to this approach.
Keywords: Educational Robots, Pet Robots, Long-Term Relationships,
Interdependence Theory, Situational Learning, Human-Robot Interaction

1. Introduction
In recent years, the use of robots to support learning has gained significant attention amongscholars (Belpaemae et al., 2018; Mubin et al., 2013). Previous studies have demonstrated
the potential of robots to engage learners and stimulate their interest in interacting withlearning materials (Al Hakim et al., 2021; 2022a; Lin et al., 2022). However, since learningoften requires several weeks or months to complete, it is essential for robots to develop
enduring relationships with learners to maintain their engagement for an extended period.
Establishing long-term relationships with robots has been identified as a significant
factor in the realm of robotics (Westlund et al., 2018; Leite et al., 2013). This is because
learning and behavioral changes often require continuous engagement and repetition, rather
than just a single interaction. However, as the frequency of interactions increases, the initialnovelty effect may diminish, potentially affecting the efficiency of learning.

611

One promising approach to cultivating long-term relationships is drawing inspirationfrom the bond between pets and their owners (Díaz-Boladeras, 2022). Research has shownthat treating robots as pets can elicit positive emotional engagement and attachment(Westlund et al., 2018). The relationship between pet owners and their pets is welldocumented, with pets providing emotional support and interacting with them fulfilling social
needs, promoting happiness, and fostering healthier personality traits (McConnell et al., 2011).
A well-known illustration is Tamagotchi, a virtual pet that sends reminders to users for theircare (Lawton, 2017). Despite its simple design, users often interact with Tamagotchi
extensively, creating a connection where they care for the pet and receive emotional support.
In education, interdependence was studied by Johnson & Johnson (2009), highlighting its
importance for successful ongoing collaboration.
Building upon the groundwork laid by Al Hakim et al. (2023) regarding pet-like robot
models, our study bridges gaps by investigating their impact on education. Through a novel
learning design rooted in interdependency, seamless integration of technology for ubiquitous
learning, and a comparative evaluation, we aim to enhance educational robot design.
Specifically, we explore learners' potential to establish lasting relationships with pet-likeeducational robots. By imbuing robots with needs like feeding, we stimulate interaction rates.
Furthermore, the robot's presence may provide emotional support, enhancing satisfaction withthe experience and covered material. This study is guided by the research questions:(1) Does a pet robot established with an interdependence mechanism enhance learning
outcomes more effectively than a conventional robot?
(2) Does a pet robot established with an interdependence mechanism increase interaction
rates more effectively than a conventional robot?
(3) How does the use of a pet robot with an interdependence mechanism, in comparison to aconventional robot, impact learners' acceptance of learning?

2. Related Works
2.1 Robots in Education
Robots have emerged as a prominent area of research in education, witnessing significant
growth and diverse applications. They have been explored as tutors or peer learners, yielding
positive outcomes in cognitive and emotional outcomes comparable to human tutoring(Belpaeme et al., 2018). They can also serve various functions and be applied in different
areas of education, including language, science or technology education (Mubin et al., 2013).
Despite the numerous studies on educational robots, there is still a lack of study concerning
the establishment of long-term relationships with these devices. To address this, it is crucial
for robots to have specific roles and interaction modes that can sustain user engagement.
Previous research has mainly focused on the role of educational robots as a tutor or friends
with personal conversation (Ligthart et al., 2023), or storyteller companions (Calvo-Barajas &Castellano, 2022), which relies solely on social and affective behaviors.

2.2 Pet Robots and Long-Term Relationships
The introduction of Bandai's "Tamagotchi" in 1996 marked a significant milestone in theacceptance of virtual pets, owing to their portability and sociability (Lawton, 2017). As a result,many studies related to artificial pet have become popular topics of study. Bylieva et al. (2019)conducted a literature review that traced the development of artificial pets from the 1990s to
the 21st century. They identified social interaction as a crucial factor in sustaining relationships
with artificial pets. However, the exact definition of a "long-term relationship" remains a topic
of debate within the domain of human-robot interaction (Díaz-Boladeras, 2022). While there
is no consensus on the specific duration or number of interactions required, a long-term
relationship is generally believed to emerge when the user becomes accustomed to the robot
and the novelty effect no longer influences their perception of it (Leite et al., 2013). Díaz-

612

Boladeras (2022) suggests measuring the long-term relationship based on changes inbehavior, such as the rates of use, rather than relying solely on the time span.
Furthermore, the relationship mechanism between a user and a robot is a vital aspect
to consider. Frude and Jandrić (2015) introduced the concept of "animism," which suggeststhat humans tend to attribute life and consciousness to non-living objects, thereby fostering
emotional bonds. This concept highlights the potential for users to form attachments withrobots. Similarly, Heljakka et al. (2021) discovered that the relationship between users androbot dogs can deepen over time as they spend more time together. However, the precise
mechanisms underlying the establishment and maintenance of such relationships have yet
to be fully explored in these studies.

2.3 Interdependency Theory
The concept of interdependence was first introduced by John Thibaut and Harold Kelley in
1959, encapsulating elements such as dependence, power, norms, rules, cooperation, andcoordination. It explains the extent to which two or more individuals rely on each other toachieve their goals within a specific context (Van Lange & Rusbult, 2014). Furthermore,
Johnson and Johnson (2009) found that positive interdependence in cooperative learning canenhance individual responsibility and achievement and foster long-term group cohesion.
When it comes to pet robots, the concept of interdependence is crucial to consider. If
an owner is tasked with caring for a pet robot and the robot reciprocates by providing
companionship, the resultant interdependence can foster a long-term relationship. Thisimbues the owner with a sense of responsibility and enhances their willingness to learn withthe robot. As such, this could increase the overall interaction rate between the learner and therobot, leading to a stronger relationship and potentially improved learning outcomes.

2.4 Situational Learning
Situational learning is an active learning approach that emphasizes the application of realworld experiences and practical knowledge. It was first introduced by Brown et al. (1989) anddraws on the principles of apprenticeship, imitation, and practical thinking, as identified by
Lave and Wenger (1991). In situational learning, learners are immersed in hands-on activities
and real-life scenarios, providing them with opportunities to solve problems and apply
theoretical concepts they have learned.
One of the key features of situational learning is its versatility, with researchers
leveraging drama and role-playing to create scenarios that allow learners to showcase theirlearning outcomes. By embedding the learning material in dialogues within a structured script,
learners are guided through interactive exchanges and scenarios that facilitate effective
learning. For example, Erbay and Doğru (2010) used drama to have learners perform together,
fostering both subject knowledge acquisition and social interaction. Similarly, Al Hakim et al.
(2022a) proposed an interactive situated learning approach using a real-time feedback
mechanism to guide and evaluate learners’ knowledge application through interactions withrobots, virtual objects, and virtual characters based on textbook context and content toenhance learning performances. By providing opportunities for learners to engage in role-play
activities and real-world problem-solving, situational drama learning helps learners develop a
deeper understanding of course material and cultivate particular skills.

2.5 Summary
The literature review highlights that robots have a positive impact on education, but the use of
pet robots in education and research on establishing long-term relationships with them islimited. The potential benefits of utilizing pet robots in education are apparent, and this study
aims to use interdependence to create a lasting relationship between the robot and the evident.
Such a relationship would enhance the user's learning experience and outcomes. Finally,learners can apply their learning results by participating in situational drama role-playing
scenarios with their pet robots, showcasing their ability to use the skills and gained knowledge.

613

3. System Design and Implementation
3.1 Learning Design
The aim of the devised learning design was not to supplant teachers' instructional approaches,but rather to employ the learning system as a means to amplify the efficacy of their teaching.
Our learning design focused on fostering interdependence between learners and robots,nurturing lasting relationships and enhancing learning performance. The design integrated
collaborative cooperation and mutual reliance on incentives (Johnson & Johnson, 2009; Van
Lange & Rusbult, 2014), aligning learners and pet robots toward the shared objective of
maintaining the pet's well-being through collaborative learning tasks that yielded food rewards.
Learners, assuming roles as caregivers and trainers, engaged in interactive tasks simulating
pet care duties, directly influencing the pet robot's well-being and its performance in situational
dramas. In this dynamic, pet robots served as companions and performance partners,
providing emotional support, companionship, and feedback. Learning materials,
encompassing foundational knowledge and a dialogue-based script, expertly curated byteachers, were seamlessly integrated into a comprehensive learning script. This script served
as both presentation material and guidance, to be learned by both pet robots and learners fortheir ultimate situational drama presentations.

3.2 Seamless Technical Design
To overcome the limitations of physical pet robots, virtual pets were introduced as acomplementary solution, seamlessly integrated with the physical pet robots. They werepowered by the ChatGPT API, which is an interface that allows developers to integrate the
ChatGPT language model into their applications, providing advanced language capabilities,
contextual understanding, and dynamic interactions. The virtual pets enabled ubiquitouscommunication and interaction, providing flexibility in terms of availability, capabilities, andclass usage time. In the execution of the ChatGPT API program utilizing Python 3.11.3 andthe GPT-3.5-turbo model, teachers effortlessly uploaded scripts in PDF, Word, or XLS formats
to a designated server PC directory, establishing a custom knowledge base for the AI model.

Figure 1. Communication mechanism linking the learner and the robot.
The ChatGPT API not only produced text responses from the custom knowledge base
but also played a pivotal role in instructing the virtual and physical pet robots to express
emotions through facial cues and body movements (as depicted in Fig. 1). For instance, whena learner posed a question that elicited the response "I feel sad now," the API could include
instructions like "Exhibit a [sad] facial expression while [nodding] in disagreement." This
integration empowered the robot to not only respond textually but also embody emotions andactions, elevating interaction and immersing students in a more engaging learning experience.
Throughout the study, virtual pets accompanied learners during their out-of-class study
sessions, fostering continuous engagement and interaction. Ultimately, the virtual pets
seamlessly merged with the physical pet robots, allowing learners to showcase their final
learning outcomes through captivating situational dramas in front of the class.

614

3.3 System Structure

Figure 2. System architecture of pet robot with an interdependence mechanism.
The system, built upon the foundation of the Al Hakim et al. (2023) model, comprised three
integral components: a virtual pet embedded within a website (Fig. 2 [A]), a dedicatedsituational drama platform (Fig. 2 [B]), and a physical pet robot.
The web-based online learning platform was developed using WAMP and offeredquizzes and a shopping mall. This platform empowered learners to engage in independent
study outside the classroom while fostering a sense of companionship with their virtual pets.
Teachers could easily insert quizzes through a user-friendly interface, customizing question
content, answer choices, and feedback. Each completed quiz rewarded learners with virtualcoins. The shopping mall provided an opportunity for learners to use their earned coins topurchase pet food and accessories. Through the web platform, virtual pets acted as supportive
companions, offering encouragement and assistance to learners in their learning journey.
The situational drama platform to enable learners to showcase their learning outcomesthrough interactive drama performances. The platform is an immersive computer-based
system that seamlessly connects a tablet app and the physical robot, allowing the design andperformance of various learning scripts. This integration enabled learners to create interactive
presentations that incorporated the physical pet robot, ultimately enhancing their learning
experience and promoting their understanding of the material presented.
In this study, the ASUS ZENBO robot served as the foundation for developing thephysical pet robot, establishing a tangible connection with its virtual counterpart. The
developed pet model was meticulously integrated with the ZENBO SDK, harnessing its
capabilities in facial expressions, body movements, audio effects, and speech module.
Notably, the speech module was seamlessly integrated with the developed ChatGPT API
model, enabling the physical pet robot to receive and process the user's voice, facilitating
advanced language recognition and understanding. Consequently, the physical pet robot
would react with a captivating array of vibrant expressions, speeches, and bodily gestures,
faithfully replicating the actions and demeanor of the virtual pet. This cohesive integration
between the tangible and virtual pets cultivated an exclusive and dynamic bond, enablinglearners to engage in an interactive educational voyage that effortlessly traversed the confinesof both the virtual and physical dimensions.

615

4. Method
4.1 Participants and Learning Materials
The study involved 100 university learners enrolled in Japanese for Hospitality and Tourism in
Taiwan. They were divided into two groups, namely the experimental group (21 females and29 males) and the control group (23 females and 27 males), which were further subdivided by
the teacher. The groups were kept separate from each other during the experiment to ensure
unbiased evaluation of the results. All participants were approximately 20 years old.
The focus of the study was on managing hospitality in Japanese restaurants that are
pet-friendly. The study material encompassed several topics such as the culture of Japaneserestaurants and the application of courteous expressions, all of which were incorporated into
the learning script and inserted into the quizzes in the website. During the drama presentation,
learners and the robot engage in a restaurant scenario. Below is an excerpt of the dialoguebased script about the culture of Japanese restaurants (translated from Japanese).
ZENBO: (“excited expression” and “spinning around”) Hi there, nice to meet you!
Waitress: (surprised) Oh, a talking pet! That's amazing. Well, we're happy to have you both. Just
to let you know, we do allow pets in our restaurant, but we ask that they remain on a leash andwell-behaved.
Customer: Thank you for letting us know. I promise to keep him on a leash. And while we're
waiting for our food, could you tell us a little more about Japanese restaurant culture?
Waitress: Of course! In Japan, it's considered polite to use honorific expressions when addressing
others, especially in a formal setting like a restaurant. So instead of saying "you," we use "anata"or "omae." And it's also important to show gratitude and respect to our customers, so we always
strive to provide the best service possible.

4.2 Measuring Tools
To evaluate the learners’ learning outcomes, interactions, and acceptance, multiple
assessment tools were employed. These included pretest and posttest questionnaires thathad 15 questions each and a maximum score of 100. The teacher developed these tests andadministered them before and after the experiment. In addition, database logs were used tomeasure interaction rate, while a five-point Likert scale questionnaire was employed in theafter the experiment to measure learning acceptance. The questionnaire was based on theowner-pet relationship (McConnell et al., 2011) and learning technology acceptance (Hwanget al., 2011), modified to suit the context of the study, and consisted of 18 questions
categorized into three dimensions: robot design, co-learning with the robot, and learning uses
online platform. The questionnaire in this study showed high reliability with a Cronbach's α
value of 0.907 (N=100). The reliability of the three dimensions was confirmed by the
Cronbach's α values, which were 0.88, 0.90, and 0.83, respectively. Interview questions basedon Al Hakim et al (2022a) were also adapted to explore factors that may impact learners’
willingness to learn with proposed approach. It consisted of six questions that were openended, such as "Overall, what do you think are the advantages of learning with the system?".

4.3 Procedure
The experiment spanned over six weeks and consisted of four stages, each with two hours ofclasses per week, followed by mandatory online practice.
x Stage I occurred during the first week and involved a pre-test, introduction to learning
scripts, and an overview of the study's system and online learning website.

616

x

During the second to fourth weeks (Stage II), learners continued to learn course materials
and practiced using the online learning website independently.
x Stage III took place during the fifth week and involved drama rehearsals where learners
became familiar with the situational drama performance process and practiced with therobot under the teacher's guidance.
x The final stage, Stage IV, was the official drama performance that occurred during thesixth week. Learners demonstrated their final learning results with the robot on situational
drama platform, with each group performing independently without assistance. Post-tests,
questionnaires, and interviews were conducted after the performance.
Both the experimental and control groups followed the same four-stage experimental
process, were taught the same materials by the same teacher, and used online learning
websites. However, the experimental group used a learning website with a virtual pet robotthat required learners to obtain virtual coins during online practice to purchase food and takecare of the virtual robot's cages. The control group used a website without a virtual pet robot
for learning. During the drama performance, the experimental group used a physical robot with
pet characteristics that displayed lively expressions and actions according to the script, while
the control group used a general robot that only read script lines and did not have pet functions.

5. Findings
5.1 RQ1: Learners’ Learning Outcomes
The analysis employed analysis of covariance (ANCOVA) to compare the post-testscores of the experimental and control groups, with the pre-test scores used as a covariate.
The homogeneity test of intra-group regression coefficient revealed no significant interaction
between the two groups and the pre-test scores (F = 0.42, p = 0.29 > 0.05), allowing the useof pre-test scores as a covariate. Subsequently, ANCOVA was utilized to analyze the posttest scores of the two groups. The results of the statistical analysis, as displayed in Table 1,
indicate a significant difference in the mean scores between the experimental and control
groups (F = 2.19, p = 0.01 < 0.05). This suggests a notable disparity in the post-test scores
between the two groups.
Table 1. ANCOVA result of learners’ learning outcomesGroup
Experimental
Control*p < 0.05

N
50
50

Mean
80.98
76.21

SD
4.89
6.89

Adjusted mean
81.03
75.46

F
2.19*

η2
0.07

The findings support the claim that a pet robot equipped with an interdependence
mechanism is a more effective tool for improving learning outcomes compared to aconventional robot. These results align with earlier investigations (Chen, 2012; Lin et al., 2022)
that have demonstrated the positive impact of including a pet-like interaction mode in thelearning system, which enhances learners learning achievement. By taking care of their virtual
pets, learners felt a sense of responsibility, knowing that neglecting their pet would result in its
demise, thereby encouraging them to learn and earn virtual coins to buy supplies. Assigningeach learner an interdependent pet can, therefore, foster a sense of duty towards the pet andtheir own learning, ultimately leading to improved learning outcomes.

5.2 RQ2: Interaction Rates
The average daily interaction in learning activities was recorded for both the control andexperimental groups across four stages. Results show that the experimental group had
significantly higher activity levels than the control group (see Fig. 3). This suggests that the617

presence of the pet robot served as a stimulus for the learners to engage more actively in thelearning process. This phenomenon is attributed to the reciprocity of pet care (Lawton, 2017;
Chen, 2012), where pupils engage in more interactions to take care of their pets by feeding
and playing with them. In this study, learners increase learning frequency to earn coins andpurchase pet food due to the need to take care of the pet robots, while pet robots provide
feedback and support to learners at all times, forming a positive interactive relationship thatkeeps learners engaged in learning activities. Notably, based on the Figure 2, it was observedthat the interaction rate among the experimental and control groups continued to increase in
the final stage, that is official drama performing. This might attribute to the motivation-driven
effect (Al Hakim et al., 2022b), which compelled learners to study before the final exhibition
with the robots to be displayed to classmates and teachers.

Figure 3. The average daily interaction in learning activities among groups.

5.3 RQ3: Learners’ Acceptance of Learning
The responses to the questionnaire were analyzed with independent t-test to compare
the learners’ acceptance of learning after the treatment. A shown by Table 2, in terms of thedesign perception of the robot, both groups interacted with the physical robot and had similar
perceptions, with no significant difference found (p = 0.620 > 0.050). It means that both ofgroups felt that the appearance, expression, voice, and interaction of the developed robot
models, whether it has pet feature or not, were similar. In terms of co-learning with the robot,
both groups scored higher than the average level, with the experimental group having asignificantly higher average score (M = 4.130) than the control group (M = 3.850), indicatingthat pet robots can provide a better educational encounter for learners. On the other hand,
there was no significant difference between the two groups in terms of user acceptance whenusing the online learning platform with or without the pet robot function (p = 0.220 > 0.050).
Table 2. T-test result of learners’ acceptance of learning
Dimension
Robot design acceptance(e.g., I find ZENBO's appearance andexpressions adorable, like a pet)Satisfaction of co-learning with the robot(e.g., Having ZENBO as a companion has
helped me learn the materials better)
Satisfaction of using the online platform(e.g., The online learning platform hasenhanced my learning experience)

Group

Mean

SD

F

Sig.

Experimental

3.70

0.76

0.253

0.617

Control

3.61

0.79

Experimental

4.13

0.66

8.91

0.004*

Control

3.85

0.94

Experimental

3.61

0.45

1.523

0.222

Control

3.60

0.59

*p < 0.05

618

Furthermore, to gain a better understanding of how learners perceive the proposed
learning approach, ten-minute interviews were conducted with six learners from theexperimental group, representing high, medium, and low academic achievement. The
outcomes of these interviews revealed intriguing findings. Firstly, learners expressed theirfondness for the pet robot's cute appearance and lifelike behavior, finding them highly
appealing. Motivated by a sense of responsibility towards their virtual pet and a genuine fearof it going hungry, learners engaged with the robot more earnestly. They also highlighted theseamless connection between the virtual and physical pet robots, enabling them to extend
their emotional bond from the virtual realm to the physical world. However, some interviewees
suggested that incorporating more pet-like facial expressions and personalized interactivemodes could greatly enhance the enjoyment factor of interacting with pet robots. Specifically,
they expressed their desire for interactive behaviors like cuddling and spinning, which would
further enrich the design and interaction possibilities of the pet robot.
These insightful interview responses indicate that the proposed learning approach not
only captivates learners by leveraging the novelty of the robot but also fosters motivation
through a heightened sense of responsibility towards their virtual pets. Furthermore, learners
expressed their anticipation for more design options and personalized responses from their
pet robots, reflecting their eagerness to have a unique and customized pet robot that builds
upon the established relationship.

6. Conclusion, Drawbacks, and Further ResearchThis study presents a novel approach to educational robot design by integratinginterdependence theory. The aim is to develop pet robots that establish enduring relationships
with learners, thereby enhancing their learning performance. The interdependent petmechanism, encompassing the owner's responsibility for the pet's care and the pet's provisionof emotional support, is seamlessly integrated into both virtual and physical platforms, alongwith situational drama learning presentations. This integration empowers learners to take careof their pet robots anytime, anywhere, fostering a sense of responsibility for their own learning.
The results of the experiment indicate that robots designed as interdependent pets
have a more positive impact on learners in terms of learning outcomes, long-term interactions,and acceptance of learning than conventional robots. The study provides a new insight on theuse of robots in education, combining generative AI, mobile and reality technologies, andpractices, resulting in an intelligent learning environment where robots act as interdependentpets. Educators and system developers can incorporate this approach into the learning system
to create a highly engaging and interactive learning experience for learners, enabling them toestablish lasting relationships with learning agents and improve their learning performances.
However, this study was limited conducted on a Japanese for Hospitality and Tourism
course for university learners, making it uncertain whether the findings can be generalized.
Further experiment could explore the potential of this approach for other age groups andsubjects. Also, future studies should consider several potential enhancements to pet robot
design based on valuable feedback received from learners. Firstly, there is a need to expand
and enhance the range of pet-like features beyond basic interactions such as satiety and basic
expressions. Incorporating a broader spectrum of pet emotions, improved cleanliness, andmore specific behaviors and attire could potentially increase user interest and willingness toengage with the pet-like robots. Secondly, strengthening the connection between virtual andphysical robots can be achieved by assigning unique names, distinct speech patterns, andindividual personalities to each pet, ultimately would enhance the personalization and deepen
the connection between the virtual and physical manifestations of the robots.

Acknowledgements
The study was supported by the National Science and Technology Council, Taiwan. Grand
Number: MOST 110-2511-H-008-004-MY3 and NSTC 112-2811-H-008-006.

619

References
Al Hakim, V. G., Yang, S. H., Liyanawatta, M., Wang, J. H., & Chen, G. D. (2022a). Robots in situatedlearning classrooms with immediate feedback mechanisms to improve students' learning
performance. Comput. & Educ., 182, 104483.
Al Hakim, V. G., Yang, S. H., Wang, J. H., Chang, Y. C., Lin, H. H., & Chen, G. D. (2023). A Pet-Like
Model for Educational Robots: Using Interdependence Theory to Enhance Learning and Sustain
Long-Term Relationships. In 2023 IEEE 23rd Int. Conf. on Adv. Learn. Technol. (ICALT). IEEE.
Al Hakim, V. G., Yang, S. H., Wang, J. H., Li, Y. J., Chen, Y. H., & Chen, G. D. (2022b, November).
Design a Robot as a Double with Micro-expression to Participate in a Virtual Situational Learning
Environment and its Effect on Students’ Learning Performance. In Proc. of the 30th Int. Conf. on
Comput. in Educ. (ICCE) (pp. 467-476).
Al Hakim, V. G., Yang, S. H., Wang, J. H., Yen, C. C., Yeh, L., & Chen, G. D. (2021, November). Robot
with Embodied Interactive Modes as a Companion Actor in Journey of Digital Situational Learning
Environment and its Effect on Students’ Learning Performance. In Proc. of the 29th Int. Conf. on
Comput. in Educ. (ICCE) (pp. 441-450).
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., & Tanaka, F. (2018). Social robots foreducation: A review. Sci. robot., 3(21), eaat5954.
Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educ. Res.,
18(1), 32-42.
Bylieva, D., Almazova, N., Lobatyuk, V., & Rubtsova, A. (2020). Virtual pet: trends of development.
In Digital Sci. 2019 (pp. 545-554). Springer International Publishing.
Calvo-Barajas, N., & Castellano, G. (2022, September). " I have an idea!" enhancing children's verbal
creativity through repeated interactions with a virtual robot. In Proc. of the 22nd ACM Int. Conf. onIntell. Virtual Agents (pp. 1-8).
Chen, Z. H. (2012). We care about you: Incorporating pet characteristics with educational agentsthrough reciprocal caring approach. Comput. & Educ., 59(4), 1081-1088.
Erbay, F., & Doğru, S. S. Y. (2010). The effectiveness of creative drama education on the teaching ofsocial communication skills in mainstreamed students. Proc.-Soc. and Beha. Sci., 2(2), 4475-4479.
Díaz-Boladeras, M. (2022). Bond formation with pet-robots: An integrative approach. Current
Psychology, 1-18.
Frude, N., & Jandrić, P. (2015). The intimate machine–30 years on. E-learn. and Digital Media, 12(34), 410-424.
Heljakka, K., Lamminen, A., & Ihamäki, P. (2021, October). A model for enhancing emotional literacy
through playful learning with a robot dog. In 2021 ICECCME (pp. 1-7). IEEE.
Hwang, G. J., Wu, C. H., Tseng, J. C., & Huang, I. (2011). Development of a ubiquitous learning platform
based on a realǦtime helpǦseeking mechanism. British J. of Edu. Technol., 42(6), 992-1002.
Johnson, D. W., & Johnson, R. T. (2009). An educational psychology success story: Social
interdependence theory and cooperative learning. Educ. Res., 38(5), 365-379.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge
university press.
Lawton, L. (2017). Taken by the Tamagotchi: How a toy changed the perspective on mobile
technology. The iJournal: Student J. of the Fac. of Infor., 2(2).
Leite, I., Martinho, C., & Paiva, A. (2013). Social robots for long-term interaction: a survey. Int. J. of Soc.
Robot., 5, 291-308.
Ligthart, M. E., de Droog, S. M., Bossema, M., Elloumi, L., Hoogland, K., Smakman, M. H., ... & Ben
Allouch, S. (2023, March). Design Specifications for a Social Robot Math Tutor. In Proc. of the2023 ACM/IEEE Int. Conf. on Human-Robot Interaction (pp. 321-330).
Lin, V., Yeh, H. C., Huang, H. H., & Chen, N. S. (2022). Enhancing EFL vocabulary learning withmultimodal cues supported by an educational robot and an IoT-Based 3D book. System, 104,
102691.
McConnell, A. R., Brown, C. M., Shoda, T. M., Stayton, L. E., & Martin, C. E. (2011). Friends with
benefits: on the positive consequences of pet ownership. J. of Person. and Soc. Psychol., 101(6),1239.
Mubin, O., Stevens, C. J., Shahid, S., Al Mahmud, A., & Dong, J. J. (2013). A review of the applicabilityof robots in education. J. of Technol. in Educ. and Learn., 1, 1-7.
Thibaut, J.W., & Kelley, H.H., (1959). The social psychology of groups. New York: Wiley.
Van Lange, P.A., & Balliet, D. (2014). Interdependence Theory. American Psychological Association.
Westlund, J. M. K., Park, H. W., Williams, R., & Breazeal, C. (2018, June). Measuring young children's
long-term relationships with social robots. In Proc. of the 17th ACM Conf. on Interact. Design andChildren (pp. 207-218).

620

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Analysis to Creation: Using the ADDIE
Model to Develop an Educational Game forChildren
Yufan ZHANGa*, Nurul Nadwa ZULKIFLIb & Ahmad Fauzi MOHD AYUBc,
Zewen SHANGdaYufan ZHANG, Universiti Putra Malaysia, Malaysiab
Nurul Nadwa ZULKIFLI, Universiti Putra Malaysia, Malaysiac
Ahmad Fauzi MOHD AYUB, Universiti Putra Malaysia, MalaysiadZewen SHANG, Singapore University of Technology and Design, Singapore* nurulnadwa@upm.edu.my
Abstract: Creating educational games for children is a notable emphasis, especiallywhen using the ADDIE model as a game development framework. Furthermore, we
introduce the ADDIE model, a recognized instructional design framework, to illustrate
its potential in the video game industry. In this study, by using constructivist and social
learning theories, educational play can be fostered through exploration, cooperation,and reflection. Hence, we create a children's educational video game using the ADDIE
model to explore creating suitable educational games. Striking the right balance
between education and entertainment necessitates a comprehensive grasp of how
children learn and develop. By leveraging this knowledge, developers and educators
can create video games that are both enjoyable and promote learning and developmentin children.
Keywords: ADDIE model, children's educational game, game design

1. Introduction
In response to the growing recognition of digital games for early childhood learning, wedeveloped an educational video game for children under 7, focusing on reading, numeracy,social, and emotional skills. Our study is utilizing the ADDIE paradigm, a systematic
instructional design framework, we detail our game's creation in this framework.

2. Related Video Game Analysis
Hull (2010) found that video games can benefit children with emotional disorders. Parents
should supervise game usage, ensuring age-appropriate choices(Gözüm & Kandır, 2021).
Video games encourage active learning in the problem-solving (Dhiman, 2023). Andinnovative thinking. Cooperative play and player instruction promote skill acquisition through(Egenfeldt-Nielsen et al., 2019). Yet, overly complex multimedia can hinder the learning(Chang et al., 2017).

3. Applying the ADDIE Model for Effective Game Development
3.1 ADDIE Model
Game designers employ ADDIE to create well-structured, engaging games aligned with
objectives for the intended audience (Davis, 2013; Muruganantham, 2015). As shown in
Figure 1, the ADDIE paradigm offers a structured, collaborative approach to game621

development, facilitating clear goal setting, comprehensive planning, and accurate progressevaluation.

Figure 1. The ADDIE Paradigm

3.2 ADDIE Model in Game Design Process
The ADDIE model ensures purposeful game development that meets audience needs, asseen in Braad et al. (2016) study It offers designers to create effective and engaging games
as illustrated in Figure
2.
gFigure 2. The ADDIE Game Design Process

4. The Production and Design of Educational Games for Children
In this research, we present an educational game flowchart in Figure 3. The flowchart details
the different actions and learning objectives of the game.

Figure 3. The Game Process622

4.1 Tool Selection for Development
We present our decision to develop our app on the iOS platform instead of Android. There
were several factors that influenced this choice, including the relative ease of development
and testing on the iOS platform, and the strong user engagement.

4.2 Function Realization
4.2.1 The Realization of Game Story
As our primary target audience is comprised of young children, specifically those aged below
seven, our game necessitates a theme that is both user-friendly and possesses a morecomprehensive storyline, as opposed to a complex user interface.
4.2.2 The Realization of User Interaction
As our main audience is under seven years old, we've connected custom buttons to visuallyappealing elements like tables and classrooms, creating a vibrant and captivating userinterface. Utilizing the ADDIE model ensures clear learning goals, content alignment, andmechanics congruence(Kapp, 2012; Lim et al., 2013). Designing personalized educational
games for children, aligned with learning objectives, and employing frameworks like ADDIE,
constructivism, and scaffolding, can result in immersive and impactful learning experiences.

Acknowledgements
I extend heartfelt gratitude to my tutors, Dr. Nadwa and Professor Fauzi, whose unwavering
guidance and support have been instrumental in my studies. Special thanks to a fellow PhD
student for their invaluable technical support alongside Zewen during this research.

References
Braad, E., Žavcer, G., & Sandovar, A. (2016). Processes and models for serious game design anddevelopment. Entertainment Computing and Serious Games: International GI-Dagstuhl
Seminar 15283, Dagstuhl Castle, Germany, July 5-10, 2015, Revised Selected Papers,
Chang, C.-C., Liang, C., Chou, P.-N., & Lin, G.-Y. (2017). Is game-based learning better in flow
experience and various types of cognitive load than non-game-based learning? Perspectivefrom multimedia and media richness. Computers in human behavior, 71, 218-227.
Davis, A. L. (2013). Using instructional design principles to develop effective information literacy
instruction: The ADDIE model. College & Research Libraries News, 74(4), 205-207.
Dhiman, D. B. (2023). Games as Tools for Social Change Communication: A Critical Review. Global
Media Journal, 21, 61.
Gözüm, A. İ. C., & Kandır, A. (2021). Digital games pre-schoolers play: parental mediation andexamination of educational content. Education and Information Technologies, 26(3), 32933326.
Hull, K. B. (2010). Computer/video games as a play therapy tool in reducing emotional disturbances inchildren Liberty University].
Kapp, K. M. (2012). The gamification of learning and instruction: game-based methods and strategies
for training and education. John Wiley & Sons.
Lim, T., Louchart, S., Suttie, N., Ritchie, J., Aylett, R., Stanescu, I. A., Roceanu, I., Martinez-Ortiz, I., &Moreno-Ger, P. (2013). Strategies for effective digital games development andimplementation. In Cases on digital game-based learning: Methods, models, and strategies(pp. 168-198). IGI Global.
Muruganantham, G. (2015). Developing of E-content package by using ADDIE model. International
Journal of Applied Research, 1(3), 52-54.

623

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Case study-based research onunderstanding app user engagement todevelop environmental literacy of urban
residents
Ewa DUDAMaria Grzegorzewska University, Poland
eduda@aps.edu.pl
Abstract: Environmental literacy enables action to be taken against climate change.
These actions should naturally be supported by developing information technology.
Mobile applications can therefore be successfully used as a tool for learning,awareness-raising, but also for shaping environmentally friendly behavior. This article
presents an analysis of the functionalities of mobile apps that enhance or undermine
user engagement with the application. Qualitative research, carried out as a casestudy, was conducted by analyzing publicly available user reviews. The results showthat a well-designed application can provide valuable support for ongoing educational
interventions aimed at enhancing the environmental literacy of urban residents. It
presents indications that are worth using when designing environmental applications
to ensure a high level of future user engagement. Results can be successfully used bymobile app designers, but also educators, city authorities, or local activist.
Keywords: environmental literacy, educational applications, green transformation,
pro-environmental behavior, urban education, user engagement

1. Introduction
Developing the environmental literacy of urban residents is becoming a necessity in the faceof ongoing climate change. Such literacy is perceived as (1) understanding the resources atour disposal, (2) understanding that the actions of people (social beings) have roots andconsequences, (3) the ability to work together to achieve common, rational goals, (4) the ability
to responsibly obtain natural goods for basic needs and desires (Roth, 1992). Indeed, a proenvironmental approach must be embraced at multiple levels, taking care not only tostrengthen awareness but also knowledge, attitudes and practical skills that will be an integral
part of urban residents' conscious daily routines.
The changing society needs and the rapid development of information technology
imply the question of how to effectively encourage people to engage in a process of shaping
their environmental literacy, and to maintain that engagement over time. This study aims to
explore the issue of co-development of solutions to foster green social transformation. The
initial concept of the tool to support educational interventions among urban residents (Duda etal., 2022) is based on the use of a mobile application to facilitate environmentally friendly
behaviors. To address this, the study attempts to answer the following research question:z
What features of a mobile application for environmental literacy encourage users to adopt
environmentally friendly behavior?
z
What features of a mobile application for environmental literacy discourage users to adopt
environmentally friendly behavior?

2. Theoretical Framework
2.1 Environmental Literacy

624

The term environmental literacy is understood as an advanced level of conscious humaninteraction with the environment. It is defined as “the ability to appropriately read and to utilize
environmental information, to anticipate rebound effects, and to adapt according to information
about environmental resources and systems and their dynamics” (Scholz, 2011:8). On theother hand, Cole drew attention to the cultural aspect of the concept, where “environmental
literacy can be understood as a culturally specific body of knowledge that fosters particular
ways of thinking and acting in the world” (Cole, 2007:39).
The development of environmental literacy is based on active education that goesbeyond the mere transmission of information. The vast majority of literature on developing
environmental literacy focuses on formal, informal or non-formal education (Forsyth, 2018;
Taylor, 2020). Research on aspects related to the use of mobile apps in education forenvironmental literacy is much more limited. Most often, they focus on application-specificresearch such as educational process among participants and gamers (Bowser et al., 2014),
motivators to reinforce pro-environmental behavior (Pacheco, & Faria, 2022).

2.2 User Engagement
The definition used in this study is as proposed by Attfield, Kazai and Lalmas (2011).
Regarding this definition “user engagement is the emotional, cognitive and behaviouralconnection that exists, at any point in time and possibly over time, between a user and a
resource, […] user engagement with a technological resource is not just about how a single
interaction unfolds, but about how and why people develop a relationship with technology andintegrate it into their lives” (Attfield, Kazai, & Lalmas, 2011:2).

3. Methods
3.1 Procedure
The Google Play Store platform was used to search for mobile applications. Downloadable,
English-language mobile applications with publicly available user reviews were included in the
searches. The keyword used was "environment", next the search continued using the snowball
method. The apps were examined for their widest range of features addressing environmental
issues. The first search yielded 86 results. In the next step entertainment games, dictionaries,
and quiz applications were excluded. Finally, three apps most related to the research objective
were selected, with a significant number of user reviews available. They were: “Environment
Challenge” (106), “Earth Hero: Climate Change” (445) and “JouleBug” (286 available reviews).

3.2 Data Analysis
As the aim of the study was to explore constructs and categories emerging from both positive
and negative user statements, the study employed inductive analysis (Mary, & Pour, 2022). In
this study, the data was divided into two main thematic areas. The first comprisedfunctionalities that, according to users, stimulate them to undertake environmental behavior
through the app. The second covered functionalities that discourage the use of the app, thus
not encouraging environmental efforts.

4. Results and Discussion
Analyzing the posted reviews, a reflection arises that the users of the above applications arepeople who are interested in environmental problems, but their knowledge in this area is notvery advanced and they need regular stimulation to take environmentally friendly actions. The
app makes it easier for people to be more eco-friendly, especially those who may be struggling
with environmental anxiety during the climate crisis.
Users stressed that their lifestyles have changed a bit since they started using the app.
The friendly, non-judgmental tone of the notifications helped them. Also positive was the lack625

of competition, a system based on goal-setting and reminders. Tips and suggestions wereeasy to follow and incorporate into a daily routine. A higher rating was given to the app
providing measurable challenges and more personalized, as is in line with Morreale et al.
(2015) research. Challenges that were too general or those that did not provide a point ofreference caused a decline in engagement. Users questioned issues such as having achallenge to consume some percent less power when they have no idea about the quantity of
their current consumption. The challenges formulated should therefore be more flexible.
Also demotivating for some users were vague statements about the challenges. Whileusers received daily reminders to keep up with the challenges, they lacked detailed guidanceon how to do this. On the other hand, seven out of ten users were satisfied with the support
provided to them. They enjoyed the articles about the issues in the environment and steps tominimizing human impact. The data provided and the concept of environmental challenges
was both useful and educative. They declared, that app made them more aware of their day
to day actions and should be further developed and maintained. Such applications can help
to develop environmentally friendly habits, and increase users’ knowledge (Rosal et al., 2021).

5. Conclusion
The main novelty of the study is based on the uniqueness of the topic addressed. In addition
to the small number of studies on the formation of various pro-environmental habits andbehaviors of residents, it is difficult to find research on the engagement of mobile app users,
especially based on a qualitative analysis of the opinions of current app users. This study,
therefore, extends the available literature on the subject by contributing to the promotion ofenvironmental action. The results of the research support the application design process ofthe project in which the authors of the text are involved. They can be also successfully usedby other mobile app designers, educators, and city authorities. A limitation of this study is that
it was carried out by a single researcher, which may introduce a bias in the coding process.

Acknowledgements
The research is supported funding from Iceland, Liechtenstein and Norway under the EEA
Funds and the state budget of Poland via the National Centre for Research and Development
within "Applied Research" Programme, grant no. NOR/IdeaLab/GC/0003/2020-00.

References
Attfield, S., Kazai, G. & Lalmas, M. (2007). Towards a science of user engagement (position paper).
WSDM workshop on user modelling for Web applications.
Cole, A.G. (2007). Expanding the field: revisiting environmental education principles throughmultidisciplinary frameworks. Journal of Environmental Education, 38(2), 35–45.
Duda, E., Anacka, H., Kowal, J., & Obracht-Prondzyńska, H. (2022). Participatory co-design approachfor Greencoin educational tool shaping urban green behaviors. 2022 ICALT, 98-100.
Forsyth, D. (2018). Education that Fosters Environmental Literacy (Doctoral dissertation).
Mary, S. R., & Pour, M. H. (2022). A model of travel behaviour after COVID-19 pandemic: TripAdvisor
reviews. Current Issues in Tourism, 25(7), 1033-1045.
Morreale, P., McAllister, J., Mishra, S., & Dowluri, T. (2015). Turning Leaf: Eco-visualization for Mobile
User Engagement. Procedia Computer Science, 52, 690-694.
Pacheco, D., & Faria, C. (2022). Roteiro Entre-Marés: an educational app for ocean literacy promotion.
Journal of Biological Education, 1-14.
Rosal, H.V., Borilla, J.R.F., [...] & Lagman, A.C. (2021). EcoAction Mobile App: A Game Practice for
Eco-activist. 11th International Workshop on Computer Science and Engineering, 23-28.
Roth, Ch.E. (1992). Environmental literacy: its roots, evolution and directions in the 1990s. Columbus,
OH: ERIC Clearinghouse for Science, Mathematics, and Environmental Education.
Scholz, R.W. (2011). Environmental literacy in Science and Society: From Knowledge to Decisions.
New York: Cambridge University Press.
Taylor, M. L. (2020). Playbuilding for Environmental Literacy: A Guidebook Resource for Secondary
Educators. Faculty of Education, Brock University St. Catharines, Ontario.

626" !   !" " "   #" !
#" !"$  #" ! #"                    #$   "!# # 

#&%!!!)$% !#%$! '#$%) " 
!#%!    !!)  %#! '#$%) " $# $%%&%!# !#%!  !!))&$& '#$%) "   
    '63 :=E A3:434471/1G 6/A 033< /< 7;>=@B/<B 7AAC3 7< B63 23A75< =4
7<AB@C1B7=</: ;3B6=2A /<2 7BA <35/B7D3 34431BA /@3 ;/7<:G A6=E< 0G B63 @32C132
E7::7<5<3AAB=:3/@</<2B6367562@=>=CB/<24/7:C@3@/B3A/;30/A32:3/@<7<56/A
@3137D32/BB3<B7=<7<B6323A75<=4B7<AB@C1B7=</:;3B6=2A031/CA3=4B631=;07</B7=<=4 :3/@<7<5 1=<B3<B /<2 5/;3A < B67A ABC2G E3 CA3 /22@3AA7<5 ;=23A :3/@<7<5 7<=>3@/B7=<AGAB3;1=C@A3/A/1/A3=0831BB=3F>:=@3B63>=B3<B7/:=47;>@=D7<5B63:=E
A3:434471/1G 0G 5/;30/A32 :3/@<7<5 '= B67A 3<2 / A723A1@=::7<5 D723= 5/;3 E/A
23A75<32B=1=;>:3B3B631/A3ABC2GB/A9B=3F/;7<3B6334431B=45/;30/A32:3/@<7<5=<A=:D7<5B63:=EA3:434471/1G7AAC311=@27<5B=B633F>3@7;3<B/:@3AC:BAB63@3E/A<=27443@3<137<:3/@<7<5/1673D3;3<B03BE33<B633F>3@7;3<B/:5@=C>ABC23<BAE6=
CA32 5/;30/A32 :3/@<7<5 /<2 B63 1=<B@=: 5@=C> ABC23<BA E6= CA32 B@/27B7=</:
:3/@<7<5 ;3B6=2A =E3D3@ E7B6 A7;7:/@ A1=@3A 7< :3/@<7<5 /1673D3;3<B B63
3F>3@7;3<B/: 5@=C> ABC23<BA A6=E32 67563@ A3:434471/1G B6/< B63 1=<B@=: 5@=C>
ABC23<BA < /227B7=< 7B E/A A6=E< B6/B B63 5/;30/A32 :3/@<7<5 />>@=/16 272 <=B
7;>=A3/227B7=</:1=5<7B7D3:=/2 /;30/A32:3/@<7<5A3:434471/1G/22@3AA7<5;=23A:3/@<7<5  

(4;54558202H8=;40A=8=602C8E8C84B70B;>=6144=>58=C4A4BCC>A4B40A274AB(8=24C78B
8BBD48B2;>B4;HA4;0C43C>BCD34=CBJ;40A=8=602784E4<4=C0=3?BH27>;>6820;2>6=8C8>=B>A
4G0<?;40??A>?A80C4B4;54558202H20=74;?BCD34=CBA43D24;40A=8=6BCA4BB8=2A40B4
;40A=8=64558284=2H0=3<08=C08=8=C4A4BC0=34=6064<4=C8=;40A=8=602C8E8C84B(8;E4<0=
0B0II0 >=E4AB4;HC74=460C8E445542CB>5;>FB4;54558202H0A4>1E8>DB(CD34=CB
F7>B?4=3;>=6?4A8>3B>5C8<48=;>FB4;54558202H502478674A;4E4;B>5BCD3H8=6BCA4BB
4G7818C0E>830=241470E8>A0=3C4=34=2HC>1>A43><F7827D;C8<0C4;H;403BC>78673A>?>DC
A0C4B#0AC8=#0AB7  
8673A>?>DC0=3508;DA4A0C4B8=2><?DC4AA4;0C432>DAB4B0A402><<>=?A>1;4<><4B#4=34B 0=34F0A3  0A6D4BC70CBCD34=CBF7>5024270;;4=68=6
0=3D=50<8;80A;40A=8=62>=C4=C0A4<>A4;8:4;HC>34E4;>?;>FB4;54558202HA43D28=6
<>C8E0C8>=0=3F8;;8=6=4BBC>;40A=#40=F78;4,48=0&8=6(7D08  2>=58A<43C70CC74<08=270;;4=64B8=2><?DC4A2>DAB4B2><45A><01BCA02C432>=24?CBF74A4:=>F;4364
?>8=CB0A438B?4AB430=3C748=CA8=B82;>682A4;0C8>=B78?14CF44=2>=24?CB8B70A3C>58=3
)74A45>A48=BCAD2C8>=0;<4C7>3BC70C<0C27C74;40A=8=62>=C4=CA4248E40CC4=C8>=B)742>=24?CD0;8I0C8>=34B86=?A8=28?;4B0=30??;820C8>=B>560<410B43;40A=8=6
70E4608=436A40C0CC4=C8>=)>180B;4C274A,8=3  0<410B43;40A=8=68B=>C0=4FC78=64B?4280;;H5>A2>=C4<?>A0AHBCD34=CBF7>0A4022DBC><43C>4;42CA>=8234E824B
40A;H>= =C78BBCD3H033A4BB8=6<>34B;40A=8=68=>?4A0C8>=BHBC4<2>DAB48BDB430B0
20B4>1942CC>DB460<410B43;40A=8=60B0=0CC4<?CC>B>;E4C748BBD4B>5;>FB4;54558202H
C70C0A8B4B8=2><?DC4AA4;0C432>DAB4B


627



 !" #$$ !$#   !"#%! $)$%!&#$
%?4A0C8>=BHBC4<2>DAB48B02>A45>D=30C8>=2>DAB4F8C7<0=H2>=24?CBBCA>=6
?A8=28?;4B0=37867346A44>501BCA02C8>=,48=0&8=6(7D08  F7827A4;0C43C>
<D2770A3F0A4:=>F;4364B0A4BD;C78673A>?>DC0=3508;DA4A0C4B70E4144=A4?>AC43
F8C75>2DB8=6>=<D270CC4=C8>=B><4B#4=34B 33A4BB8=6<>34B<40=BC74
B?4285820C8>=>530C0;>20C8>=BA4@D8A431H0=>?4A0C8>=;B><0BC4A8=6033A4BB8=6<>34B
8=E>;E4BC78=:8=60=3A40B>=8=6C70CDBD0;;HA4@D8A435>A>?4A0C8>=BHBC4<;40A=8=6B>=4
>5385582D;C84B8=;40A=8=6>?4A0C8>=BHBC4<<>A434<0=3B5>A?A02C8201;4;4BB>=BF8C7
A40B>=01;4;40A=8=60??A>0274BC70=?DA4;HC74>A4C820;>=4B70E4144=<034F74=
;40A=8=6033A4BB8=6<>34B>;0=0:8B4C0; 0243F8C7270;;4=68=6;40A=8=62>=C4=C
8C8B<>A4;8:4;HC>70E48=BD558284=CB4;54558202H#0AC8=#0AB7   "%!!($)! #  %'%$(4;54558202HC74>AH>A868=0;;HBC4<<435A><>1942C8E40=0;HB8B>=270=64B02784E438=
540A5D;0=30E>830=C1470E8>A0=3DA034B2A8143B4;54558202H8B2>=24A=43F8C78=38E83D0;BJ
14;845B8=C748A2>=CA>;>E4A270;;4=68=634<0=3B0=3DA0  8E4AB445542CB0A4
?A>3D24334?4=38=6>=E0A8>DB;4E4;B>=B4;54558202H CF0B5>D=3C70C0BCA>=6B4=B4>5
B4;54558202H8B7867;H2>AA4;0C43F8C78<?A>E4<4=C>57D<0=022><?;8B7<4=C#033DG
>BB4;8=  (27D=:  A4?>AC43C70C?4>?;4F8C77867B4;54558202HA460A3385582D;C
C0B:B0B270;;4=64BC>14<0BC4A43A0C74AC70=0BC7A40CBC>140E>8343 =CA8=B828=C4A4BC
0=3344?4=6A>BB<4=C8=;40A=8=602C8E8C84B<0H145>BC4A4340B8;HF8C774;?>5BD27B4;5
4558202H/0902>E0"H=27B?4=B7034  ;B>78674AB4;54558202H<0HA43D24
;40A=8=65ADBCA0C8>=5028;8C0C4;40A=4AJB455>AC0=38<?A>E4BDBC08=8=68=C4A4BC8=270;;4=68=6
02C8E8C84B(8;E4A<0=0B0II0 ;C7>D67B4;54558202H70B<>34A0C43A4;0C8>=B78?
F8C7;40A=8=6?4A5>A<0=248C8B=>C=424BB0A8;HC70C7867B4;54558202H;403BC>78674A;40A=8=6
?4A5>A<0=240;:8B  0E8B 5>D=3C70C?4>?;4F8C778674AB4;54558202H70E4
78674ABD1B4@D4=C;40A=8=6?4A5>A<0=24  $! $#  $ ??;820C8>=?>C4=C80;>560<410B43;40A=8=6F0B6A>D=38=0<>D=C>5A4B40A27)74
0224?C0=24>560<410B43;40A=8=670B144=F834;HC0:4=8=43D20C8>=0;2>=C4GC<>=6
0??;820C8>=14=458CB5028;8C0C8=6;40A=8=61H5>BC4A8=6<D;C8?;4?4AB?42C8E4BBD270B2>6=8C8E4
05542C8E41470E8>A0;0=3B>28>2D;CDA0;4=6064<4=C8B5D;;>54G?42C0C8>=5A><02CD0;
0??;820C8>=!0=4C0;  >F4E4A4<?70B8B>=;H>=;40A=8=6?4A5>A<0=2470B144=
5038=68=2>=CA0BCC>6A>F8=6B785CC>02><18=0C8>=>5;40A=8=60=3?BH27>;>6H45542C
$>C01;HB><4@D4BC8>=BF4A4A08B430;>=6F8C70CC4=C8>=>=7>FC><0:4DB4>5C7460<4
?A><8B45>A43D20C8>=?DA?>B4BF78;4C0:8=610;0=24>54=C4AC08=<4=C0=3;40A=8=68=
022>D=C=>C74A270;;4=64>52>=E4AC8=6;40A=8=6>1942C8E4BC>60<434B86=8B?A><8=4=C,74=34B86=8=660<410B43>=?0AC82D;0A?4306>6HC70CA4@D8A4B;>6820;018;8CHC>
D=34ABC0=301BCA02C=4BB8CF8;;4G024A10C4C74270;;4=64>5C78B2>=E4AB0C8>=#>A4=>4A
4C0; &4306>6820;A4@D8A4<4=CB3A8E4=1H385582D;C;40=8=62>=C4=C<0HA4BD;C8=;>F
F8;;8=6=4BBC>?4AB8BC8=E>;E4<4=C8=;40A=8=6C0B:B,4BCF>>3  4=4A0;;40A=8=6
385582D;C84B20=70??4=>F8=6C>8=034@D0C4>A8=0??A>?A80C4C40278=6,4BCF>>3  
4B?4280;;H8=C7440A;HBC064B>5;40A=8=6F8C72><?;820C43;40A=8=6?0CC4A=BC70C70A3C>
34C4A<8=4A40B>=B/70>F0=6.8=  
0<434B86=8=6;02:B>52>=B4=BDB>=D=85>A<AD;4B1420DB48C8=E>;E4B>5
2><?;820C460<434B86=502C>ABBD270B;40A=8=6>1942C8E4B<4270=8B<50=C0BHE0;D4
8=C4A02C8>=5A443><=0AA0C8E4B4=B0C8>=270;;4=64BB>280;8CH0=3<HBC4AH(78(787

628 %=4>5C74<>BC2A8C820;?A8=28?;4BF0B?A4B4=C431H(7010;8=04C0;  
4<?70B8I8=6C70C0B4@D4=24>560<402C8E8C84B8BBCAD2CDA431H2><8=6;40A=8=6>1942C8E4B
0=360<402C8E8C84B)>BD??>ACC7445542C4<14338=660<48=;40A=8=6<44C8=6C74=443B
>52>DAB4F8C705>2DB>=;40A=8=66>0;B5443102:0=38=C4A02C8>=8=C7460<45D=2C8>=B
;0BF03$03>;=H  (>C70C8C8B4BB4=C80;C>2>=B834A;40A=8=60CCA81DC4B8=2;D343>5
01>E460<434B86=502C>AB0=360<45D=2C8>=B )74BCD3H8=6@D4BC8>=B0A40B5>;;>FB
>FC>2><18=4;40A=8=62>=C4=CB0=360<4>?4A0C8>=0;?A>24BB4B8=60<434B86=
>D;3C7434E4;>?4360<48<?A>E4B4;54558202H0=302034<8202784E4<4=C
>D;3C7434E4;>?4360<41A8=60338C8>=0;2>6=8C8E4;>038=B>;E8=6;>F4558202H8BBD4B
 $!

)7460<4<>3D;48=2;D34B * <>3D;42>=B8BCB>5=0E860C8>=<>342>=C4=C38B?;0H0=3
10B822>=CA>;B &7HB82B<>3D;4C7460<4=443BC>8=C4A02CF8C7C74?4A5>A<0=24>5
>1942CB68E4=?7HB820;270A02C4A8BC82BBD270BC74B8I4>5C745>A24F74=C74270A02C4A
9D<?B (24=4<>3D;4C74C4AA08=;4E4;0=3270A02C4A<0=064<4=C8=C7460<4,74=
34E4;>?8=6C7460<4C74C7A44<>3D;4BF4A434E4;>?43DB8=6C74*=8CH60<44=68=40B
C7434E4;>?<4=CC>>;8ABC;HDB4C74?7HB82BBHBC4<6A0?782BBHBC4<0=30D38>4;4<4=CB
8=*=8CHC>34E4;>?B24=4<>3D;4(D1B4@D4=C;H8=C741D8;C60<4B24=460<4270A02C4AB
0A4033430=3C74270A02C4A1470E8>A0=360<4;>6828=C74B24=40A48<?;4<4=C43C7A>D67
*=8CHB2A8?CB8=0;;HDB4D* C>34B86=C7460<48=C4A02C8>=8=C4A5024BD270B60<445542CB$4GCC745>;;>F8=6CF>0B?42CBF8;;<08=;H148=CA>3D2437>FC>CA0=B;0C4;40A=8=6
2>=C4=C8=C>60<44;4<4=CB0=334B86=2>A460<4<>3D;4B " %!#  ! % % %$ !$)$%

>=24?CB0=3>?4A0C8>=0;?A>24BB4B0A4C74CF><08=;40A=8=62>=C4=CF74=;40A=8=6
033A4BB8=6<>34B)74B420=145DAC74AA458=438=C>30C0;>20C8>=B?A>?4A=>D=B0=3
033A4BB8=6>?4A0C8>=B =60<434B86=0=334E4;>?<4=CC7A44;40A=8=62>=C4=C0A4
CA0=B;0C438=C>60<4<0?B60<4C4A<B0=360<402C8E8C84BA4B?42C8E4;H "$$%#"!!&)!&"&"!%!  #%
)74BC>A0640=3?A>24BB8=6>530C03DA8=6033A4BB8=68=E>;E4BC7A44<08=30C0;>20C8>=B
C74&*?A>24BB>AC74<4<>AH0=3C740DG8;80AH<4<>AH70A338B:8ABC;HC74A40A4
38A42C0=38=38A42CF0HB>50224BB8=630C0&*70B38A42C0224BBC><4<>AH30C00=3=>
38A42C0224BBC>70A338B:30C0)78B8=38A42C0224BBC>70A338B:30C01HC74&*20=>=;H
143>=4F8C7C7474;?>5<4<>AH(42>=3;HC74<4C7>3>530C0BC>A064B7>F=8=86DA4 #4<>AH0=370A338B:30C0BC>A064A4;84B>=05>A<>530C0BC>A06420;;4330C01;>2:B
*=;8:470A338B:B<4<>AH30C0BC>A0640;B>A4@D8A4B05>A<>530C0BC>A064:=>F=0B
?068=60=3B46<4=C0C8>=C01;4B

'$+8ACD0;<4<>AH<0=064<4=C

0B43>=2><?DC4ABCAD2CDA4B0=330C0;>20C8>=BF434B86=4360<4<0?BA4;0C8E4;H
C>60<85HC7401>E4:=>F;436486DA4 A4?A4B4=CBC742>AA4B?>=38=6A4;0C8>=B78?14CF44=

629

C7430C0;>20C8>=B0=360<4<0?B =86DA4 C740A40BF74A460<4270A02C4A<>E4<4=C
0=360<402C8E8C84B>22DA8=C7460<42>AA4B?>=3C>C74;>20C8>=BF74A4C7430C0BC>A064B
>A4G0<?;4C74<4<>AH<0?8=C7460<48BC74<4<>AH;>20C8>=F74A4C7430C0BC>A064B
;B>C7434C08;43:=>F;43642>=C4=CA4@D8A435>AC7438554A4=C30C0;>20C8>=B3DA8=6
033A4BB8=68B<0C2743C>C742>AA4B?>=38=6<0?(?4285820;;H1HC4B0A4A4?A4B4=C431H
24;;B8=0<4<>AH<0?0=3?0640=3B46<4=CC01;4B0A4A4?A4B4=C431H24;;0=3A>FC01;4B
A4B?42C8E4;H =0338C8>=60C4B0A4B4CD?14CF44=38554A4=C60<4<0?B0=3F74=C74
270A02C4A4=C4ABC7460C4>5038554A4=C<0?C78B8=3820C4BC742>AA4B?>=38=630C0BC>A064
;>20C8>=5>AA4038=6'$>AA4B?>=34=2414CF44=30C0;>20C8>=B0=360<4<0?B "$$%#"!!&)!#$"#$!"'!%!&%!" & %
#08=270;;4=64B8=60<434B86=2><45A><CA0=B5>A<8=6C74?A>?4A=>D=B8=%(8=C>C74
2>AA4B?>=38=660<48C4<B =86DA4 C7460<42;D4B0=3?A>?B0A4<0C2743C>C74%(
?A>?4A=>D=B8ABC60<42;D4B)748=8C80;2;D4>5C7460<48BC74;>6820;033A4BBBC>A438=&*B8=24C74;>6820;033A4BB3>4B=>C=443C>1420;2D;0C430BC748=8C80;2>=38C8>=5>A
033A4BB8=6)74B42>=360<42;D48B>1C08=431H0BB86=8=60A>F8=C74?064>AB46<4=C
C01;45A><C74;>6820;033A4BBF78278BA4?A4B4=C431H0?A>?1>GF8C702;D4<0A:4A
(42>=3;H60<4?A>?B22>A38=6C>C74033A4BB8=6?A>24BB05C4AC74?7HB820;033A4BB8B
20;2D;0C43C742>AA42C1;>2:8=<4<>AH=443BC>14A403>>AB8=C7460<4A4?A4B4=CC74
1;>2:0=3C74?>B8C8>=>5C743>>A8=3820C4BC74=D<14A>51;>2:B%=248=B834C743>>AC74
270A02C4A20=B40A275>AB?42858230C08=C70C1;>2:<0?A4?A4B4=C431H0?A>?1>G'$>AA4B?>=34=2414CF44=?A>?4A=>D=B0=360<48C4<B "$$%#"!!&)!$%%!"#$&"!%! &"!%)74CA0=B5>A<0C8>=>5?A>?4A=>D=B8=%(8=C>60<48C4<B8B10B43>=033A4BB8=6?A>24BB
F74A42>A48=5>A<0C8>=01>DCC74>?4A0C8>=0;?A>24BB8=033A4BB8=68B60<858438=C>60<4
2;D4B0=3?A>?B

630





'$>AA4B?>=34=2414CF44=033A4BB8=6>?4A0C8>=B0=360<402C8>=B )74586DA4 B7>FBC70CC74BC0AC>5033A4BB8=6>?4A0C8>=8=C7460<48BA4?A4B4=C43
1HC74270A02C4A0224?C8=60C0B:5A><C74C0B:1>0A300=3>1C08=8=6=4F2;D4BC7A>D6710=32=C4A8=6C74<0?4F74A430C01;>2:8BBC>A438=?A>?1>GC7A>D67C743>>A3
58=38=6C742>AA4B?>=38=6?A>1>G5022>A38=6C>C7420;2D;0C8>=A4BD;CB0=32><?;4C8=6C74
BC>A064>5C748C4<B60=37 #  $%#%) !#"!#% %$ !$)$%!&$
BB7>F=8=86DA4 C7A4460<4<>3D;4BF4A434B86=43 <8BB8>=<>3D;42>=C08=43
2D4BC>74;?BCD34=CBD=34ABC0=3C74B4@D4=24>5BC4?B8=C74033A4BB8=6<>34B 6D834
<>3D;4B2055>;3B?0AC828?0=CBF8C78=BCAD2C8>=B>=7>FC>>?4A0C4C7460<40=3;40A=C74
C0B:.8=4C0;  0=3 5443102:<>3D;4F0BA4B?>=B81;45>A5443102:>=C0B:BC0CDB
0=3270A02C4A?>B8C8>=


'$(HBC4<<>34;B '%! %%"! "'=C74<8BB8>=<>34;F460<85HB8G033A4BB8=6<>34B0=334B86=C74<8=C>B8G<8BB8>=B
BB7>F=8=)01;4 C74B8G<8BB8>=B2>AA4B?>=3C>C74B8G033A4BB8=6<>34B =4027
<8BB8>=C7460<4?A>E834BC742>AA4B?>=38=6>?4A0C8>=0;BC4?B5>AC74B?428582033A4BB8=6
<>34)7486DA4B7>FBC70C0B?428582>?4A0C8>=?A>24BB01>DC<8BB8>= 
8ABCC74?0AC828?0=C64CBC7458ABC2;D45A><C74C0B:1>0A3F78278BC74;>6820;
033A4BBBC>A438=C74&*$4GC?0AC828?0=CBF4A468E4=0B42>=32;D48=C74?064C01;4
?A>?1>GF7827F0BC>20;2D;0C4C74?7HB820;033A4BB1HA4038=6C74?064C01;4 =C74C0B:
E <40=BC70C01;>2:>530C08=<4<>AH8BA403>C74AF8B48C<40=BC70C01;>2:>530C0>=
C7470A338B:=443BC>14A403,74=E C74?0AC828?0=C20;2D;0C4BC74?7HB820;033A4BB
0=3A403B01;>2:>530C08=<4<>AH8=C74=4GCBC4?)74?0AC828?0=CC74=A403BC7430C08=
01;>2:10B43>=C7420;2D;0C43>55B4CF78;4C74270A02C4A>?4=B01>G8=C741;>2:6A0?78=0;;HC74&*64CBC7430C00=3C7460<4C0B:64CB2><?;4C43

631



)01;4 * %%"!%"'&$%%! "%
#8BB8>==D<14A33A4BB8=6<>34B

#8BB8>= %1C08=8=630C08=<4<>AHDB8=6?064C01;4

#8BB8>= %1C08=8=630C08=70A338B:DB8=6?064C01;4F8C7>DCBF0??8=68=

#8BB8>= %1C08=8=630C08=70A338B:DB8=6?064C01;4F8C7BF0??8=68=

#8BB8>= (F0??8=6>DC30C08=70A338B:

#8BB8>= %1C08=8=630C08=70A338B:1HB46<4=CC01;4F8C7>DC4AA>AA4?>AC8=6

#8BB8>=

%1C08=8=630C08=70A338B:1HB46<4=CC01;4F8C74AA>AA4?>AC8=6'$ '4038=630C0?A>24BB8=<4<>AH1H?068=6#%! "'
)78B<>3D;48=2;D34B60<4>?4A0C8>=0=3;40A=8=6<8BB8>=8=BCAD2C8>=0BB7>F=8=86DA4
)7460<4>?4A0C8>=6D830=24<08=;H8=BCAD2CB60<4?0AC828?0=CB7>FC><0=8?D;0C4
270A02C4A1470E8>ABF8C2714CF44=38554A4=C<0?B0=38=C4A02CF8C74;4<4=CB8=C74<0?
)74;40A=8=6<8BB8>=6D83420=C0:4>=C74A>;4>50=8=BCAD2C>A6D838=660<4?0AC828?0=CB>=7>FC>02@D8A4C0B:BB7>F8=6C74<40=8=60=35D=2C8>=>560<44;4<4=CB0=3
BD664BC8=6B>;DC8>=B5>A<8BB8>=2><?;4C8>=$>C01;H;40A=8=6<8BB8>=B0A434B86=43C>
?A>E834B2055>;38=65>A60<4?0AC828?0=CBF8C7C8<4;H6D830=24F7827342A40B4B0BC74H
?A>6A4BBC7A>D67C74<8BB8>=B0=38<?A>E4C748A;40A=8=6018;8C84B,74=60<4?0AC828?0=CB
?4A5>A<C74<8BB8>=5>AC7458ABCC8<4C74H0A4?A>E8343F8C734C08;438=BCAD2C8>=0;
8=5>A<0C8>=BC74=D<14A>560<4B8=2A40B4BC748=BCAD2C8>=0;8=5>A<0C8>=342A40B4B


'$ )74<8BB8>=6D830=24


632



#%!' "'
)78B<>3D;48B34B86=43C>:44?60<4?0AC828?0=CBF8C7C74<8BB8>=BC0CDB?A>E838=6
8=5>A<0C8>=>=60<4;>20C8>=0=32;D4B8=0C8<4;H<0==4ABB7>F=8=86DA4C74
<8BB8>=;>20C8>=8=5>A<0C8>=20=74;?60<4?0AC828?0=CBD=34ABC0=3C74?A>6A4BB270=64B>5
C74<8BB8>=0=35028;8C0C4<0BC4A8=6C742DAA4=C<8BB8>=)74?0AC828?0=C;>20C8>=8=5>A<0C8>=
F78278B0:8=3>52;D48=5>A<0C8>=148=6543102:20=8=2A40B4C74D=34ABC0=38=6>5C74
60<4?0AC828?0=CB>=C74D?30C43BC0CDB>5C7460<42;D4B0=3?A>?B'$ )745443102:8=5>A<0C8>=

 # #%" %$
=4G?4A8<4=CF0B2>=3D2C43C>E4A85HC742>=CA81DC8E4=4BB>5C7434E4;>?4360<48=C4A<B>5;40A=8=645542C8E4=4BB0=3B4;54558202H
C>C0;>5 ?0AC828?0=CB5A><02><?DC4AA4;0C4338B28?;8=40C0!0?0=D=8E4AB8CH
C>>:?0AC8=C744G?4A8<4=C)744G?4A8<4=C?0AC828?0=CBF4A40BB86=43C>0=4G?4A8<4=C0;
0=302>=CA>;6A>D?)74?0AC828?0=CB8=C744G?4A8<4=C0;6A>D?;40A=43C74033A4BB8=6
<>34;DB8=6060<410B43;40A=8=60??A>027F78;4C74?0AC828?0=CB8=C742>=CA>;6A>D?
2><?;4C43C74;40A=8=6C0B:1HA4038=6C74C4GC1>>: $%#& %$)748=BCAD<4=CB8=E>;E438=C744G?4A8<4=C<08=;H8=2;D343<40BDA4B>5;40A=8=6
?4A5>A<0=24B4;54558202H0=32>6=8C8E4;>03)74;40A=8=6?4A5>A<0=24<40BDA4B
2>=B8BC43>5?A4C4BC0=3?>BCC4BC)74?A4C4BC2>=B8BC43>5 <D;C8?;427>8240=3B7>AC
0=BF4A@D4BC8>=B0=38CB<40BDA4F0BC7410B82B>5>?4A0C8=6BHBC4<B)74?>BCC4BC>=
C74>C74A70=32>=B8BC43>5=8=42><?A474=B8E4@D4BC8>=B;8<8C43C>:=>F;4364>5C74
B40A27<>34;>6=8C8E4;>03C4BC8B0F0HC>4G0<8=4C748<?02C>50;40A=8=6BCA0C46H>=
BCD34=CBJ;40A=8=602C8E8C84B/70>4C0;  (4;54558202H0=32>6=8C8E4;>03<40BDA4B
F4A403<8=8BC4A43C7A>D670@D4BC8>==08A4<>3858435A><C74@D4BC8>==08A434E4;>?431H
&8=CA8270=34A>>C 
 #!&#86DA4B7>FBC744G?4A8<4=C?A>243DA445>A4C74;40A=8=602C8E8CHC744G?4A8<4=C0;0=3
2>=CA>;6A>D?BC>>:0 <8=DC4?A4C4BC0=30 <8=DC4?A4@D4BC8>==08A4A4B?42C8E4;H
(D1B4@D4=C;H1>C7>5CF>6A>D?BF4A468E4=0 <8=DC4;40A=8=602C8E8CH0CC74B0<4C8<4
8=F7827C744G?4A8<4=C0;6A>D?DB43C7434B86=4360<4C>;40A=C74033A4BB8=6<>34
F78;4C742>=CA>;6A>D?;40A=431HA4038=6C74C4GC1>>:5C4AF0A30 <8=DC4?>BCC4BC
0=30 <8=DC4?>BC@D4BC8>==08A4F4A40B:43633

'$ G?4A8<4=C?A>243DA4

 # 

5C4A30C0?A>24BB8=6F4>1C08=43E0;8330C05>A ?0AC828?0=CB)>E4A85HC742>=CA81DC8>=>5
C7434B86=4360<48=;40A=8=602784E4<4=CB4;54558202H0=32>6=8C8E4;>038ABC0=>A<0;
38BCA81DC8>=C4BCF0B03>?C43C>0=0;HI4C7A44:8=3B>530C0DB8=6C74(70?8A>,8;:C4BC
(?4285820;;HC74?>BCC4BCB7>F43B86=85820=24F ? 8<?;H8=6C70C8C3>4B=>C
<44CC740BBD<?C8>=>5=>A<0;8CH?  =0338C8>=?A4C4BCF? B4;5
4558202HF ? 0=32>6=8C8E4;>03F ?
383=>C?A4B4=C
B86=85820=24? 8<?;H8=6C70C8C<44CBC740BBD<?C8>=>5=>A<0;8CH(>C70C0CC4BC
F0BDB43C>0=0;HI4F74C74AC74A4F4A438554A4=24B8=?A4C4BCB4;54558202H0=32>6=8C8E4
;>03;B>C74=>=?0A0<4CA82#0==,78C=4H)4BCF0BDB43C>0=0;HI4F74C74AC74A4F0B0
38554A4=248=?>BCC4BC

)01;4 %$#&(%&&%&%!&&%&$%'&%"&#$&%&
+0A801;4
A>D?
$
#40=
(


&A4C4BC
G?4A8<4=C6A>D?




 
>=CA>;6A>D?

 
 
$>C4#  )7460<4;40=8=6<4C7>3>=;40A=8=6?4A5>A<0=24F0B4G0<8=43DB8=6C74#0==
,78C=4HC4BC)74C01;4 B7>FBC70CC74<40=A0=:>5C744G?4A8<4=C0;0=32>=CA>;6A>D?B
F4A4  0=3 A4B?42C8E4;H0=3C74BD<>5A0=:BF4A4   0=3  
0B43>=#0==,78C=4HA4BD;CBI ? 8=3820C4C70CBCD34=CB8=C74
4G?4A8<4=C0;6A>D?F7>DB43C7460<4;40A=8=60??A>027C>;40A=033A4BB8=6<>34;383=>C
38554AB86=85820=C;H5A><C742>=CA>;6A>D?BCD34=CB8=C4A<B>5;40A=8=6?4A5>A<0=24
)01;4 %$#&(%&&%&%!!!&!+&%&$%'&%"&#"%&&%&
+0A801;4
&A4C4BC


A>D?
G?4A8<4=C6A>D?
>=CA>;6A>D?$

#40='0=:
 


(D<>5'0=:B   

,








$>C4 #  )74CC4BCF0BDB43C>4G0<8=4C7438554A4=24B8=B4;54558202H14CF44=C74
4G?4A8<4=C0;0=32>=CA>;6A>D?B)74C01;4 B7>FBC70CC74#40=F0B  0=3  5>A
C744G?4A8<4=C0;0=32>=CA>;6A>D?B0=3C74(F0B  0=3  A4B?42C8E4;H)74C
C4BCA4BD;CBC 
? B7>FC70CBCD34=CB8=C744G?4A8<4=C0;0=32>=CA>;6A>D?B
F4A4BC0C8BC820;;H38554A4=C8=C4A<B>5B4;54558202H =>C74AF>A3BBCD34=CBF7>DB43C74
60<4;40A=8=6033A4BB8=6<>34;70378674AB4;54558202HC70=70378674AB4;54558202H
)01;4 %$#&(%&&%&%!&&%&$%'&%"&%+634

+0A801;4
A>D?
$
#40=
(

&A4C4BC
G?4A8<4=C6A>D?

 
 


>=CA>;6A>D?

 
 
$>C4#  22>A38=6C>C01;4 B7>FBC70CC744G?4A8<4=C0;0=32>=CA>;6A>D?B8=#40=F4A4 0=3  A4B?42C8E4;H0=3C74(F0B  0=3 )74BC0C8BC820;A4BD;CBC
? B7>FC70CC74BCD34=CB8=C744G?4A8<4=C0;6A>D?F8C7C7460<4;40A=8=6
033A4BB8=6<>34703B8<8;0A2>6=8C8E4;>030BC74BCD34=CBF8C7C74CA038C8>=0;0??A>027
F78278=3820C4BC70CC7460<4;40A=8=60??A>027383=>C1A8=6<>A42>6=8C8E4BCA4BBC>
2>=CA>;6A>D?BCD34=CB)01;4 %$#&(%&&%&%!&&%&$%'&%"&"!&("+0A801;4
&A4C4BC

$>C4# A>D?
G?4A8<4=C6A>D?
>=CA>;6A>D?

$



#40=
 
 

(
 


&






  
=C74BCD3H060<4F0B34E4;>?43C>B>;E4C74;>FB4;54558202H8BBD4>5C74BCD34=CB8=C74
;40A=8=602C8E8CH1H2><18=8=6C74;40A=8=62>=C4=C0=3C7460<4BC4?BF8C7C7434B86=
2>=24?C>560<4;40A=8=64B834BC7460<4F0B34B86=431HC0:8=6C74033A4BB8=6<>34
>5C74>?4A0C8=6BHBC4<2>DAB40BC74;40A=8=62>=C4=C8=C7A44F0HBDB8=6C74C7A4430C0
;>20C8>=B>570A338B:<4<>AH0=3&*0BC7460<4<0?CA0=B5>A<8=6C74?4A=>D=BBD27
0B1;>2:30C00=3>55B4C8=C>>=4C>>=460<44;4<4=CB2>=E4AC8=6C74>?4A0C8>=1470E8>AB
8=C74033A4BB8=6<>348=C>60<4>?4A0C8>=1470E8>AB;B>C7460<48B34B86=43F8C7C7A44
<>3D;4BC70C2>=C08=;40A=8=6BCA0C4684B)74H0A4C74<8BB8>=<>3D;4F7827?A>E834B2D4B
C745443102:<>3D;4F7827?A>E834BC8?B0=3C746D830=24<>3D;4F782770B0B8<8;0A
5D=2C8>=BF8C768E8=6C8?B )>4G0<8=4F74C74AC7434E4;>?4360<420=B>;E4C74?A>1;4<>5;>FB4;54558202H
4G7818C431HC74BCD34=CB8=C74;40A=8=602C8E8C84B02><?0A8B>=4G?4A8<4=CF0B2>=3D2C43
0B43>=C74A4BD;CB>5C744G?4A8<4=C8=C7A4438<4=B8>=B;40A=8=602784E4<4=CB4;5
4558202H0=32>6=8C8E4;>03C7460<4F0BBC0C8BC820;;H0=0;HI435>A0E>838=6;>FB4;54558202H
1470E8>A8ABC022>A38=6C>C74A4BD;CB>5C74CC4BCC74BCD34=CB8=C744G?4A8<4=C0;6A>D?
F7>DB4360<4;40A=8=6383=>C38554AB86=85820=C;H5A><C74BCD34=CB8=C742>=CA>;6A>D?
F7>DB43CA038C8>=0;;40A=8=68=C4A<B>5;40A=8=602784E4<4=C(42>=3F8C7B8<8;0A;40A=8=6
02784E4<4=CBCD34=CB8=C744G?4A8<4=C0;6A>D?38554A43B86=85820=C;H5A><BCD34=CB8=C74
2>=CA>;6A>D?8=C4A<B>5B4;54558202H0=3BCD34=CB8=C744G?4A8<4=C0;6A>D?70378674A
B4;54558202HF78278<?;843C70CC7434E4;>?4360<42>D;38<?A>E4B4;54558202H =0338C8>=
)C4BCF0B03>?C43C>C4BCF74C74AC748=CA>3D2C8>=>5C7460<4;40A=8=6<4C7>31A>D67C
01>DC2>6=8C8E4;>03>=BCD34=CB0B43>=C74A4BD;CBB7>F43C70CC74A4F0B=>B86=85820=C
38554A4=2414CF44=C744G?4A8<4=C0;0=32>=CA>;6A>D?B8=C4A<B>52>6=8C8E4;>03)78B
8=3820C4BC70CC7460<410B43;40A=8=6BCA0C46H383=>C1A8=601>DC0338C8>=0;2>6=8C8E4;>03"

)78BA4B40A27F0B?0AC80;;HBD??>AC431HA0=CB8=835>A(284=C8582'4B40A27$>
 5A><C74#8=8BCAH>53D20C8>=D;CDA4(?>ACB(284=240=3)427=>;>6H#-)8=!0?0=



635



:/AE/2,"/2=:<G 3A75<7<54=@5/;30/A32:3/@<7<5'6334431B7D37<B35@/B7=<=4
B316<=:=5GB=AC>>=@B:3/@<7<5=C@</:=42C1/B7=</:'316<=:=5G&GAB3;A   /:97A!
1/23;7134471/1G/A/;327/B=@/<2;=23@/B=@D/@7/0:37<B63@3:/B7=<A67>
03BE33</1/23;71>@=1@/AB7</B7=</<2/1/23;71/1673D3;3<BC@/A7/<=C@</:=42C1/B7=</:
%3A3/@16  3 /<2C@/  &3:434471/1G<)&%/;/16/C2@/<2<1G1:=>327/=46C;/<036/D7=@
)=: >>  "3E+=@91/23;71$@3AA
=:/</97AD/<53:/97A:/D/A=BA7A'
&3>B3;03@'3/167<5B63
/22@3AA7<5;=23A=4B63! $(0G;3/<A=4/>@/1B71/0:3:3AA=<< !
!   !!  "! #$ "!
! > M  
/D7A!!
<3F>:=@/B7=<=44/1B=@A/4431B7<5B63/1/23;71AC113AA=4ABC23<BA7</1=::353
?C/<B7B/B7D30CA7<3AA1=C@A3:=@72/B:/<B71(<7D3@A7BG%3B@73D324@=;&#6=AB3@71
2/B/0/A36BB>5/B3E/G>@=?C3AB1=;=>3<C@:C@:-D3@ O , 
@4B-D/:-4;B O 7<4==474;B93D;BF27AA3@B/B7=<@3A-2/B O F@7>?27AA@4B-2/B O F@7>?27AA 
=;3A!3<23A
&3>B3;03@ 3/@<7<5B=>@=5@/;274471C:B73A/<2A=:CB7=<A<<B3@</B7=</:=<43@3<13=<<57<33@7<52C1/B7=<N)=:
3E/@2* 
F13>B7=</:67:2@3<B632<(>>3@&/22:3%7D3@"!3@@7::$@3<B713/::/< $:/AA@C13=;3@6/@:3A7<H3@=C<2/B7=<A=4/;3/A32 3/@<7<5
2C1/B7=</:$AG16=:=57AB     #   
  

 
!/22CF=AA3:7<'
&3:434471/1G<!% 3/@G$'/<5<3G2A/<20==9=4A3:4/<2723<B7BG>> N
'63C7:4=@2$@3AA
!/@B7<!/@A6*3/@=44/7:C@3@73<2=@4=3CAB@/:7/<$AG16=:=57AB   N
!=@3<=3@$C@5=A!/@BJ<3H#@B7H&73@@/ 3@<I<23H!/<8K<

2C1/B7=</:5/;323A75<4=@=<:7<332C1/B7=<=;>CB3@A7<C;/<36/D7=@     
&16C<9 &3:44471/1G/<22C1/B7=</<2<AB@C1B7=<<!/22CF32A&3:44471/1G
2/>B/B7=</<228CAB;3<B'63$:3<C;&3@73A7<&=17/::7<71/:$AG16=:=5G&>@7<53@=AB=<!6BB>A2=7=@5  - &6/0/:7</#!=H3:7CA$!/::7/@/97A'=;=A/:/<#:/193G3@9CA63<9=

#1B=03@=;07<7<55/;34:=E/<2:3/@<7<5=0831B7D3A7<32C1/B7=</:5/;3A<B6
C@=>3/<=<43@3<13=</;3A0/A32 3/@<7<5&67+%&676 /;34/1B=@A/<25/;30/A32:3/@<7<523A75<;=23:<B3@</B7=</:
=C@</:=4=;>CB3@/;3A'316<=:=5G
  &7:D3@;/<& /A/HH/!
 3/@<7<5/<223D3:=>;3<B!/97<51=<<31B7=<AB=3<6/<13
B3/167<5&/<@/<17A1==AA3G/AA'=07/A&:3B163@*7<2$/;30/A32:3/@<7<5/<20==9=4@3A3/@16=<
32C1/B7=</:1=;;C<71/B7=<A/<2B316<=:=5G  *37</$7<5+&6C/7 
%3A3/@16=<B63@3:/B32B3/167<5;3B6=2=41=;>CB3@
=>3@/B7<5AGAB3;1=C@A3<3 3/@<7<532C1/B7=</<2#<:7<3'@/7<7<5 B6<B3@</B7=</:=<43@3<133 #'
&6/<56/767</>@7:N
$@=13327<5A >>   
&>@7<53@<B3@</B7=</:$C0:7A67<5
*3ABE==2$&=;;=<A3<A3;3B6=2A4=@167:2@3<E7B6A>317/:32C1/B7=</:<332A B6
32< =<2=<%=CB:3253/:;3@
+7<&=<5+'/0/B/+#5/B/E/<5
3D3:=>7<5/<27;>:3;3<B7<5/
4@/;3E=@9=4>/@B717>/B=@GA7;C:/B7=<4=@;=07:3:3/@<7<5CA7<5A1/44=:27<5=C@</:=42C1/B7=</:
'316<=:=5G&=173BG     ,/8/1=D/ G<16&!A>3<A6/23'
&3:434471/1GAB@3AA/<2/1/23;71AC113AA
7<1=::353 "! ,6/=E/<5+7<
@3AC:B1=<47@;/B7=<0/A32:3/@<7<5036/D7=@/</:GA7A
4@/;3E=@94=@3F>:=@7<5B6367223<@3/A=<A0367<2>/BB3@<A/<2AB@/B3573A2C1/B7=</:
'316<=:=5G&=173BG     
,6/= 7C,,6=C+7<
 3/@<7<5</:GB71A@/;3E=@9/A32=<C;/<3<B3@32@B74717/:<B3::753<134=@23<B74G7<5B63#>B7;/: 3/@<7<5&B@/B35GB=<B3@D3<37<
3/@<7<536/D7=@ "!$!$ 
 




636

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in Education.
Asia-Pacific Society for Computers in Education

Investigating Cognitive Biases in
Self-Explanation Behaviors during
Game-based Learning about Mathematics
J.M. Alexandra ANDRESa*, Elizabeth B. CLOUDEa, Ryan S. BAKERa & Seiyon LEEaa
Graduate School of Education, University of Pennsylvania, USA*aandres@upenn.edu
Abstract: The study investigates the impact of cognitive biases on middle-school
students' affective experiences while learning about math in a game-based learning
environment (GBLE). The study focused on students' confrustion, an affect constructthat unifies the several manifestations of confusion and frustration. We studied
confrustion in the context of students self-explaining erroneous examples, where they
had to find and fix common errors in given math problems and self-explain theirproblem-solving processes either with or without scaffolding. Text replays were utilized
to examine student interactions during game-based learning and identify behaviors
that emerged in response to cognitive biases and affect and its impact on learning andperformance outcomes. The results revealed that students who demonstrated morepseudo-confidence in their self-explanations had higher self-reported self-efficacy, butwere more likely to submit incongruent responses, exhibit confrustion, make errors,
and take longer to finish the game. Overall, the findings show that students werevulnerable to cognitive biases and did not always respond in ways that accuratelyreflected their approach to solving math problems. The insights into how students
approach and learn from math games inform the design and implementation of GBLEs
by addressing cognitive biases.
Keywords: game-based learning,self-explanation, self-efficacy

confrustion,

cognitive

bias,

confidence,

1. Introduction
Game elements designed to scaffold instructional support are essential for promoting
behaviors that support mathematical learning (e.g., prompting students to generate
self-explanations about their solutions; Ke et al., 2019; Wouters & van Oostendorp, 2013). A
study by McLaren and others (2022) showed that students who generated open-ended andfocused self-explanations (i.e., open-ended responses answering a focused prompt),
compared to menu-based self-explanations (i.e., select an explanation from a menu), hadsignificantly higher learning. However, in further studies, Johnson and Mayer (2010) found nodifferences in learning outcomes between open-ended and menu-based self-explanation
types.
To better understand the impact of each type of prompt, it is important to explore whichlearning behaviors, processes, and strategies that emerge during open-ended and focused
self-explanations contribute to actively constructing knowledge. One factor impacting
successful math learning and achievement is the student’s confidence or self-efficacy (Pan etal., 2022). However, students’ judgments of their abilities are vulnerable to cognitive biases,
possibly skewing their perceptions.
Cognitive biases stem from heuristic strategies, or decisional short cuts
(Blumenthal-Barby & Krieger, 2015), to reduce the amount of information considered when
making a decision (Schwenk, 1986). The emergence of biases is mediated by contextual
factors such as attributed risk (Halpern, 1989), age and expertise (Forbes, 2005), and time
constraints (Lehner et al., 1997), and these biases can impact a student’s confidence, interest,637

and achievement in math. For example, when students have high confidence but low
knowledge (Pseudo-Confidence), they tend to experience a higher occurrence of confusionand frustration during learning (Di Leo et al., 2019).
Previous studies have also suggested that low-knowledge learners may attempt toalleviate confusion by avoiding activities that require deep thought (Rodrigo et al., 2007); forinstance, gaming the system, where a student exploits a platform’s help/feedback systems to
succeed instead of attempting to learn (Baker et al., 2006), and Without Thinking Fastidiously(WTF) behaviors, where a student engages in activities that are unrelated to the learning goal
(Wixon et al., 2012).
In this paper, we extend previous findings by examining pseudo-confident refusal, acognitive bias where a student refuses to self-explain and instead asserts their correctness
despite system feedback indicating otherwise. Specifically, the current study seeks to identify
and examine the role of pseudo-confidence during self-explanations on students’ affect,learning, and performance within a game-based learning environment (GBLE) called Decimal
Point. Focused and open-ended prompts were used to elicit students’ problem-solving
processes in math and identify learning behaviors that emerged during knowledge
construction. Through the use of qualitatively coded text logs of student interactions, thecurrent study investigated the influence of pseudo-confidence, on students’ affect,
performance, self-efficacy, as well as interaction with the platform.

2. Methods
Decimal Point
The platform used in this study was Decimal Point, a GBLE that is based on an amusementpark metaphor. Players were guided through different sets of mini games built within the
Cognitive Tutor Authoring Tools (CTAT; Aleven et al., 2009) to learn about decimals. The
students played a total of 24 mini games during regular class periods (across an average of
3-5 days) which lasted around 45 to 55 minutes each.
In the current study, players used the erroneous examples play mode, where they
were required to find and fix common errors in given math problems and then self-explain theirproblem-solving process either with or without scaffolding (“Think about how you came up withthis answer, and drag the correct option to the following blank(s).” versus “How do you know?”,
respectively). The self-explanation prompts were designed to actively engage a student todevelop a deeper understanding of critical concepts in mathematics. Each student was
randomly assigned to one of three self-explanation conditions: menu-based, scaffolded, or
focused format. In this paper, we study the focused, open-ended, self-explanations, which
prompted students to explain their response with minimal guidance.

Sample and Student Level Metrics
Data were collected from a middle school in Pennsylvania between March and November
2021. Three hundred and fifty-eight (n = 358; 44% females) students participated in the study;
however, only 85 were retained for the current set of analyses due to data completenessacross variables of interest. Students participated in a survey before and after they played
Decimal Point. The survey measured Decimal Efficacy, which indicates a student’s level ofself-efficacy or their confidence in using decimal operations. The survey was adapted fromliterature (Pintrich et al. 1993) to better align with the context of the learning session, as usedin Hou et al. (2020). Students were asked to rate statements on a 5-point Likert scale. We
averaged three items to compute the Decimal Efficacy score: 1) I can do an excellent job on
decimal number math assignments, 2) I can understand the most difficult material presented
in decimal number lessons, and 3) I can master the skills being taught in decimal number
lessons.
Students’ interaction data were examined using text replays (Baker et al., 2006), whichdisplay interaction logs from student platform usage in an easy-to-read form. Text replays

638

were then investigated to identify potential behaviors that emerged from cognitive biases andexperiences of confrustion (when a student is either confused or frustrated; Mogessie et al.,2020). This method has been used in previous studies to label a variety of student variables(Baker et al. 2006; DiCerbo and Kidwai, 2013; Zhang et al., 2022) quickly and with highinter-rater reliability. For this study, analyses were focused on self-explanation behaviors that
occurred within the problem-solving process. As such, the log data were divided into clips at
the level of entire problems.
Construct Operationalization and Qualitative Coding
The team first analyzed student responses for indicators of self-explanation behaviors
using qualitative categories. This approach followed the recursive, iterative process used in
Weston et al. (2001), including reviewing literature on previous work using the Decimal Point
platform (Forlizzi et al., 2014) and student engagement behaviors within GBLEs (Zhang et al.,2022). Using grounded theory (Charmaz, 1983), we identified common behaviors that wereindicative of 1) struggles involving self-explanation from the current mathematical content and2) were indicative of difficulty generating accurate responses for the given problem.
Following the process used in (Zhang et al., 2022), two coders (1st and 4th authors)
coded a set of 900 clips together, identified four behaviors occurring during self-explanations(see details in Table 1), outlined the criteria for each indicator, and created a rubric. The
coding manual was reviewed and discussed by the research team to ensure a sharedunderstanding of the criteria and constructs being studied. This process was repeated until thewhole team had a mutual understanding of the codebook's criteria and constructs.
Table 1. Behaviors coded through Text Replays
Behavior

Definition

Confrustionk = .68
Without Thinking
Fastidiously(WTF) k = .89
Incongruent
Response
k = .62

The learner explicitly mentions they do not know what to do or how to answer the
problem and makes repeated answer submissions that are incorrect.
The learner does not acknowledge the prompts for finding a solution and instead
inputs text unrelated to the task at hand.
(e.g., “F*** off,” “Stop asking me,” “Ohmygahd”
The learner inputs correct information, but it is phrased differently from responses
the system can recognize. These responses often involve minimal changes
between response submissions where changes do not semantically change theanswer; the student is trying to find a way to phrase the answer so that the system
will accept it.
The learner refuses to explain their answers or mathematical process and insteadrefers to their inherent capacity to understand the mathematical problems by citing
themselves as an authority to know the final answer.
(e.g., “I know because it’s the right answer,” “Because I’m smart”)

Pseudo-Confident
Refusalk = .79

3. Results
Spearman correlations with Benjamini-Hochberg post-hoc controls were computed to findassociations between pseudo-confidence, performance metrics, and decimal efficacy survey
responses. Students who demonstrated more pseudo-confidence in their self-explanations
were more likely to submit incongruent responses (r = 0.29, adj. α = 0.01, p = 0.008) andexhibit confrustion (r = 0.33, adj. α = 0.007, p = 0.002). In addition, pseudo-confidence inself-explanations were moderately and positively associated with making errors (r = 0.25, adj.
α = 0.02, p = 0.02) and taking longer to finish the game (r = 0.37, adj. α = 0.003, p = 0.004).
Students who demonstrated more pseudo-confidence in their self-explanations tended toscore higher on Post-test Decimal Efficacy surveys (r = 0.25, adj. α = 0.025, p = 0.023) and
had increases in the normalized change in self-efficacy from before to after learning with thegame (r = 0.24, adj. α = 0.028, p = 0.028). These results implyed that pseudo-confident
responses may emerge in response to challenges responding to self-explanation prompts.
Experiences of confrustion were related to increased errors and time spent problem solving,and in response, students may turn to pseudo-confident responses to avoid further engaging

639

with the material. What is interesting to note is that pseudo-confident students were also more
likely to positively estimate their decimal self-efficacy. This relationship may point to atendency of pseudo-confident students to overestimate their abilities despite generating
multiple errors and needing more time to solve math problems.

1

0.06

-0.53

-0.47

-0.54

-0.25

-0.26

-0.26

Change inLearning

Change inDel. Post

Change inDec. Eff.

0.25
0.35

0.24

0.25

Del. Post

Dec. Effi.
(post)

0.37
0.43

Post-test

Dec. Eff.
(pre)

0.25
0.59

Pre-test

Time

0.29

Errors

0.33
1

WTF

1

Incon.
Resp.

Confru.

Pseudo-Confident
Confrustion
Incongruent
Response
WTF
ErrorsTime
problem-solving
Decimal Efficiency(pre)Decimal Efficiency(post)
Change in
Decimal Efficacy
Pre-test
Post-test
Delayed Post-test
Change in
Delayed Post-test
Change inLearning

PseudoConf

Table 2. Correlations for behaviors, achievement metrics, and survey level responses

1
1

0.67

0.31

0.28

-0.68

-0.72

-0.72

1

0.32

0.27

-0.55

-0.52

-0.49

1

0.47

-0.25

-0.26

-0.31

0.83
1

0.76
0.84
1

1

0.75

-0.29

-0.26

1
1

0.41
0.69

0.58
0.43

1

0.59
1

A linear regression predicting the rate of pseudo-confident responses using theinteraction between time spent problem-solving and pre-test scores of achievement revealed
that increased pre-test scores positively predicted pseudo-confidence (β = .02, p = .022),
suggesting that students who were more familiar with the content were more likely to assert
their perceived correctness and refuse to explain. A significant interaction was also observed
between pre-test and time: students with higher pre-test scores who spent a longer amount oftime problem solving also applied more pseudo-confident responses (β = .28, p = .018).
Table 3. Results for linear regressions predicting achievement and Decimal Efficacy (N = 85, p< .05). Significant results shown in bold.

β
-0.07
-0.13
-0.48

P
<0.0010.02<0.001

β
-0.03
-0.05
-0.25

p
0.001
0.43
0.03

Decimal Efficacy(post-test)
βp
0.01<0.001
0.15
0.56
0.20
0.11

0.21

0.02

0.08

0.43

-0.02

Post-test(Intercept)
Pseudo-confidence
TimeInteraction(Pseudo-confidence* Time)2R

0.29

Learning Gains

0.07

0.08

0.87

Decimal Efficacy(Gains)
β
P
-0.01
0.22
0.09
0.95
0.08
0.65
0.02

0.81

0.02

Table 3 shows that across both achievement metrics (post-test and learning gains), a student
who spent more time problem solving was related to decreases in their math performance.
Students who gave more pseudo-confident responses were less likely to perform well on the
post-test math assessment. A significant interaction was observed between time and the rateof pseudo-confident behaviors for predicting post-test. There is a complicated relationshiphere -- a simple slopes analysis revealed a significant relationship where pseudo-confident
students who spent less time were less likely to have increases in their post-test scores.
However, pseudo-confident students who spent more time did better.

640

4. Discussion
This study examined pseudo-confidence, a cognitive bias impacting math learning, and its
relation to affect (e.g., confrustion) and other learning behaviors (e.g., WTF, incongruent
responses) during open-ended and focused self-explanations within a GBLE. We foundpositive associations between pseudo-confident responses with time spent problem solving
and errors made during game-based learning about math. The positive correlation between
these variables indicated a potential struggle self-explaining problem solving, where students
may defer to heuristics to complete the activity. Additionally, students may demonstrate
pseudo-confident responses as a means of self-presentation and impression management
around math difficulties (Schwenk, 1986).
We also found that higher pseudo-confidence was associated with more confrustion
and incongruent responses during game-based learning. These relationships can be
explained by the student failing to actively construct their knowledge when self-explaining,
contributing to more cognitive incongruencies (i.e., confrustion) or more incongruent
responses (Di Leo et al., 2020). If a student made multiple errors during problem solving, but
continued to refuse to explain their problem-solving approach (pseudo-confident), this may
result in more confrustion in the long term. The results also showed that the morepseudo-confident a student was, the higher their self-reported Decimal Efficacy was aftergame-based learning, indicating that responding in this fashion may increase student efficacy
despite lower knowledge.
Next, we examined whether pre-test scores and time spent in the game impacted therate of pseudo-confidence in self-explanations. The results showed that students with higher
pre-test scores also had higher rates of pseudo-confidence in their self-explanations. Thisfinding suggested that students who had more prior knowledge of math topics demonstrated
more pseudo-confidence. Students with higher prior knowledge may be vulnerable to theDunning-Kruger effect; they may be knowledgeable enough to be confident in their responses,
but unable to detect blind spots in their knowledge.
Lastly, linear regression analyses revealed that the more time spent in the game andpseudo-confident responses made, the worse students performed on the post-test
assessment. A simple slopes analysis revealed that pseudo-confident students who spend
less time problem solving performed worse on the post-test assessment. These findings mayalign with previous research which suggest that students tend to make less effective decisions
when required to solve problems or make decisions with less time (Lehner et al., 1997).
However, pseudo-confident students who spent more time performed better on the post-test,
possibly suggesting that this subset of students was still learning.
This study used a mixed-methods approach to investigate pseudo-confidence whilestudents explained their approach to solving math problems during game-based learning.
While self-report data outside the context of the activity was valuable for measuring global
confidence in math, this approach presents challenges for studying the role of confidence
during math learning. Confidence is a dynamic construct, one that changes during a learning
session based on a student’s goals and in response to various types of feedback they may
receive (e.g., time left or confrustion; Pavlas, 2010). Thus, real-time measurement ofconfidence is important for advancing our understanding on if, when, and how confidence
evolves during game-based learning about math. Future research should evaluate the role ofother common cognitive biases that impact math learning, including examining the presenceof stereotype threat or the hard-easy effect (overconfidence on harder problems) during mathlearning. For example, to what extent does the rate of pseudo-confidence change depending
on how difficult the math problems are? This research has implications for identifying key
learning behaviors and building adaptive self-explanation prompts and interventions thatmaximize students actively constructing knowledge about math with GBLEs.

References
Aleven, V., Mclaren, B. M., Sewall, J., & Koedinger, K. R. (2009). A new paradigm for intelligent tutoring
systems: Example-tracing tutors. Int’l Journal of AI in Education, 19(2), 105-154.

641

Baker, R.S.J.d., Corbett, A.T., Wagner, A.Z. (2006) Human Classification of Low-Fidelity Replays ofStudent Actions. Proc. of the Educational Data Mining Workshop at ITS2006, 29-36.
Baker, R.S.J.d., D'Mello, S.K., Rodrigo, M.M.T., Graesser, A.C. (2010) Better to be frustrated thanbored: The incidence, persistence, and impact of learners' cognitive-affective states during
interactions with three different computer-based learning environments. Int’l Journal of
Human-Computer Studies, 68 (4), 223-241.
Blumenthal-Barby, J. S., & Krieger, H. (2015). Cognitive biases and heuristics in medical decision
making: A crit. rev. using a systematic search strategy. Med. Decision Making, 35(4), 539-557.
Charmaz, K. (1983). The grounded theory method: An explication and interpretation. Contemporary
Field Research, R. Emerson, Eds. Boston, MA: Little, Brown and Company. 109–126.
Di Leo, I., Muis, K. R., Singh, C. A., & Psaradellis, C. (2019). Curiosity… confusion? Frustration! Therole and sequencing of emotions during mathematics problem solving. Contemporary
Educational Psychology, 58, 121-137.
Dunning, D. (2011). The Dunning–Kruger effect: On being ignorant of one's own ignorance. In
Advances in Experimental Social Psychology (Vol. 44, pp. 247-296). Academic Press.
Forbes, D. P. (2005). Are some entrepreneurs more overconfident than others?. Journal of Business
Venturing, 20(5), 623-640.
Forlizzi, J., McLaren, B. M., Ganoe, C., McLaren, P. B., Kihumba, G., & Lister, K. (2014,). Decimal point:
Designing and developing an educational game to teach decimals to middle school students.
In 8th Euro. Conf. on Games-Based Learning: (pp. 128-135).
Halpern, J. J. (1989). Cognitive factors influencing decision making in a highly reliableorganization. Industrial Crisis Quarterly, 3(2), 143-158.
Hou, X., Nguyen, H. A., Richey, J. E., & McLaren, B. M. (2020). Exploring how gender and enjoyment
impact learning in a digital learning game. Proc. Artificial Intelligence and Education, 255-268.
Kahneman, D., & Tversky, A. (2000). Choices, Values, and Frames. NY; Cambridge
Karumbaiah, S., Baker, R.S., Tao, Y., Liu, Z. (2022). How does students' affect in virtual learning relate
to their outcomes? A systematic rev. challenging the positive-negative dichotomy. In Proc. of
the 12th International Conference on Learning Analytics & Knowledge, 24-33.
Ke, F., M. Clark, K., & Uysal, S. (2019). Architecture game-based mathematical learning by making.
International Journal of Science and Mathematics Education, 17, 167-184.
Lehner, P., Seyed-Solorforough, M. M., O'Connor, M. F., Sak, S., & Mullin, T. (1997). Cognitive biases
and time stress in team decision making. IEEE Transactions on Systems, Man, andCybernetics-Part A: Systems and Humans, 27(5), 698-703.
McLaren, B. M., Nguyen, H. A., Richey, J. E., & Mogessie, M. (2022). Focused self-explanations lead tothe best learning outcomes in a digital learning game. In Proc. of the 16th International
Conference on the Learning Sciences (pp. 1229-1232).
McNamara, D. S., Jackson, G. T., & Graesser, A. (2010). Intelligent tutoring and games (ITaG). In
Gaming for Classroom-Based Learning: Digital Role Playing as A Motivator of Study, 44-65.
Mogessie, M., Elizabeth R., J., McLaren, B. M., Andres-Bray, J. M. L., & Baker, R. S. (2020).
Confrustion and gaming while learning with erroneous examples in a decimals game. In
Proceedings of the International Conference on Artificial Intelligence in Education, 208-213.
Pan, Y., Ke, F., & Xu, X. (2022). A systematic review of the role of learning games in fostering
mathematics education in K-12 settings. Educational Research Review, 100448.
Pekrun, R., & Stephens, E. J. (2012). Academic emotions, APA Educ. Psych. Handbook: Individual
Differences & Cultural and Contextual Factors, Vol. 2, 3–31.
Richey, J. E., McLaren, B. M., Andres-Bray, M., Mogessie, M., Scruggs, R., Baker, R., & Star, J. (2019).
Confrustion in learning from erroneous examples: does type of prompted self-explanation maketha difference?. In Proc. of the 20 Int’l Conf. on Artificial Intelligence in Education, pp. 445-457.
Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S.,
Santillano, J.Q., …& Viehland, N.J.B. (2007). Affect and usage choices in simulation problem
solving environments, Proc. of the 13th Int’l Conf. on AI in Education, pp. 145–152.
Sanchez, C., & Dunning, D. (2018). Overconfidence among beginners: Is a little learning a dangerous
thing?. Journal of Personality and Social Psychology, 114(1), 10.
Schwenk, C. H. (1986). Information, cognitive biases, and commitment to a course of action. Academyof Management Review, 11(2), 298-310.
Weston, C., Gandell, T., Beauchamp, J., McAlpine, L., Wiseman, C., & Beauchamp, C. 2001. Analyzing
interview data: the development and evolution of a coding system. Qual. Soc., 24(3), 381–400.
Zhang, J., Andres, J. M. A. L., Hutt, S., Baker, R. S., Ocumpaugh, J., Nasiar, N., ... & Young, T. (2022).
Using machine learning to detect SMART model cognitive operations in mathematicalproblem-solving process. Journal of Educational Data Mining, 14(3), 76-108.

642

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

GaMINLab - Meaningful gamification to
engage students in science inquiry
practices through simulation labs
Archana RANEab*, Sahana MURTHYa & Sasikumar Mb.a
Indian Institute of Technology, Bombay, Indiab
CDAC, Mumbai, India*archana.rane@iitb.ac.in
Abstract: Practical labs in science are expected to foster disciplinary practices of“doing science” in learners. Simulation-based labs which provide the convenience ofconducting practical lab experiments online however, come with their own set ofchallenges such as steep learning curves, learner working in isolation, lack ofpersonalization which may adversely affect learner engagement and motivation. OLabs
is a set of simulation-based labs for schools widely used in India. The main goal of thiswork is to redesign simulation-based labs like OLabs to engage learners in science
practices and to motivate learners to engage in science practices. Meaningful
gamification is one of approaches that focuses on intrinsically motivating learners tofind meaning in the given learning context. GaMINLab, which is this proposed redesignof simulation-based labs like OLabs, was informed by literature recommendations forfostering disciplinary practices, addressing challenges in simulation-based labs, andmeaningful gamification. We discuss the design features of GaMINLab.
Keywords: Simulation-based labs, meaningful gamification, science disciplinary
practices, inquiry labs, scientific inquiry

1. Introduction
Practical labs, are expected to help learners learn science, learn about science and learn todo science. In recent years, the extensive emergence of simulation-based labs has facilitatedlearners to do virtual experiments anytime, anywhere, and any number of times. These labs
also offer possibilities to illustrate concepts beyond the boundaries of traditional labs, forexample illustrating magnetic field intensity, lifecycle of mosquito, etc. Despite world-wide
proliferation and advantages, there are reported challenges such as learner isolation, steep
learning curves, lack of suitable guidance, lack of personalization, having cookbook-based
exercises, etc. The challenges may lead to frustration among learners, resulting in early
dropout or loss of motivation and engagement. Thus, it is critical to give adequate attention to
engage and motivate learners to do the underlying lab related activities.
OLabs is a significant initiative by the government of India to address concerns such aslack of basic lab infrastructure, limited practice opportunities, etc and supplement physical labs
(M Sasikumar, 2016). OLabs, which is a set of simulation-based labs in various subjects at
school level, has been adopted by thousands of state schools in India and lakhs of users
access it on a daily basis. In its current form there are no tasks/problems built-in to engage
learners in science disciplinary practices. The goal of this work is to re-design simulationbased labs like OLabs to a) provide opportunities for engaging in science disciplinary practicesb) engage and motivate learners as they engage in these practices. This re-design, called
GaMINLab (Gamification (Meaningful) in Inquiry Labs) is informed by literature
recommendations for a) fostering disciplinary practices, b) addressing challenges insimulation-based labs, and theory of meaningful gamification. In this paper, we discuss
GaMINLab design features.

643

2. Literature & Theory
Simulation-based labs facilitate conducting experiments online, overcoming the geographicaland time limitations of traditional labs. To tackle challenges like steep learning curves,
isolation, lack of personalization, design guidelines are recommended in literature a) providing
investigation opportunities beyond the classroom, b) posing driving questions to focus
learners' exploration, c) implicit scaffolding, d) encouraging reflection on findings, and e)
promoting peer interaction and engaging in collaborative work (Moore et al., 2013).
Science education is increasingly focusing on involving learners in disciplinary practices,
as emphasized in global curriculum recommendations. These practices include asking
questions, developing and using models, planning and conducting investigations, analyzing
and interpreting data, using mathematics and computational thinking, constructing
explanations, engaging in evidence-based arguments, and obtaining, evaluating, andcommunicating information (Jaber et al., 2018). To foster disciplinary practices in a learning
environment, literature suggests guidelines such as: a) providing direct experience withphenomena, b) making learner thinking visible to encourage grappling with ideas forconstructing meaning, and c) assigning tasks with essential procedures and relevant
resources for completing investigations. Inquiry learning is recommended for effective sciencelearning, where students engage in sense-making, discussions, evidence-based explanations,etc (Jaber et al., 2018) (Moore et al., 2013) .
One of the goals for redesigning OLabs is to motivate learners to engage in lab related
activities. Gamification, an established approach to fostering student engagement, uses game
attributes to encourage game-like behavior in non-game contexts. Game-based mechanics
can effectively engage learners, motivate their actions, and aid problem-solving. Rewardbased gamification, a popular type of gamification, mainly relies on extrinsic motivation andmay not yield lasting changes, and can be potentially unsatisfactory for some learners.
Meaningful gamification, another type, employs game design elements to cultivate intrinsic
motivation in non-game settings by creating an enjoyable learning environment whereparticipants can explore and find meaning (Nicholson, 2015). This approach aims forsustained engagement compared to the short-term, reward-driven nature of extrinsic
motivation. Meaningful gamification is based on self-determination theory (Ryan & Deci, 2000),
which mentions competence, autonomy, and relatedness to form intrinsic motivation. The
theory highlights that when these three psychological needs are fulfilled, people find tasks
meaningful and continue participating.

3. Design
We propose a framework GaMINLab - Gamification (Meaningful) in INquiry Labs which derives
its basis from the research recommendations for fostering disciplinary practices in scienceinquiry labs, theory of meaningful gamification. The design features are listed below:z
The learner will be given challenges in form of problems to be solved, each categorized
in different problem scenarios like park, beach, etc. The system prompts learner for
problem-solving activities that can be attempted in quest to solve the chosen problem.
z
Learner is prompted for problem-solving activities which can be attempted in question tosolve problem. These activities are adaptation of the problem-solving inquiry frameworkin technology enhanced classroom (Kim & Hannafin, 2011). Exploration, Reconstruction,
Presentation/Communication are broadly mapped to tasks Investigate using the Lab,
Propose Answers and predictions and Share and Discuss your Lab Investigation Report.
The learner is provided access to specific OLabs simulation lab to investigate, makepredictions, etc.in quest to solve the given problem. It is embedded in specific tasks.
zLearner is encouraged to prepare artifacts like investigation plan, interim and final lab
investigation report comprising investigation plan, observations, proposed answers withreasoning, etc. Artifacts can be self, peer or teacher evaluated using given rubrics.
z
There is no predefined sequence of problem-solving activities and at every stage thesystem prompts the learner with options to proceed further and learner can choose any
one from those options, thus making a ‘choice’ in strategy to solve the problem. Potentiallya learner can view the problem and directly jump to attempting the final assessment or go

644

z

z

z
z

through one or more problem solving tasks or restart all over again. Motivating learners
to use variants & try other aspects of lab is an important need.
Each disciplinary task completion is acknowledged by system in form of corresponding
badge. These acknowledgments given in recognition to them as they are progressing
towards their final goal fulfils their needs of mastery and also motivates them to try further
problems, and activities including variants.
Lab onboarding activities make learner familiar with the UI objects in the lab, lab variants,
actions needed for taking observations, activities include interactive content using
hotspots, quiz and awards badge on completion.
Chat option is given to discuss ideas, clarify misunderstanding regarding predictions,investigations with peers. They can also share lab investigation reports with peers.
In the system scaffolds are provided when learners are preparing the investigation plan
and lab investigation report. When learners choose the options from alternate paths,
system provides brief information indicating what can be expected in the chosen stage
which will help learner to move ahead or retrack the action.

Conclusion
GaMINLab, a re-design of OLabs, offers opportunities for learners to engage participate indisciplinary practices such as planning and executing investigations within the simulation lab,
analyzing and interpreting recorded observations, constructing explanations based onevidence (observations), and communicating information by discussing and sharing the labreports with peers. Collaborative discussions address learners' need for connection,
preventing feelings of isolation during tasks. Making autonomous choices regarding whichpath to follow and which problem to solve fulfils their psychological desire to make selfdetermined choices, autonomy. The freedom to retry or restart reduces the fear of failure,
motivating learners to persist in solving problems even after initial setbacks. Acknowledgmentin form of badges motivates them to repeatedly engage in various problem-solving activities.
Lab onboarding tasks and scaffolding help prevent early frustration during initial lab
encounters. The inclusion of badges and lab onboarding supports the learners' pursuit ofmastery. Therefore, the design of GaMINLab seeks to intrinsically motivate learners (byfulfilling their needs for autonomy, mastery, and relatedness) to actively engage in given
scientific practices.
. Currently GaMINLab is designed with two labs with one problem each. To validate thedesign a pilot study with 10-15 students is being planned. In this study we aim to gather
information related to pattern of engagement of learners, motivation, degree of disciplinary
practices reflected in learner artifacts, learner perception, usability etc. Study-2 will be
conducted for larger set of learners (50-60), after incorporating suitable changes in system
based on the findings of pilot study.

Acknowledgement
We would like to thank members from ETU, CDAC and IDP-ET department, IIT Bombay

References
Jaber, L. Z., Dini, V., Hammer, D., & Danahy, E. (2018). Targeting disciplinary practices in an online
learning environment. Science Education, 102(4), 668–692. https://doi.org/10.1002/sce.21340
Kim, M. C., & Hannafin, M. J. (2011). Scaffolding 6th graders’ problem solving in technologyenhanced science classrooms: A qualitative case study. Instructional Science, 39(3), 255–282.
https://doi.org/10.1007/s11251-010-9127-4
Moore, E. B., Herzog, T. A., & Perkins, K. K. (2013). Interactive simulations as implicit support for
guided-inquiry. Chemistry Education Research and Practice, 14(3), 257–268.
https://doi.org/10.1039/c3rp20157k
M Sasikumar. (2016). OLabs makes school laboratories accessible anytime, anywhere - Open Source
For You. Open Source for You. https://opensourceforu.com/2016/09/olabs/
Nicholson, S. (2015). A recipe for meaningful gamification. In Gamification in Education and Business(pp. 1–20). Springer International Publishing. https://doi.org/10.1007/978-3-319-10208-5_1
Ryan, R. M., & Deci, E. L. (2000). Self-Determination Theory and the Facilitation of Intrinsic
Motivation, Social Development, and Well-Being. 55(1), 68–78.

645

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Skill Tracing Model for Player Character
Control in STG
Peizhe HUANGa*, Wanxiang LIb, Wen GUc, Kouichi OTAc, Shinobu HASEGAWAc
aDivision of Advanced Science and Technology, JAIST, Japanb
Mobilus Corporation, Japanc
Center for Innovative Distance Education and Research, JAIST, Japan{*s2320023, wgu, ota, hasegawa}@jaist.ac.jp, lwx9688@gmail.com
Abstract: STGs, a longstanding video game subgenre, have grown more intricate overtime, deterring new players. To address this, a training system is required to improvecharacter control skills in STG games. Bayesian Knowledge Tracing (BKT) is acommon approach researchers use to monitor and assess students' progress. While
BKT is effective in evaluating intellectual knowledge, it falls short in assessing character
control skills in STG, a form of motion knowledge. This study proposes a Skill Tracing(ST) model that combines BKT approaches to monitor both cognitive knowledge andcharacter control abilities. Results indicate its superiority in skill-tracking tasks overtraditional BKT, offering a more accurate prediction of players' skill levels.
Keywords: STGs, Bayesian Knowledge Tracing, skill modeling, character movementclassification

1. Introduction
STG, a challenging video game subgenre, demands exceptional reaction speed and skill
mastery. Despite a loyal fanbase, its complexity hampers novice skill acquisition, affecting
their experience. Matching challenge to skill level is vital for enjoyment (Sweetser & Wyeth,2005), making skill training crucial. Traditional methods like repetitive practice are timeconsuming and unspecific. Designing a focused training system for STG novices is essential
for rapid skill mastery, requiring skill learning tracing.
Skill tracing constitutes a vital component of the STG skill training system, allowing it
to comprehend a player's learning progress and skill proficiency. The Bayesian knowledge
tracing (BKT) method (Corbett & Anderson,1994) is commonly employed to model and monitor
a student's mastery of knowledge. While in most concept-based learning scenarios,
researchers merely assess whether a student has comprehended a concept or not, thesituation differs when a player aims to master a gaming skill. In this context, it becomes
important to not only gauge a player's familiarity with concepts such as game tips or tactics,
but also to evaluate whether the player can effectively apply this information during gameplay,drawing upon their physical capabilities. This distinction implies that BKT might not be optimal
for accurately modeling player skills.
This research aims to extend the BKT model to monitor player control skill learning in
STG game. To achieve this, the following research questions will be addressed:
1. How can the BKT model be extended from knowledge learning tracing to encompass skill
learning tracing?
2. Dose the proposed model outperform the BKT model in the context of STG skill tracing?

2. Proposed Method
2.1 Skill Tracing Model646

In skill learning, applying acquired knowledge involves the body's physical ability to become
adept at the technique, while knowledge learning involves purely intellectual engagement.
Physical ability here refers to coordinating bodily functions for motor tasks, encompassing
coordination, reflexes, balance, and spatial orientation.
Assuming skill mastery (‘S’ in Equation (1)) results from the overlap of conceptual
understanding (‘K’ in equation (1)) and physical familiarity (‘B’ in Equation (1)), it can be
represented as: S = K ∩ B (1)Figure 1. The structure of ST model
We proposed a Skill Tracing model (ST) based on the BKT model and Equation (1).
Figure 1 is the structure of the ST model. The parameters comprise two sets of [P(L), P(T),
P(G), P(S)], each belonging to a distinct BKT model. These four parameters represent theprobabilities of prior knowledge, learning new material, guessing, and making errors during
the learning process. We combine the predictions of the two BKTs and determine the player's
skill mastery state S using Equation (1). Two sets of responses, PK and PB, are gathered fromthe observable layer. These responses are obtained from the same game stage at each stepof the learning sequence. PK denotes whether a player used a tactic in the game, helping
predict the knowledge level (K) representing their understanding of that specific tactic in astage. Tactic use is detected by analyzing keypresses and character movement. Treating a
stage as a time series, the MrSQM time series classifier (Nguyen, T. L., & Ifrim, G., 2021) is
employed to identify tactic application. PB indicates whether a player completed a stage, andit's used to predict the body ability (B), indicating the player's capacity to finish a game stage
at a specific difficulty level.

2.2 Game environment and skill setting
In this study, the gaming environment is the self-developed game "stgST," created using theopen-source game engine LuaSTG (LuaSTG Wiki, n.d.). StgST comprises 18 stages with
distinct bullet patterns outlined in Table 1. Our research centers on character control skills,
vital for survival in the game. In this context, we define skills as employing specific tactics to
dodge corresponding bullet patterns as shown in Table 1).
Table 1. Abstract character control skill from bullet pattern and its tactic
Pattern name
Aimed Shot

TacticTiny Moving
RestreamRound Shot
Circling
Bottom Bullet
Upper Avoiding
Revenge Bullet Hold FireSkill
Use tiny moving to avoid aim-shot
Use restream movement to avoid aim-shot in screen edge
Use circling movement to avoid round shot
Move to upper screen to avoid bottom bullet
Stop shooting to avoid revenge bullet

647

3. Experiment and Result
We conducted an experiment to collect player data, recording a total of 957 game rounds from21 players. The author manually recorded instances of player failures per round, while
keypress information and character movement were automatically recorded. We assumed astage to be cleared if a player failed in it fewer than four times. Player keypress and charactermovement were categorized into different patterns using a pre-trained MrSQM model, derived
from author-generated play data.
The player data were modeled using the PyBKT Python library (Badrinath, A., Wang,F., & Pardos, Z., 2021). We compared the modeling outcomes of the ST and BKT models on
player data, additionally assessing the impact of introducing the "forgets" parameter. The"forgets" parameter, introduced by Qiu, Y. et al. (2011), signifies the tendency for students toforget knowledge at a certain rate during the learning process.
Taking into account the imbalanced input data, we employed F1 score and balanced
accuracy as metrics to assess the models. As indicated in Table 3, the results demonstratethat irrespective of the presence of the "forgets" parameter, the ST model surpasses the BKT
model in F1 score by approximately 7% and in balanced accuracy by about 5.5%.
Table 2. F1 score and balanced accuracy of four models.
F1 Score
Balanced Accuracy
BKT
0.500
0.684
BKT + forgets
0.511
0.695
ST
0.570
0.747
ST + forgets
0.577
0.751

4. Conclusion & Future WorkIn this study, we proposed a new model called the ST model to track a player's progress on
the task of character control skill. The ST model is an extension of the BKT model,
differentiating between knowledge-learning and skill-learning tasks. We validated the modelon the player character control skill-learning task and found that it outperforms BKT. In
addition, we found that the parameter forgot had essentially no effect on the results of theexperiment, probably because the duration of the experiment was so short that players wereunlikely to forget.
In future research, we would like to complete a skill training system for novice STG
players. Additionally, we plan to explore the potential application of the ST model to othertypes of skill-learning tasks beyond game-based scenarios.

References
Badrinath, A., Wang, F., & Pardos, Z. (2021). pyBKT: an accessible python library of Bayesianknowledge tracing models. arXiv preprint arXiv:2105.00385.
Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of proceduralknowledge. User modeling and user-adapted interaction, 4, 253-278.
LuaSTG Wiki. (n.d.). Main Page. Retrieved from http://en.luastg.shoutwiki.com/wiki/Main_Page.
Nguyen, T. L., & Ifrim, G. (2021). MrSQM: Fast time series classification with symbolic representations.
arXiv preprint arXiv:2109.01036.
Qiu, Y., Qi, Y., Lu, H., Pardos, Z. A., & Heffernan, N. T. (2011, July). Does Time Matter? Modeling theEffect of Time with Bayesian Knowledge Tracing. In EDM (pp. 139-148).
Sweetser, P., & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games.
Computers in Entertainment (CIE), 3(3), 3-3. doi: 10.1145/1077246.1077253

648

6KLK -/ HW DO (GV   3URFHHGLQJV RI WKH VW ,QWHUQDWLRQDO &RQIHUHQFH RQ &RPSXWHUV LQ (GXFDWLRQ$VLD3DFLILF 6RFLHW\ IRU &RPSXWHUV LQ (GXFDWLRQ

7KH 'HVLJQ DQG 3UDFWLFH RI 6FLHQWLILF,QTXLU\ $FWLYLWLHV IRU &KLOGUHQ $JHG %DVHG RQ DQ $5 )ODVKFDUG (QYLURQPHQW
7LDQ7LDQ *28 0LQ6KHQJ )$1%LQ/L :$1*
6FKRRO RI (GXFDWLRQDO 7HFKQRORJ\ 1RUWKZHVW 1RUPDO 8QLYHUVLW\ &KLQD[PDQIPV#FRP$EVWUDFW ,Q WKLV VWXG\ D GHVLJQ IUDPHZRUN IRU VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ
DJHG  EDVHG RQ DQ $5 IODVKFDUG HQYLURQPHQW LV FRQVWUXFWHG FDVHV RI VFLHQWLILF
LQTXLU\ DFWLYLWLHV DUH GHVLJQHG DFFRUGLQJ WR WKLV IUDPHZRUN DQG TXDVLH[SHULPHQWDOVWXGLHV DUH FDUULHG RXW $QG LW VHOHFWHV WZR ODUJH FODVVHV RI VWXGHQWV LQ D NLQGHUJDUWHQ
LQ ;LDKH &RXQW\ *DQVX 3URYLQFH RI &KLQD DV H[SHULPHQWDO VXEMHFWV DQG VHWV XS DQ
H[SHULPHQWDO FODVV DQG D FRQWURO FODVV ZLWK  VWXGHQWV HDFK7KH UHVHDUFK UHVXOWVLQGLFDWH WKDW FRPSDUHG ZLWK WKH WUDGLWLRQDO OHDUQLQJ HQYLURQPHQW WKH LPSOHPHQWDWLRQHIIHFW RI  FKLOGUHQ V VFLHQWLILF LQTXLU\ DFWLYLWLHV EDVHG RQ DQ $5 IODVKFDUG
HQYLURQPHQW LV REYLRXV $5 WHFKQRORJ\ IXOO\ PRELOL]H FKLOGUHQ V LQWHUHVW DQG
HQWKXVLDVP IRU LQTXLU\ REWDLQ D EHWWHU LQTXLU\ H[SHULHQFH DQG WKXV LPSURYH FKLOGUHQ V
VFLHQWLILF LQTXLU\ DELOLWLHV.H\ZRUGV $5 IODVKFDUG FKLOGUHQ V VFLHQFH HGXFDWLRQ FKLOGUHQ V VFLHQWLILF LQTXLU\
DELOLWLHV ,QWURGXFWLRQ,Q $SULO  WKH 0LQLVWU\ RI (GXFDWLRQ LVVXHG WKH 6FLHQFH &XUULFXOXP 6WDQGDUGV IRU 3ULPDU\
6FKRROV RI &RPSXOVRU\ (GXFDWLRQ6XEVHTXHQWO\ WKH 0LQLVWU\ RI (GXFDWLRQ RI &KLQD LVVXHG
WKH *XLGHOLQHV IRU WKH /HDUQLQJ DQG 'HYHORSPHQW RI &KLOGUHQ $JHG DQG WKH *XLGHHPSKDVL]HV WKDW WKH HVVHQFH RI FKLOGUHQ V VFLHQFH HGXFDWLRQ LV WR VWLPXODWH FKLOGUHQ V LQWHUHVW
LQ LQTXLU\ DOORZLQJ WKHP WR H[SHULHQFH WKH SURFHVV RI LQTXLU\ DQG FXOWLYDWLQJ WKHLU HDUO\ LQTXLU\
DELOLWLHV =KDQODQ /LX  &KLOGUHQ V VFLHQWLILF LQTXLU\ DFWLYLWLHV EHORQJ WR WKH LPSRUWDQW
FRQWHQW RI NLQGHUJDUWHQ VFLHQFH HGXFDWLRQ *HQHUDOO\ VSHDNLQJ FKLOGUHQ V VFLHQWLILF LQTXLU\DFWLYLWLHV DUH GLYLGHG LQWR IRUPDO DQG LQIRUPDO VFLHQWLILF LQTXLU\ DFWLYLWLHV /DQ /X 0HL+DQJ +RZHYHU WKHUH DUH PDQ\ XUJHQW SUREOHPV WKDW QHHG WR EH VROYHG LQ FKLOGUHQ V VFLHQWLILF
LQTXLU\ DFWLYLWLHV LQ UHDOLW\ VXFK DV WKH IRUPDOL]DWLRQ RI LQTXLU\ WKH ODFN RI QHFHVVDU\ OHDUQLQJUHVRXUFHV ZKLFK DUH QRW FRQGXFLYH WR WKH GHYHORSPHQW RI FKLOGUHQ V VFLHQWLILF LQTXLU\ DELOLWLHV,Q WKH YLUWXDO DQG UHDO HQYLURQPHQW FUHDWHG E\ $5 IODVKFDUGV FKLOGUHQ FDQ LQWHUDFW ZLWK WKH '
PRGHOV LQ UHDOWLPH DIWHU VFDQQLQJ WKH $5 IODVK FDUG WKURXJK WKH VPDUW PRELOH GHYLFH ZKLFKFDQ VWLPXODWH FKLOGUHQ V LQWHUHVW LQ OHDUQLQJ DQG LQTXLU\ DQG SXW WKHPVHOYHV LQ D VDIHHQYLURQPHQW IRU H[SHULPHQWDO RSHUDWLRQ DQG LQGHSHQGHQW LQTXLU\ WKHUHE\ LPSURYLQJ WKH
SUREOHPV H[LVWLQJ LQ FKLOGUHQ V VFLHQFH DFWLYLWLHV
%DVHG RQ WKH DERYH UHDVRQV WKLV VWXG\ DLPV WR DSSO\ WKH $5 IODVK FDUGEDVHG OHDUQLQJHQYLURQPHQW WR WKH GHVLJQ DQG LPSOHPHQWDWLRQ RI VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ DJHG
 PDLQO\ VWXG\LQJ WKH IROORZLQJ LVVXHV +RZ WR GHVLJQ D IUDPHZRUN IRU VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ DJHG  EDVHG
RQ WKH $5 IODVKFDUG HQYLURQPHQW"
 &DQ WKH IUDPHZRUN IRU VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ DJHG  EDVHG RQ WKH$5 IODVKFDUG HQYLURQPHQW HIIHFWLYHO\ LPSURYH FKLOGUHQ V VFLHQWLILF LQTXLU\ DELOLWLHV"

649

 /LWHUDWXUH 5HYLHZ
 $5 7HFKQRORJ\ )HDWXUHV DQG LWV DSSOLFDWLRQ LQ HDUO\ FKLOGKRRG HGXFDWLRQ$5 WHFKQRORJ\ FDQ LQWHUDFW ZLWK YLUWXDO DQG UHDO REMHFWV GXH WR LWV FKDUDFWHULVWLFV ZLWK WKH
WHFKQLFDO FKDUDFWHULVWLFV RI $5 WKH XVH RI WHFKQRORJ\ WRROV LQ HGXFDWLRQ FDQ SURYLGH QHZOHDUQLQJ RSSRUWXQLWLHV WR LQFUHDVH WKH LQWHUDFWLRQ EHWZHHQ LQGLYLGXDOV DQG HQWHUWDLQPHQW
OHDUQLQJ EULQJ D ULFK OHDUQLQJ H[SHULHQFH DQG PDNH WKH OHDUQLQJ SURFHVV PRUH SRVLWLYH
HIIHFWLYH DQG PHDQLQJIXO $OVXPDLW DQG 0XVDZL 
7KURXJK OLWHUDWXUH UHYLHZ LW LV IRXQG WKDW WKH FXUUHQW H[SUHVVLRQ RI HDUO\ FKLOGKRRG
HGXFDWLRQ FRQWHQW LV PDLQO\ FRJQLWLYH FDUGV JDPHV SLFWXUH ERRNV HWF$5 WHFKQRORJ\ EOHQGVWKH SK\VLFDO DQG YLUWXDO ZRUOGV WR SURYLGH D QHZ OHDUQLQJ SODWIRUP DQG WKH WHFKQRORJ\ LV QRZZLGHO\ XVHG LQ WKH HGXFDWLRQDO HQYLURQPHQW RI HOHPHQWDU\ KLJK VFKRRO DQG FROOHJH VWXGHQWV,Q WKH VWXG\  WKH H[SHULPHQWDO UHVXOWV VKRZHG WKDW $5 LPSURYHG VWXGHQWV OHDUQLQJ PRWLYDWLRQHQJDJHPHQW DQG DFKLHYHPHQW FRPSDUHG ZLWK WUDGLWLRQDO WHDFKLQJ PHGLD %DFFD - HWDO 6LPLODUO\ LQ RQH VXUYH\ VWXGHQWV ZHUH IRXQG WR XQGHUVWDQG DQG UHPHPEHU FRQWHQWDQG SHUIRUP EHWWHU FRRSHUDWH PRUH DQG VKRZ PRUH PRWLYDWLRQ DQG IRFXV ZKHQ XVLQJ $5
FRPSDUHG WR QRQ$5 WUDGLWLRQV RU PHGLD 5DGX , $FFRUGLQJ WR WKH DERYH VWDWHPHQW GXH
WR WKH SUREOHPV HQFRXQWHUHG E\ FKLOGUHQ DJHG  LQ WKH LPSOHPHQWDWLRQ RI VFLHQWLILF DFWLYLWLHV
DV ZHOO DV WKH FXUUHQW JRDOV DQG UHTXLUHPHQWV RI FKLOGUHQ V VFLHQFH HGXFDWLRQ DQG WKHDSSOLFDWLRQ RI $5 WHFKQRORJ\ $5 WHFKQRORJ\ LV XQGRXEWHGO\ D VXLWDEOH WRRO RU SURGXFW IRUFKLOGUHQ V OHDUQLQJ 'HVLJQ RI 6FLHQWLILF ,QTXLU\ $FWLYLWLHV IRU &KLOGUHQ $JHG  %DVHG RQ DQ $5)ODVKFDUG (QYLURQPHQW )UDPHZRUN &RQVWUXFWLRQ
7KLV VWXG\ DLPV WR LPSURYH FKLOGUHQ V VFLHQWLILF LQTXLU\ DELOLWLHV DQG DFFRUGLQJ WR WKH $'',(WHDFKLQJ GHVLJQ PRGHO 0HUULHQERHU  IURP WKH ILYH VWDJHV RI DQDO\VLV GHVLJQGHYHORSPHQW LPSOHPHQWDWLRQ DQG HYDOXDWLRQ WKLV VWXG\ EHQFKPDUNV WKH SUHOLPLQDU\
DQDO\VLV DFWLYLW\ SURFHVV GHVLJQ DFWLYLW\ UHVRXUFH GHYHORSPHQW DQG SUHSDUDWLRQ DQG DFWLYLW\
FDVH LPSOHPHQWDWLRQ RI FKLOGUHQ V VFLHQWLILF LQTXLU\ DFWLYLWLHV EDVHG RQ DQ $5 IODVKFDUG
HQYLURQPHQW $ GHVLJQ IUDPHZRUN LQ )LJXUH  IRU VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ DJHG
 EDVHG RQ DQ $5 IODVKFDUG HQYLURQPHQW LV FRQVWUXFWHG $PRQJ WKHP LQ WKH DFWLYLW\ SURFHVVGHVLJQ VWDJH WKH ( WHDFKLQJ PRGHWKDW LV LQ OLQH ZLWK WKH JRDOV DQG FRQFHSWV RI VFLHQFH
HGXFDWLRQ LV VHOHFWHG DV WKH EDVLV ZKLFK LV D QHZ WHDFKLQJ PHWKRG GHYHORSHG E\ $PHULFDQHGXFDWRUV LQ WKH ILHOG RI ELRORJ\ %6&6  ZKLFK LV ILYH OLQNV (QJDJHPHQW ([SORUDWLRQ([SODQDWLRQ (ODERUDWLRQ DQG (YDOXDWLRQ)LJXUH  $ )UDPHZRUN IRU 6FLHQWLILF ,QTXLU\ $FWLYLWLHV IRU &KLOGUHQ $JHG %DVHG  RQ DQ $5)ODVKFDUG (QYLURQPHQW

650

 /HDUQLQJ $FWLYLWLHV 'HVLJQ $FWLYLW\ &DVH 'HVLJQ
7KH WKHPH RI WKLV UHVHDUFK DFWLYLW\ LV VHOHFWHG IURP WKH  +DUPRQLRXV GHYHORSPHQW FXUULFXOXPWHDFKHU V %RRN  7DEOH  LV D FRQFUHWH H[DPSOH RI DQ DFWLYLW\ FDVH
7DEOH  $FWLYLW\ &DVH 'HVLJQ$FWLYLW\ WKHPH$FWLYLW\ VHVVLRQ$FWLYLW\ FRQWHQW&RQWH[W LPSRUW&UHDWH SUREOHPVLWXDWLRQV WR VWLPXODWHFKLOGUHQ V LQWHUHVW

7KH ZLQWHULQJ PHWKRGV RI VPDOO DQLPDOV7HDFKHU DFWLYLWLHV&KLOGUHQ DFWLYLWLHV

$ 3OD\ D YLGHR RI -XOLH KLEHUQDWLQJ IURP$ )UHHO\ DQVZHU WKH
6SRQJH%RE 6TXDUH3DQWVDQLPDWHG FKDUDFWHUV
³&KLOGUHQ KDYH \RX VHHQ 6SRQJH%RE
LQ WKH FDUWRRQV 
6TXDUH3DQWV" DQG :KDW DQLPDOV DUH LQ " % :DWFK WKH YLGHR :K\ GRHVQ W -XOLH VOHHS IRU PRQWKVDQG DQVZHU \RXU
LQVWHDG RI SOD\LQJ ZLWK KHU IULHQGV"WHDFKHU V TXHVWLRQV$VN TXHVWLRQV
% 6KRZ SLFWXUHV RI PXOWLSOH DQLPDOV
& ([FKDQJH H[LVWLQJ
0DNH DVVXPSWLRQVWKURXJK 337H[SHULHQFHV QDPH
7KURZ TXHVWLRQV :H KDYH DOUHDG\ NQRZQ PDQ\ VPDOO
DQLPDOV DQG KRZ WR
HOLFLW DFWLYLWLHVDQLPDOV EHIRUH VR KRZ WKHVH VPDOO DQLPDOV VSHQG WKH ZLQWHUVXUYLYH WKH FROG ZLQWHU"
([SORUH RSHUDWLRQV
& 8VH $5 IODVKFDUGV WR VKRZ FKLOGUHQ WKH ' 8VH $5 WRROV WR&KLOGUHQ XVH $5 WRROV FKDUDFWHULVWLFV RI DQLPDOV 
H[SORUH WKH QDPHVWR H[SORUH KRZ
' *XLGH FKLOGUHQ WR XVH $5 WRROV WRFKDUDFWHULVWLFV DQG
DQLPDOV VXUYLYH WKH
H[SORUH WKH ZD\ DQLPDOV VXUYLYH WKH ZLQWHU ZLQWHULQJ VW\OHV RIZLQWHU( 'LVWULEXWH WDEOHWV $5 IODVKFDUGV DQG
DQLPDOVORJ VKHHWV DQG H[SODLQ WKH XVH RI $5
( 7KH JURXS ZRUNVIODVKFDUGV DQG ORJ VKHHWVWRJHWKHU WR FRPSOHWH) 3DWURO DQG JXLGH FKLOGUHQ GXULQJ WKH
WKH DFWLYLW\ UHFRUG
REVHUYDWLRQ SURFHVV HQFRXUDJH FKLOGUHQ WR VKHHW
SDUWLFLSDWH LQ DFWLYLWLHV DQG UHPLQG FKLOGUHQ
WR WDNH QRWHV LQ D WLPHO\ PDQQHU
6KDUH DQG
* $VN WKH FKLOGUHQ LQ HDFK JURXS WR UHSRUW ) 6KDUH WKHLU RZQFRPPXQLFDWH
RQ WKHLU UHFRUG VKHHWV DQG WDON DERXW WKHLUILQGLQJV DQG ZKDW
6KDUH ILQGLQJVJDLQV
WKH\ KDYH OHDUQHG
([SODLQ GRXEWV
+ 6XPPDUL]H WKH UHVXOWV RI WKH LQTXLU\
* /LVWHQ FDUHIXOO\ WR
7HDFKHUV DQVZHUDQG DQVZHU WKH FKLOGUHQ V TXHVWLRQVWKH WHDFKHU V TXHVWLRQV
TXHVWLRQV DQG
DQG VXPPDULHVLQWHJUDWH NQRZOHGJH([WHQGHG PLJUDWLRQ , 6WDUW IURP WKH OLIH VLWXDWLRQ DJDLQ DQG
+ %DVHG RQ ZKDW*XLGH FKLOGUHQ WR
JXLGH FKLOGUHQ WR VROYH SUREOHPV ZLWK QHZKDV EHHQ REVHUYHG
VROYH SUREOHPV ZLWKNQRZOHGJHEHIRUH WKLQN DQG
QHZ NQRZOHGJH
- 6KRZ WKH WDVN FRQWHQW DQG OHW WKH
DQVZHU TXHVWLRQV LQ
WRGGOHU WKLQNUHODWLRQ WR WKH &DQ VXFK VPDOO DQLPDOV OLYH WRJHWKHU" SUHVHQW
:K\" +RZ WR FODVVLI\ LW"0XOWLGLPHQVLRQDO
. (QFRXUDJH FKLOGUHQ WR VHOIHYDOXDWH WKH , (YDOXDWH WKH
HYDOXDWLRQFRPSOHWLRQ RI WKHLU RZQ JURXS UHFRUG VKHHWV UHSRUWLQJ RI WKHLU RZQ
6HOIHYDOXDWLRQ DQG
DQG WR SHHUHYDOXDWH WKH FRPSOHWLRQ RIJURXSV DQG 3HHU
PXWXDO HYDOXDWLRQ DUH RWKHU JURXSVUHSRUWLQJHQFRXUDJHG

651

 0HWKRGV 5HVHDUFK 0HWKRGV DQG 'HVLJQ
7KLV VWXG\ DGRSWV TXDVLH[SHULPHQW DQG FROOHFWV UHOHYDQW GDWD RI FKLOGUHQ WKURXJK VFDOH DQG
LQWHUYLHZ FRQWHQW EHIRUH DQG DIWHU WKH H[SHULPHQW &RPSUHKHQVLYH TXDOLWDWLYH DQG TXDQWLWDWLYHPHWKRGV DUH XVHG WR DQDO\]H DQG FRPSDUH ZKHWKHU WKH WUDGLWLRQDO OHDUQLQJ HQYLURQPHQW DQG
WKH $5 IODVK FDUGEDVHG OHDUQLQJ HQYLURQPHQW FDXVH GLIIHUHQFHV LQ WKH FXOWLYDWLRQ RI FKLOGUHQ V
VFLHQWLILF LQTXLU\ DELOLWLHV DQG ZKHWKHU WKH FKLOGUHQ V VFLHQWLILF LQTXLU\ DELOLWLHV DUH VLJQLILFDQWO\
LPSURYHG)LJXUH  6SHFLILF ([SHULPHQWDO 3URFHGXUHV 5HVHDUFK 2EMHFWV DQG ,PSOHPHQWDWLRQ,Q WKLV VWXG\ WZR ODUJH JURXSV LQ D NLQGHUJDUWHQ LQ ;LD KH &RXQW\ *DQVX 3URYLQFH DUH
VHOHFWHG DV H[SHULPHQWDO REMHFWV 7KH QXPEHU RI VWXGHQWV LQ ERWK WKH H[SHULPHQWDO DQG WKH
FRQWURO JURXS LV  7KH H[SHULPHQWDO JURXS DGRSWV WKH VFLHQWLILF LQTXLU\ DFWLYLW\ SURFHVV EDVHG
RQ DQ $5 IODVKFDUG HQYLURQPHQW WR WHDFK ZKLOH WKH FRQWURO JURXS DGRSWV WKH WUDGLWLRQDOWHDFKLQJ HQYLURQPHQW  DQG FRQGXFWV  WHDFKLQJ DFWLYLWLHV LQ WRWDO IRU  ZHHNV %HIRUH WKH
H[SHULPHQW EHJLQV WKH UHVHDUFK WRROV DUH XVHG WR SUHWHVW WKH VFLHQWLILF LQTXLU\ DELOLWLHV RIFKLOGUHQ VR DV WR HQVXUH WKDW WKH OHYHO RI VWXGHQWV LQ WKH WZR JURXSV LV VLPLODU DQGKRPRJHQHRXV ,Q WKH H[SHULPHQWDO JURXS WZR VWXGHQWV FRRSHUDWH WR FRQWURO WKH WDEOHW DQG
FRPSOHWH WKH H[SORUDWLRQ WDVN $FFRUGLQJ WR WKH WHDFKHU V H[SODQDWLRQ DQG GHPRQVWUDWLRQ DVZHOO DV WKH VWXGHQW¶V SHUIRUPDQFH WKH SRVWWHVW GDWD LV FROOHFWHG DQG DQDO\]HG DIWHU WKH
WHDFKLQJ DFWLYLWLHV)LJXUH  7HDFKLQJ ,PSOHPHQWDWLRQ ([DPSOHV

 0HDVXUHPHQW 7RROV 7HVW 6FDOH
7KH WHVW VFDOH RI WKLV VWXG\ LV WKH (YDOXDWLRQ 7DEOH RI &KLOGUHQ V 6FLHQWLILF ,QTXLU\ $ELOLWLHV ,WV
REVHUYDWLRQ LQGLFDWRUV DUH GLYLGHG LQWR VL[ GLPHQVLRQV 7KHUH DUH  JUDGLQJ VFDOHV XQGHU HDFK652

GLPHQVLRQ ,WV &KURQEDFK¶V $OSKD FRHIILFLHQW LV  LQGLFDWLQJ JRRG UHOLDELOLW\ 7KH .02
YDOXH LV  LQGLFDWLQJ WKDW WKH YDOLGLW\ PHHWV WKH UHTXLUHPHQWV ,QWHUYLHZ 2XWOLQH
7KLV VWXG\ GHVLJQV WKH WHDFKHU DQG FKLOGUHQ LQWHUYLHZ RXWOLQHV 7KH WHDFKHU LQWHUYLHZ GHVLJQV
VL[ TXHVWLRQV VXFK DV 7R LPSOHPHQW VFLHQWLILF LQTXLU\ DFWLYLWLHV EDVHG RQ DQ $5 IODVKFDUG
HQYLURQPHQW ZKDW DUH WKH FKDQJHV LQ \RXU WHDFKLQJ PHWKRGV"´ 7KH FKLOGUHQ V LQWHUYLHZ DOVRGHVLJQV VL[ TXHVWLRQV VXFK DV :KDW GR WKH FKLOGUHQ OHDUQ LQ FODVV WRGD\"´ 'DWD $QDO\VLV 3UHH[SHULPHQWDO +RPRJHQHRXV $QDO\VLV
7KH GDWD LQ 7DEOH  VKRZV WKDW WKH 6LJ RI WKH ERWK RI JURXSV DUH DOO JUHDWHU WKDQ  LQ VL[GLPHQVLRQV ZKLFK GR QRW UHDFK D VLJQLILFDQW GLIIHUHQFH ,W VKRZV WKDW WKH H[SHULPHQWDO JURXSDQG FRQWURO JURXS FDQ FDUU\ RXW WKH QH[W LQWHUYHQWLRQ H[SHULPHQW ZLWK WKH VDPH OHYHO RI
VFLHQWLILF LQTXLU\ DELOLWLHV
7DEOH  +RPRJHQHRXV $QDO\VLV RI 6FLHQWLILF ,QTXLU\ $ELOLWLHV LQ ([SHULPHQWDO DQG &RQWURO*URXSV&XULRVLW\ DQG LQWHUHVW
3URSHQVLW\ IRU LQTXLU\
2SHUDELOLW\ RI PDWHULDOV'HSWK RI LQTXLU\$ELOLW\ WR UHFRUG
LQIRUPDWLRQ$ELOLW\ WR FROODERUDWH DQG
FRPPXQLFDWH
7RWDO VFRUH*URXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS([SHULPHQWDO JURXS&RQWURO JURXS

1















0















6'















7

6LJ





























 3RVWH[SHULPHQWDO 6LJQLILFDQFH RI 'LIIHUHQFHV $QDO\VLV$FFRUGLQJ WR WKH GDWD DQDO\VLV UHVXOWV RI 7DEOH  7KH DYHUDJH VFRUH RI WKH H[SHULPHQWDO JURXSLV VLJQLILFDQWO\ KLJKHU WKDQ WKDW RI WKH FRQWURO JURXS LQGLFDWLQJ WKDW WKH LPSOHPHQWDWLRQ HIIHFW RI
WKH H[SHULPHQWDO JURXS LV EHWWHU WKDQ WKDW RI WKH FRQWURO JURXS
7KH UHVXOWV LQ 7DEOH  EHORZ RI WKH LQGHSHQGHQW VDPSOHV 7WHVW DQDO\VLV RI WKH
H[SHULPHQWDO JURXS DQG WKH FRQWURO JURXS VKRZV WKDW WKH VLJQLILFDQFH RI WKH 7WHVW LV LQGLFDWLQJ WKDW WKHUH LV D VLJQLILFDQW GLIIHUHQFH EHWZHHQ WKH DYHUDJH YDOXH RI WKHWRWDO VFRUH RI VFLHQWLILF LQTXLU\ DELOLWLHV RI VWXGHQWV LQ WKH H[SHULPHQWDO JURXS DQG WKH FRQWUROJURXS
7DEOH  ,QGHSHQGHQW 6DPSOHV 7WHVW RI 6FLHQWLILF ,QTXLU\ $ELOLWLHV LQ ([SHULPHQWDO DQG &RQWURO*URXSV

7RWDO
VFRUH

*URXS
([SHULPHQWDO JURXS&RQWURO JURXS

1



0 5HODWLYH VFRUH



653

6'



)


W
6LJ
 

$FFRUGLQJ WR WKH 7WHVW DQDO\VLV UHVXOWV RI SDLUHG VDPSOHV LQ 7DEOH  EHORZ WKH DYHUDJH
VFRUH GLIIHUHQFH RI VL[ GLPHQVLRQV RI VFLHQWLILF LQTXLU\ DELOLWLHV LV  DQG WKH FRUUHODWLRQVLJQLILFDQFH RI WKH WRWDO VFRUH RI SDLUHG VDPSOHV LV OHVV WKDQ  LQGLFDWLQJ WKDW WKHUH LV D
VLJQLILFDQW GLIIHUHQFH EHWZHHQ FKLOGUHQ V WRWDO VFRUH RI VFLHQWLILF LQTXLU\ DELOLWLHV EHIRUH DQG
DIWHU WKH SUDFWLFH RI LQTXLU\ DFWLYLWLHV
7DEOH  3DLUHG 6DPSOHV 7WHVW RI 6FLHQWLILF ,QTXLU\ $ELOLWLHV LQ ([SHULPHQWDO *URXS

3UHWHVW7RWDO VFRUH

10 5HODWLYH VFRUH

3RVWWHVW7RWDO VFRUH





GI


W


6LJ




 ,QWHUYLHZ $QDO\VLV
7KURXJK WKH DQDO\VLV RI LQWHUYLHZV EHWZHHQ WHDFKHUV DQG \RXQJ FKLOGUHQ LW FDQ EH VHHQ WKDWWHDFKHUV DUH VDWLVILHG ZLWK WKH GHVLJQ DQG LPSOHPHQWDWLRQ RI VFLHQWLILF LQTXLU\ DFWLYLWLHV EDVHG
RQ WKH $5 HQYLURQPHQW FKLOGUHQ V VFLHQWLILF LQTXLU\ DELOLWLHV KDYH EHHQ LPSURYHG LQ WKH
H[SHULPHQW DQG WHDFKHUV DUH DOVR ZLOOLQJ WR XVH $5 LQ IXWXUH DFWLYLWLHV ,W ZDV DJUHHG WKDW $5
WRROV ZHUH PRUH IXQ DQG FRQYHQLHQW WKDQ SUHYLRXV DFWLYLWLHV DQG WKDW FKLOGUHQ FRXOG H[SORUH
DQG UHFRUG LQ DQ DFWLYH DWPRVSKHUH

 &RQFOXVLRQ
7KURXJK WKLV H[SHULPHQW WKLV VWXG\ PDLQO\ GUDZV WKH IROORZLQJ FRQFOXVLRQV 7KH GHVLJQ IUDPHZRUN RI VFLHQWLILF LQTXLU\ DFWLYLWLHV IRU FKLOGUHQ DJHG  EDVHG RQ WKH$5 IODVKFDUG LV HIIHFWLYH $5 IODVKFDUGV FDQ SURPRWH WKH HIIHFWLYH GHYHORSPHQW RI FKLOGUHQ V VFLHQWLILF LQTXLU\DFWLYLWLHV

5HIHUHQFHV
7KH 0LQLVWU\ RI (GXFDWLRQ   ,QIRUPDWLRQ 7HFKQRORJ\ &XUULFXOXP 6WDQGDUGV IRU &RPSXOVRU\(GXFDWLRQ  (GLWLRQ  3HRSOH¶V (GXFDWLRQ 3UHVV/LX =KDQODQ   6FLHQFH HGXFDWLRQ IRU SUHVFKRRO FKLOGUHQ %HLMLQJ 1RUPDO 8QLYHUVLW\ 3UHVV/X /DQ +DQJ 0HL   (DUO\ FKLOGKRRG VFLHQFH HGXFDWLRQ DQG DFWLYLW\ JXLGDQFH %HLMLQJ 1RUPDO
8QLYHUVLW\ 3UHVV$OVXPDLW $ $O0XVDZL =6   &UHDWLYH DQG LQQRYDWLYH HOHDUQLQJ XVLQJ LQWHUDFWLYH VWRU\WHOOLQJ,QWHUQDWLRQDO -RXUQDO RI 3HUYDVLYH &RPSXWLQJ DQG &RPPXQLFDWLRQV    
GRL,-3&&
%DFFD -%DOGLULV 6 )DEUHJDW 5 *UDI 6 .LQVKXN - 7KH WUHQG RI DXJPHQWHG UHDOLW\ LQ (GXFDWLRQV\VWHPDWLF HYDOXDWLRQ RI UHVHDUFK DQG $SSOLFDWLRQ (GXFDWLRQ7HFKQRORJ\ DQG VRFLHW\   SS ± 
5DGX ,$XJPHQWHG UHDOLW\ LQ HGXFDWLRQ D PHWDUHYLHZ DQG FURVVPHGLD DQDO\VLV3HUV 8ELTXLWRXV&RPSXW    SS  V\654

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Toward Game-Based Learning of JapaneseWriting for Elementary School Students
Kazumasa OMURAa*, Kei KUBOb, Frederic BERGERONc & Sadao KUROHASHIa,da
Graduate School of Informatics, Kyoto University, Japanb
Center for Japanese Language, Waseda University, Japancd
SystemI Co., Ltd., Japan
National Institute of Informatics, Japan*omura@nlp.ist.i.kyoto-u.ac.jp
Abstract: It is a long-standing problem that many elementary school students in Japan
have an aversion to writing compositions. To address the problem, we designed an AI
educational game for elementary school students to study Japanese writing utilizing
existing language resources. In the game, players construct simple and complex
sentences by connecting given word cards with particle marks. The constructed
sentences are automatically scored using large-scale language resources, allowing
players to receive on-the-spot feedback, such as how to improve the use of a casemarker. We also developed smartphone and web applications of the game andconducted a user study to assess it. The results of the user study demonstrated that
our application can be used as a good introduction to studying Japanese writing.
Keywords: Game-based learning, Japanese writing, Language resources

1. Introduction
Since we use language as a means of cognition and communication, language education
plays an essential role in our lives. In language education, written language production is also
indispensable for learning how to express our thoughts.
However, in Japanese writing education, it is a long-standing problem that many
elementary school students have an aversion to writing compositions (National Institute for
Educational Policy Research, 2008; Ritsumeikan University Library, 2017). One of thepossible reasons is that they learn Japanese writing only through open-ended writing
assignments such as book reports and essays on topics related to daily life. Such assignments
rarely motivate the students to go further than merely complete them, and they often struggle
to come up with what to write due to the open-ended format. In addition, they receive little
feedback on their writing, leading to a vicious cycle where they become increasingly aware ofbeing poor at writing without learning how to improve.
In order to ameliorate the current situation, Japanese education is in need of an
educational material that offers a more engaging experience, i.e., allows students to constructsentences with fun and receive feedback on their writing. However, there are two major
challenges to achieving this: how to reduce an aversion to constructing sentences and how toevaluate writing automatically.
Game-based learning (Vandercruysse et al., 2012), which aims to make the learning
process more fun with a game, is a promising solution to the first challenge. There has beenno AI educational game for studying Japanese writing to the best of our knowledge; thus, it isworth attempting to develop such a game and investigate its effectiveness.
Regarding the second challenge, we focus on existing language resources. In natural
language processing, large-scale language resources have been built so far to teach linguistic
and world knowledge to computers (Fellbaum, 1998; Kawahara & Kurohashi, 2006; Speer etal., 2017), some of which can be utilized for human learning as well as machine learning. By
exploiting them, it is now possible to automatically generate and score simple writing practicequestions, and the aforementioned educational material has become more feasible.

655

Against the above background, we design an AI educational game for elementary
school students to study Japanese writing utilizing existing language resources. Hereafter, wecall it “Kotoba-musubi”. (“Kotoba” and “musubi” are the Japanese words that mean “words”
and “connecting”, respectively.) Section 2 describes the details.
We also develop smartphone and web applications of Kotoba-musubi and conduct auser study to investigate its effectiveness. Section 3 presents the positive results showing theeffectiveness in reducing an aversion to writing compositions. We have released the game foreducational purposes, the link to which is provided in Section 4 with a conclusion.

2. Kotoba-musubi
2.1 Game Design
Kotoba-musubi is a word-based game, where the player constructs simple sentences byconnecting content words with case markers and combining the simple sentences withdiscourse markers into complex sentences with contingent relations (cf. Figure 1). In order toconstruct sentences, players arrange given 12 word cards and nine particle marks according
to the empty rectangle and oval frames, respectively.

Figure 1. Screenshot of Kotoba-musubi with gloss (left) and its scoring results (right). “NOM”
and “ACC” in the left figure stand for “nominative” and “accusative” cases, respectively.
Word Card: The 12 word cards break down into six noun cards, five verb/adjective
cards, and one wildcard that allows players to enter a word freely. The role of each card isdistinguishable by its color or shape; thus, players can learn how to construct sentences even
if they do not understand the concept of parts of speech. In addition, ruby characters are
written above Chinese characters to facilitate reading. Regarding verb/adjective cards, thepredicates can be conjugated in the present, past, negative, or past-negative form, whichmakes it possible to create more diverse sentences.
Particle Mark: The nine particle marks break down into five of major case particles(nominative, accusative, dative, and instrumental cases, and “with” or “and”) and four of
discourse connectives representing contingent (causal and conditional) relations. As with theword cards, the role of each mark is distinguishable by its color or shape. Each mark is arrowshaped, its direction indicating the dependency between words.
The score of each sentence is automatically computed by pattern matching withlanguage resources, allowing players to receive instant feedback. For instance, if no examples
of a simple sentence are found in the Japanese case frames (Kawahara & Kurohashi, 2006)
due to incorrect usage of a case marker, our system gives feedback suggesting a more
appropriate case marker (cf. Figure 1). The player’s objective is to achieve a higher score by
constructing basic simple sentences and complex sentences with contingent relations.

2.2 Method for Building Word Card Sets

656

In order for the game to be more fun, it is preferable that players can construct several simple
and complex sentences from a given word card set. To guarantee this, we focus on core
events (Omura et al., 2020; Omura & Kurohashi, 2022).
Core events are defined as the high-frequency predicate-argument structures acquired
from case frames (Kawahara & Kurohashi, 2006); that is, each core event consists of one
predicate and a few of its frequently co-occurring arguments and case markers. We adopt
core event pairs with contingent relations such as “㞵ࡀ㝆ࡿЍ㛗㠐ࢆᒚࡃ (it rainsɦwear rainboots)”, which can be extracted at scale from the Kyoto University Commonsense Inference
dataset (cf. https://nlp.ist.i.kyoto-u.ac.jp/EN/?KUCI).
Our method is to automatically generate word card sets from core event pairs. It
roughly consists of the following three steps (cf. Figure 2).

Figure 2. Overview of the method for building word card sets. Green rectangles and orange
rounded rectangles correspond to noun and verb/adjective cards, respectively.
Step 1: Filter Core Event Pairs by Word Difficulty. First, we adjust vocabulary using
the Japanese word difficulty database (Mizutani et al., 2019), considering the target users are
elementary school students. This database contains 26k words of basic Japanese vocabularylabeled with their average acquisition time using crowdsourcing. Acquisition time is regarded
as word difficulty and expressed as a number from 1 to 5 (cf. Table 1).
Table 1. Correspondence between numbers and acquisition times in the Japanese worddifficulty database (Mizutani et al., 2019). ES and JHS stand for elementary school and juniorhigh school, respectively.
Number1
2
3
4
5
Acquisition time Before ES Early ES Late ES After JHS
Never seen or heard
In this study, we set the following threshold conditions.
Easy (for early elementary school students): Maximum word difficulty of words in a core
event pair does not exceed 2.0.
zMedium (for middle elementary school students): Average word difficulty of words in acore event pair exceeds 1.5, and maximum word difficulty does not exceed 2.5.
z
Hard (for late elementary school students): Average word difficulty of words in a core
event pair exceeds 2.0.
We exclude core event pairs not satisfying a threshold condition. We set a strict upper boundbut allow easy words to get mixed in harder word card sets to smooth the change in difficulty.
Step 2: Select Core Event Pairs. Second, we select five core event pairs from theones satisfying a threshold condition. Specifically, we take one core event pair (hereafter,seed) and randomly select the other four pairs that share the former or latter core event of theseed. If we fail to get five pairs, including a seed, we skip generating a word card set.
Step 3: Generate a Word Card Set. Lastly, we generate a word card set by breaking
down the five pairs selected in the previous step into predicates and arguments. If we obtainsix nouns and five or six verbs/adjectives after de-duplication, we regard the words as a wordz657

card set. In case that six verbs/adjectives are obtained, we replace the least frequentverb/adjective with a wildcard.

2.3 Careful Examination of Core Event Pairs and Building Word Card Sets
We must be extremely careful not to give inappropriate questions as an educational
application. Accordingly, we request two linguistic experts to manually classify the core event
pairs used for word card sets into the following categories.
z
Valid: A core event pair has contingent relation.
z
Invalid: A core event pair has no contingent relation.
z
Inappropriate: A core event pair contains educationally inappropriate expressions.
We use the “valid” core event pairs for generating word card sets.
Prior to the examination, we excluded core event pairs containing minor case particles
or words unregistered in the Japanese word difficulty database. As a result, 9k core event
pairs were left and examined. We also automatically assigned reading to each word andconjugated forms to each predicate using the Japanese morphological analyzer, Juman++(Morita et al., 2015; Tolmachev et al., 2018). We asked the experts to correct auto-assigned
readings in addition to the examination.
As a result, we obtained 4,362 valid, 3,560 invalid, and 1,120 inappropriate core event
pairs; then, we generated word card sets from the verified core event pairs according to themethod described in Section 2.2. Finally, we obtained 99 easy, 594 medium, and 284 hard
word card sets, achieving a size that is sufficient for solo play.

2.4 Automatic Scoring
When players tap a scoring button, the information on how cards and marks are placed is sent
to a back-end server and automatically scored. The automatic scoring is performed in thefollowing three steps.
Step 1: Recognize Simple and Complex Sentences. A simple sentence isrecognized as a sequence of a verb card at the end and noun cards in the rest connected bycase particle marks, and a complex sentence as two simple sentences connected by adiscourse connective mark. We consider all possible combinations of simple sentences.
Step 2: Score Each Sentence Automatically. Our policy is to prioritize simple
sentences whose predicate and argument frequently co-occur (i.e., idiomatic) or whose length
is longer. The score of each simple sentence is determined based on the number of examples
in the Japanese case frames (Kawahara & Kurohashi, 2006). Specifically, regarding the caseframe ݂ܿ where examples are found, we compute a score of each argument and case pairሺܽǡ ܿሻ with the following function ܵ and sum it up.
݂௔ǡ௖ǡ௖௙
ܵሺܽǡ ܿǡ ݂ܿሻ ൌ ͲǤͷ ൅ ͲǤͷ ൈ ሺͳǡൈ ͷሻ݂௖௙where ݂௔ǡ௖ǡ௖௙ is the frequency of argument ܽ in case ܿ of case frame ݂ܿ , and ݂௖௙ is thefrequency of case frame ݂ܿ.
Each score is converted into a symbol for further interpretability. Specifically, a simple
sentence is graded as “˝˝” if the score is 1 or more, as “˝” if the score is between 0 and 1,
and as “?” if the score is 0. Regarding complex sentences, they are graded as “˝˝˝” if they
contain one of the verified core event pairs; otherwise, as “?”. When examining thecontainment relation, we take into account the polarity of negation of a predicate.
Step 3: Generate Feedback based on the Results. The following feedback isgenerated from the sentences graded as “?” depending on the reasons behind them.
z
There is a more appropriate case marker that allows for the sentence to achieve a score.
z
The sentence is grammatically incorrect (e.g., the sentence starts with a verb/adjective,
a verb/adjective depends on a noun, and so on.)z
The sentence is not found in language resources.
Regarding the third point, we expect to collect unknown contingent relations through an error
reporting function and thus improve our evaluation system.

658

3. User Study
We developed iOS, Android, and web applications of Kotoba-musubi and conducted a user
study to assess engagement.

3.1 Settings
We recruited 80 pairs of elementary school students and their parents across Japan and had
the children play the game for an hour in total over two days. 80 students consist of 10 boys
and 10 girls each from third to sixth grade in elementary school. In order to let them play freely,
we neither set their quota nor specify the difficulty of questions they tackled. After playing thegame, they answered the questionnaire described in Table 2.
Table 2. Main items and options of the questionnaire answered by the participating children.
Regarding the second and subsequent items, the option with a larger number is preferable.
Do you like writing compositions?
5. Like
4. Like a little
3. Neither 2. Dislike a little
1. Dislike
Did you enjoy playing Kotoba-musubi?
5. Enjoyed 4. Enjoyed a little
3. Neither 2. Didn’t enjoyed a little
1. Didn’t enjoy
Do you want to continue to play Kotoba-musubi?
5. Yes
4. Yes, a little
3. Neither 2. No, a little
1. No
Which do you think Kotoba-musubi is: “game” or “study”?
5. Game
4. A litte game
3. Neither 2. A little study
1. Study
Do you think it was worth playing Kotoba-musubi?
2. Yes
1. No

3.2 Results and Discussion
Figure 3 shows the aggregate results of the questionnaire. We can see that elementary school
students tend to dislike writing compositions, as mentioned in Section 1. Despite thisdisadvantageous situation, 70% of the participating children enjoyed Kotoba-musubi, and 90%
answered it was worth playing the game. In addition, 70% expressed their will to continue toplay the game, which suggests that it is a good introduction to studying Japanese writing.

Figure 3. Aggregate results of the questionnaire described in Table 2. The numbers in the
figure represent those of children who chose the option.
We also investigated the number of children who disliked writing compositions but
enjoyed the game. We found that 18 of 37 children who disliked writing compositions enjoyed
the game. The result supports effectiveness in reducing an aversion to writing compositions.
The excerpted feedback comments from the participating children are listed below.
z+ It was fun and refreshing to create sentences using words I didn’t usually use.
z
+ I’d honestly like to see the game introduced to school tablets.
z± It will be more fun if we can compete with our friends for higher scores.
z− I couldn’t fully understand how to play the game.
z− It was hard to get sentences scored, which made me frustrated.
While many children noted they enjoyed themselves, there were also several comments thatit took time to understand how to play, which raises the need for improving a tutorial.

659

The user study also revealed some current issues of Kotoba-musubi. For instance,
there is room for improvement in reducing the feeling of studying while playing the game. As
one of the feedback comments suggests, we need to enhance the enjoyment of our application
by introducing game elements such as a match game system. Another issue is the quality of
automatic scoring. One of the possible remedies is to incorporate neural language models into
automatic scoring for more flexible evaluation. We will explore how to utilize neural language
models considering the computational cost.

4. Conclusion
We proposed an AI educational game for elementary school students to study Japanesewriting, which fully utilizes existing language resources such as case frames, a word difficulty
database, and a commonsense inference dataset. We also developed smartphone and web
applications of the game and carried out a user study involving 80 pairs of elementary school
students and their parents to assess it. The results of the user study demonstrated theeffectiveness in reducing an aversion to writing compositions. We expect elementary school
students to develop their vocabulary and reasoning skills in a ludic manner with the game.
The link to the game and the supplementary material of this paper are available athttps://nlp.ist.i.kyoto-u.ac.jp/EN/?Kotobamusubi.
We will address the remaining issues referring to the feedback comments and further
investigate long-term educational effects such as vocabulary development. We also consider
collecting unknown contingent relations through error reports.

Acknowledgements
This work was supported by the Japan Kanji Aptitude Testing Foundation. The first author was
also supported by Information/AI/Data Science Doctoral Fellowship of Kyoto University and
Grant-in-Aid for JSPS Fellows #22J15958.

References
National Institute for Educational Policy Research (2008). Survey on a particular subject (Japanese)
results. https://www.nier.go.jp/kaihatsu/tokutei/04002010000004000.pdf (in Japanese)
Ritsumeikan University Library (2017). Detail of Reference Example. National Diet Library, Japan.
https://crd.ndl.go.jp/reference/detail?page=ref_view&id=1000209543 (in Japanese)
Vandercruysse, S., Vandewaetere, M., & Clarebout, G. (2012). Game-Based Learning: A Review on
the Effectiveness of Educational Games. Handbook of Research on Serious Games as
Educational, Business and Research Tools. vol. 1. 628-647.
Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. MIT Press.
Kawahara, D., & Kurohashi, S. (2006). Case Frame Compilation from the Web using HighPerformance Computing. Proceedings of LREC 2006.
Speer, R., Chin, J., & Havasi, C. (2017). ConceptNet 5.5: An Open Multilingual Graph of General
Knowledge. Proceedings of AAAI 2017.
Morita, H., Kawahara, D., & Kurohashi, S. (2015). Morphological Analysis for Unsegmented
Languages using Recurrent Neural Network Language Model. Proceedings of EMNLP 2015.
Tolmachev, A., Kawahara, D., & Kurohashi, S. Juman++: A Morphological Analysis Toolkit for Scriptio
Continua. Proceedings of EMNLP 2018: System Demonstrations.
Omura, K., Kawahara, D., & Kurohashi, S. (2020). A Method for Building a Commonsense Inference
Dataset based on Basic Events. Proceedings of EMNLP 2020.
Omura, K., & Kurohashi, S. (2022). Improving Commonsense Contingent Reasoning by Pseudo-data
and Its Application to the Related Tasks. Proceedings of COLING 2022.
Mizutani, Y., Kawahara, D., & Kurohashi, S. (2019). Construction of a Word Database based on
Educational Level Specific Words using Crowdsourcing Questionnaires. Proceedings of
NLP2019. https://www.anlp.jp/proceedings/annual_meeting/2019/pdf_dir/C6-4.pdf (in Japanese)660

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Impact of Digital Game-based
Learning with a Mathematical Game
Application on Calculation Abilities ofGrade 4 Students
Y濝濢-Bei LIUa*, Alex Wing Cheung TSEb*
Faculty of Education, The University of Hong Kong, Hong Konga*u3598295@connect.hku.hk; b*awctse@hku.hk
Abstract: Calculation abilities are essential in elementary school mathematics andlearning with game applications have been demonstrated to improve student’s learningof mathematics, which is conducive to developing their calculation abilities. As one of
the core mathematical skills, calculation abilities can be improved through continuous
calculation practice. However, there is currently just a little research on the effects ofusing digital game-based learning (DGBL) with a mathematical game application inelementary school for developing students’ calculation abilities. Therefore, 78 students
participated in this quasi-experimental study to evaluate the possible influence of DGBL
with a mathematical game application “Oral Math Hero” through iPads on thecalculating abilities of fourth-grade students in a mainstream school in Mainland China.
A mathematical game application was integrated into a four-week classroom activity
for the experimental classes, with pre-test and post-test utilizing a standardized
calculation abilities test: Abilita diCalcoloz Calculation Abilities-Memory and Training
Group 6-11 (Cornoldi et al., 2002) for both experimental and control groups. Data
analysis results using ANOVA showed that there were significant differences instudents’ calculation abilities when learning with the mathematical game application on
iPad in mathematical classrooms, with significant differences in overall calculation
abilities between the experimental group (n=40) and the control group (n=38) in grade
4. In other words, we found that after practicing calculations with the mathematicalgame application, students were more likely to achieve better calculation abilities,
especially in terms of faster calculation speed, and lower error rates. However, there
were no significant differences in numerical knowledge, learning with this mathematicalgame application might not lead to acquiring more mathematical knowledge. This study
provides a realistic perspective for elementary mathematics educators and teachers tounderstand the potential of learning with the mathematical game application: it can be
an effective tool to improve the calculation abilities of students in grade 4. The second
stage of this project is to explore the reasons behind the results of the study, revealing
possible factors of DGBL with the mathematical game application that may facilitate
certain aspects of calculation abilities. Further suggestions are made to integrate DGBL
into the elementary mathematics classroom.
Keywords: Digital game-based learning, calculation abilities, mathematical games

1. Introduction
Students’ calculation refers to the abilities to understand patterns and relative quantities
between numbers and to perform operations between numbers (addition, subtraction,
multiplication, and division) in a more flexible way in mathematics (Feigenson et al., 2004;
Tall & Dehaene, 1998). They are intrinsically important for mathematical performance at theelementary level (Cowan et al., 2011). Similar to elementary school mathematics curricula indifferent leading countries, grade 4 students have to master four operations (addition,
subtraction, multiplication, and division) according to the latest curriculum standard in
Mainland China, and they need to keep practicing calculation abilities to find easier solutions(Ministry of Education of the People’s Republic of China, 2022). Students’ performance and661

efficiency in calculation can be improved through constantly repeated practice (Imbo &Vandierendonck, 2008). However, repeated paper and pencil exercises are generally boringfor students, and too many repetitions can make them lose interest to continue the practiceand thus not effective to improve their calculation abilities (Chen et al., 2020). It can result inpoorer mathematical skills and fewer arithmetic scores, which can have an impact on
students’ future performance in mathematics (Núñez Castellar et al., 2014).
Digital Game-based Learning (DGBL) is a learning approach that uses digital games
as a means of delivering educational material. The use of mobile game applications in theclass can help students to participate more actively in educational activities and achieve
higher instructional goals and outcomes (Vasiliou & Economides, 2007; García-Bárcena &García-Crespo, 2007) because of its funny nature. Students are more likely attracted toparticipate in and thus learn and acquire new knowledge and skills during the games (Corti,
2006). As past research has found that using DGBL in mathematics classes encouraged
students to stay on track and increased their engagement with the learning process (Jabbar& Felicia, 2015; Tsai et al., 2012). However, it is not known whether the use of these gameapplications can improve students’ calculation abilities in mathematics. Besides, prolonged
use of digital games for study may lead to game addiction, which can generate adverse
effects on students’ academic performance and affect their physical and mental health(Eyimaya & Irmak, 2021). Thus, many past research on calculation abilities focused on
cognitive factors and the relationship between previous academic skills and interests (e.g.,
Maedamuerk et al., 2018, Commodari & Di Blasi, 2014), or focused on kindergarten
students’ numeracy (Tajudin et al., 2015; Zhu & Chiu, 2019), but less on the effects of digital
games.
Therefore, the following research question is posed in this study: Does learning with amathematical digital game affect the calculation abilities of grade 4 students?

2. Literature Review
2.1 Research gap derived from literature review
Research has found that digital game-based learning can be a promising educational
approach that can help students easier to achieve targeted learning outcomes (Wang et al.,2022) and motivate students to learn because it generally provides timely feedback on task
evaluation (Mathrani et al., 2016), promotes students’ interest in learning and increases
classroom participation (Ke, 2008; Tsai et al., 2012). However, other research on DGBL are
still inconclusive in this issue. For instance, there is no significant evidence that DGBL can
improve students’ understanding of mathematical skills (Ke, 2008). Or Young et al. (2012)
stated that limited data is supporting the validity of digital games in the field of mathematics.
According to recent studies, calculation abilities are important for students’mathematics learning and even affect their later mathematics performance in secondary andpost-secondary schools (e.g., Cahoon et al., 2021). However, previous studies on
calculation abilities more tended to focus on factors such as attention, including selective
visual attention and the ability to suppress distracting information, and how they affect theaccuracy of verbal calculations in second and third-grade students (e.g., Commodari & Di
Blasi, 2014; Bigozzi et al., 2021). Other studies have explored the impact of playenvironments on elementary school students’ flexible number knowledge and mathematicalfluency, as well as the relationship between calculation and previous skills and interests(Brezovszky et al., 2019; Maedamuerk et al., 2018). There are only little studies on theeffects of digital games on students’ calculation abilities at the elementary level though there
are studies about the speed and accuracy of calculation for first and second-grade students(Núñez Castellar et al., 2014; van der Ven et al., 2017; Núñez Castellar et al., 2015).
To sum up, based on the literature review, there is still a research gap: few studies
have examined the effects of digital game-based learning on fourth graders’ calculation
abilities.

662

2.2 Theoretical framework
McCloskey et al. (1985) made a distinction between calculation abilities and the ability tocomprehend as well as that of generating numbers. According to their model, the calculation
system involved processing operational symbols, retrieving basic arithmetic facts, andperforming calculated operations, in addition to numerical processing. These elements
involved varying degrees in both mental and written calculations. Furthermore, any
calculation work requires a certain level of number comprehension and generation(Commodari & Di Blasi, 2014). Based on Computing System Model, calculation abilities canbe divided into three main dimensions: numerical knowledge, calculation accuracy, andcalculation speed (Cornoldi & Lucangeli, 2004).
According to Dehaene et al. (1990), numerical knowledge is the set of abilities andknowledge that enables a student to comprehend numerical quantities and theirconversions. A student’s ability to perform mathematical calculations quickly and accurately
can be defined and determined by the concept of calculation speed and calculation accuracy
respectively (Jordan & Montani,1997). Based on these three key factors, Cornoldi et al.
(2002) designed a standardized calculation abilities test: Abilita diCalcoloz Calculation
Abilities- Memory and Training Group 6-11 (AC MT 6-11).
The strength of this framework is its ability to take the abstract concept of calculation
abilities and visualize it into three dimensions for concrete measurement, thus performing astandardized proficiency test of whether students’ calculation abilities have changed. As
previously highlighted by Commodari and Di Blasi, (2014), any computation requires acertain level of calculation methods and knowledge and emphasizes the ability to perform
calculations quickly and accurately.
In this study, three sub-abilities of calculation abilities as Figure 1 shows (calculation
speed, calculation accuracy, and numerical knowledge) were used as the theoreticalframework for a more accurate measurement of students’ calculation abilities. AC MT 6-11
thus was adopted as an instrument of this study.

Figure 1. Framework for Calculation Abilities

3. Methodology
3.1 Procedures and ethical measures
The study was conducted in a school in Shenzhen, a southeast city in Mainland China
covering Grade 1 to 9, and consent were obtained from the participating students and theirparents, the mathematics teacher, and the school principal before the study was663

implemented. Since there are different classes for each grade in this school, the study usedcluster sampling, with one class in grade 4 randomly selected as an experimental group(n=40). For the control group, another class taught by the same mathematics teacher in thesame grade was selected as the control group (n=38). A total of 78 students participated in
the study.
In this study, a pre-and post-test quasi-experimental approach was used, as shown in
Figure 2. Three days before starting and after ending the intervention, all students took a
pre-test and post-test of the calculation test (i.e. AC MT 6-11). A 40-minute teacher training
was conducted before the pre-test to ensure that the teacher understood the purpose andprinciples of the study. Both the pre-test and post-test were completed in classrooms at thetargeted school, with proctoring and marking completed by a mathematics teacher in thesame grade level who was not teaching in the participating classes, and a 20-minute teacher
training was held with the marking teacher to agree on the scoring requirements and rules
for the calculation test before scoring the test.

Figure 2. The Overall Research Design.
Before the main study, two additional teachers and six students from grade 4 wereinvited to participate in the pilot study to collect their opinions on the research instrument.

3.2 Intervention
3.2.1 The adopted mathematical game application“Oral Math Hero” was adopted in the intervention of this study, and it was designed forelementary school students to practice mathematics in a fun way. As shown in Figure 3,
students could choose the range of calculations and operations they want to practice. The
game was based on players playing as different heroes, who need to fight monsters by
answering questions then they got progressively harder. Students who gave correct answers
could earn gold coins, and five consecutive correct answers unleashed a big move to defeat
the monster. Gold coins could be used to buy higher-level hero characters. After each“battle” (stage), there was task feedback and a leaderboard to motivate students.

664

Figure 3. Some interface of Oral Math Hero.
3.2.2 Pedagogy
The experimental group experienced the intervention which was based on the pedagogicalmodel of Play Curricular Activity Reflection Discussion (PCaRD) (Foster & Shah, 2015). The
PCaRD model consists of four phases (gameplaying phase, classroom activity phase,reflection phase, and discussion phase) to help teachers effectively integrate digital games
into their instruction in the classroom (Denham, 2019). During the month-long game-based
intervention, students in the experimental group were required to engage in a 15-minuteactivity of the mathematical game “Oral Math Hero” three times a week on iPads provided by
the school, while the control group maintained the traditional classroom practice activities
without learning with the mathematical game application.
Before the intervention, the researchers provided clear instructions to the teachers,
who designed lesson plans based on the PCaRD model, including the objectives, content,
and game-based activities. During the gameplaying time, every student learned with an
iPad, and during each 40-minute session, the teacher briefly delivered 20 minutes ofmathematics lesson content and divided the subsequent 15 minutes into four sections basedon the PCaRD model, as shown in Figure 4.

Figure 4. Intervention based on PCaRD model.
3.2.3 Calculation practice in the control group
To gain a more comprehensive understanding of the impact of the intervention on calculation
abilities, while the experimental group implemented the classroom intervention of the665mathematical game “Oral Math Hero”, the control group implemented the traditional
calculation practice method, which is based on paper and pencil training, and performed oralcalculation practice or calculation training in exercise books at the same frequency andduration as the experimental group.

3.3 Instrument
As stated, Abilita diCalcoloz Calculation Abilities Memory and Training Group 6-11 (AC MT
6-11) was used as a calculation abilities test in this study, which was a standardized
calculation test that requires students to determine if students’ calculation abilities levels
changed before and after the four-week intervention and were expected to take 40 minutes.
The calculation abilities test included scores in three dimensions: numerical knowledge,
calculation speed, and calculation accuracy (Cornoldi et al., 2002). There were eight tasks inthis calculation abilities test: Written Calculation; Size Comparison, Word–Number
Transcoding, Numerical Facts, and Number Ordering; Mental Calculation; Written
Calculation2; Enumeration; and Numerical Facts (Cornoldi & Lucangeli, 2004). According toCornoldi and Lucangeli (2004), the calculation abilities test is a psychologically trustworthy
assessment of calculation abilities, and the sources of scores for the three dimensions ofcalculation abilities were shown in Table 1.
Table 1. Sources of Scores for Different Dimensions
Dimension
Calculation Speed

ScoresTime for answering Written
Calculation, Mental Calculation,
Enumeration, Written Calculation2
Combinedscore:
Size
Comparison,Word–number
transcoding,
Numerical
facts,
Number ordering
Total Error Scores

Numerical Knowledge

Calculation Accuracy

3.4 Data analysis
The pretest scores of the experimental and control classes in both grades were subjected toseparate independent sample T-tests to ensure that there was no significant difference in thebasis of calculation abilities between the two classes. After that, a single-factor analysis ofvariance was conducted for all pretest and post-test scores. Therefore, ANOVA (including
Levene’s test of the quality of error variances) with a confidence interval of 0.95 is chosen toeliminate the effect of the intervention on the results and was followed by a normal
distribution and homogeneity of the regression slopes to ensure that there was no interaction
between the pretest scores and the intervention (“Oral Math Hero” game application). The
above data are analyzed using IBM SPSS 28.0.

4. Results and Findings
4.1 Descriptive analysis
Cronbach's alpha was calculated using the pre-test results to ensure the reliability andreliability of the test before conducting the post-test. The reliability analysis showed that
Cronbach's alpha was greater than 0.7 for all factors (ߙCS = 0.727, ߙCA = 0.841, ߙNK =0.769), confirming that the standard calculation abilities test is a reliable research instrument.
The results of the independent samples T-test (p=0.930<0.05) showed that there was no
significant difference in the pre-test scores of the experimental and control groups in grade666

4, which meant that the two groups had a similar basic level of calculation abilities before thestart of the study. The normality test was performed on the pre-test and post-test data of thecalculation abilities test, and since the sample size of each group was <50, the results
according to the Shapiro-Wilk test (p Pre-CS = 0.056; p Post-CS = 0.095; p Pre-CA = 0.154; p Post-CA= 0.085; p Pre-NK = 0.280; p Post-NK = 0.050) were all greater than 0.05, so the collected data
conformed to a normal distribution. The homogeneity of variance was tested using Levene’s
test, and the results (p Pre-CS =0.427; p Post-CS = 0.882; p Pre-CA = 0.237; p Post-CA = 0.740; p Pre-NK= 0.573; p Post-NK = 0.622) suggest that the differences in the overall variance of the groups
are small.
The descriptive analysis of the pre-test and post-test results of the calculation abilities
test was shown in Table 2. Students in the experimental group demonstrated a significant
increase in calculation accuracy and calculation speed, whereas students in the control
group demonstrated a slight increase in calculation accuracy but a decrease in calculation
speed and numerical knowledge compared to the pretest. Meanwhile, students in theexperimental group drastically dropped their numerical knowledge.
Table 2. Descriptive Statistics of Calculation Abilities Test in Pretest and Posttest
澳 Dimension

Pretest3.88

Std.
Deviation
澳
3.560

6.42

9.081

6.58

11.387

15.48
18.34

2.828
1.805

38.58
39.18

5.679
3.303

16.18
17.00
澳
33.55
32.55

2.688
2.857
澳
0.959
2.177

Mean
Calculation Accuracy
Experimental Group
Control GroupCalculation Speed
Experimental Group
Control Group
Numerical Knowledge
Experimental Group
Control Group

Posttest
Mean

Std.
Deviation

6.03

11.173

4.2 Analysis of variance (ANOVA)
According to the results of the T-test, there was no significant difference between the pretest
scores of the experimental group and control group (p=0.930>0.05), indicating that the twogroups of the fourth grade had similar bases. Therefore, a single-factor ANOVA was
conducted to investigate whether there was a significant difference between the data of theexperimental and control group.
Table 3. ANOVA results of Calculation abilities in Experimental and Control GroupFp
Mean Square
Overall Calculation Abilities
Class
2.708
0.004*
126.323
Calculation Accuracy
Class
3.908
0.004*
137.894Calculation Speed
Class
28.1740.001*
160. 109
Numerical Knowledge
Class
6.972
0.063*
19.385
The results (Table 3) showed that overall speaking, students in the experimental group
had significantly higher calculation abilities than those in the control group (p=0.004).

667

Specifically, among the three dimensions of calculation abilities, calculation accuracy(p=0.004) and calculation speed (p=0.001) were significantly different between the twogroups in the posttest. Therefore, it was possible to draw a conclusion that digital gamebased learning with Oral Math Hero on iPad significantly improved the calculation abilities ofthe experimental group of grade 4 students, especially the students’ calculation speed andcalculation accuracy. However, no significant difference was found between the students’ability in the experimental and control classes in terms of numerical knowledge (p=0.063).
Learning with this mathematical game application might not lead to acquiring moremathematics knowledge.

5. Discussion and Conclusion
Calculation accuracy, calculation speed, and numerical knowledge as three objective
measurements were considered to measure the impact of digital game-based learning with amathematical game application on the calculation abilities of grade 4 students (Cornoldi &Lucangeli, 2004). Specifically, we predicted that DGBL with the mathematical gameapplication, in terms of calculation accuracy, calculation speed, and numerical knowledge,
would be as effective as normal mathematics class activities in improving the calculation
abilities of grade 4 students, and the experimental group would perform better than thecontrol group. Consistent with this hypothesis, the results of the post-test showed that theexperimental group had significantly higher overall calculation abilities test scores andscores in terms of calculation accuracy and calculation speed than the control group. Suchfindings generally confirmed some studies on the use of mathematical games inmathematics classrooms (e.g., Núñez Castellar et al. 2014; van der Ven et al. 2017;
Castellar et al. 2015), suggesting that DGBL with the mathematical game can improvecertain mathematical skills by increasing students’ calculation accuracy and speed of mental
computation. The findings are also in line with those of Miller and Robertson (2011) thatelementary school students who played mathematical games had twice the calculation
speed of control group students who did not play.
However, an unexpected finding occurred in this study: the numerical knowledge score
gained in the post-test was lower for the experimental group than in the pre-test. Morespecifically, the findings of this study did contradict some previous studies (Brezovszky et al.,2019) that DGBL with mathematical game application did not improve students’ numericalknowledge significantly. Three potential explanations for the observed phenomenon, each ofwhich is not exclusive of the others. Firstly, it is plausible that the “Oral Math Hero” game did
not affect numerical knowledge to a significant extent since it provided tasks of a lesscomplex and purely arithmetic nature. Secondly, students might not think deeply aboutcertain calculation rules, such as the multiplication exchange law or the distributive law, inorder to complete the task quickly in the exercise. Thirdly, the superior calculation methods
were not presented in the task feedback or the answer situation screen of the game, so
students had no chance to achieve more numerical knowledge in this game.
There were several limitations to this study. Firstly, although we have explored theeffects of DGBL on calculation abilities by standardized calculation abilities tests, howcalculation abilities are altered by digital games, how it varies with the type and content ofdigital games, and how the afterschool environment may affect calculation abilities remains
unanswered. Future research would be expected to examine how DGBL affects students’calculation abilities, taking the afterschool environment and the type and content of games
into consideration. Secondly, the duration of the intervention (four weeks, 2-3 times perweek) was relatively short and it was only used in the classroom, so its impact on students’calculation abilities might have been limited. In addition, the duration of this study lasted only
one month, which enabled the study to explore the short-term impact of DGBL withmathematics games on students’ calculation abilities but lacked a longer period ofinvestigation to explore whether this impact would change according to the time change.
Longitudinal trials and tracking of students’ calculation performance over time might be used
to offer an understanding of potential long-term impacts on students’ calculation abilities.

668

Thirdly, the sample size of this study was relatively small, only 78 students, and the fact thatall data were collected through calculation abilities tests in a mainstream school in Mainland
China might have led to an under-representation of the study, limiting the generalizability of
the findings to other educational settings or groups of students. In addition, the participantsof this study were mainly focused on grade 4 students, with no attempt to cover the entire
elementary school level or include students' and teachers’ interviews, lacking multiple data
sources to provide triangulation. A study on a larger scale involving after-schoolinterventions in more districts with more sources of data such as classroom observations
and individual interviews could be considered. Fourth, the study concentrated on theinfluence of DGBL with a mathematical game on students’ calculation abilities while ignoring
the impact of the game on students’ basic mathematical skills, such as problem-solving,
spatial thinking, numeracy, and sense of amount, as well as their motivation. Future
research can investigate whether mathematical game impact students’ motivation and othercomponents of mathematics learning.
In conclusion, this study showed that DGBL with a mathematical game application,compared to traditional calculation practice, could improve the calculation abilities of fourth
graders, especially in terms of calculation speed and accuracy. Nevertheless, to present acomprehensive picture of how learning with a mathematical digital game affect calculation
abilities, the data of the second-stage qualitative research is now analyzed to explain in whatways students were affected by game settings or elements. After that, implications for future
DGBL with a mathematical game application can be proposed and all these will bepresented in a forthcoming paper.

Acknowledgments
We would like to thank all the people who prepared and revised previous versions of thisdocument.

References
Bigozzi, L., Pezzica, S., & Malagoli, C. (2021). The contribution of attentional processes to calculation
skills in second and third grade in a typically developing sample. European Journal of
Psychology of Education, 36(4), 965–988. https://doi.org/10.1007/s10212-020-00515-z
Brezovszky, B., McMullen, J., Veermans, K., Hannula-Sormunen, M. M., Rodríguez-Aflecht, G.,
Pongsakdi, N., Laakkonen, E., & Lehtinen, E. (2019). Effects of a mathematics game-basedlearning environment on primary school students’ adaptive number knowledge. Computers &Education, 128, 63–74. https://doi.org/10.1016/j.compedu.2018.09.011
Cahoon, A., Gilmore, C., & Simms, V. (2021). Developmental pathways of early numerical skills
during the preschool to school transition. Learning and Instruction, 75, 101484.
https://doi.org/10.1016/j.learninstruc.2021.101484
Chen, S., Jamiatul Husnaini, S., & Chen, J.-J. (2020). Effects of games on students’ emotions
oflearning science and achievement in chemistry. International Journal of Science
Education,42(13), 2224–2245. https://doi.org/10.1080/09500693.2020.1817607
Cornoldi, C., Lucangeli, D., & Bellina, M. (2002). AC-MT Test: Test per la valutazione delle difficoltà dicalcolo [The AC-MT arithmetic achievement test]. Trento, Italy: Erickson.
Cornoldi, C., & Lucangeli, D. (2004). Arithmetic education and learning disabilities in Italy. Journal of
Learning Disabilities, 37(1), 42–49.
Cowan, R., Donlan, C., Shepherd, D.-L., Cole-Fletcher, R., Saxton, M., & Hurry, J. (2011). Basiccalculation proficiency and mathematics achievement in elementary school children. Journal of
Educational Psychology, 103(4), 786–803.
Commodari, E., & Di Blasi, M. (2014). The role of the different components of attention on calculation
skill. Learning and Individual Differences, 32, 225–232.
Corti, K. (2006). Games-based Learning; a serious business application. Informe de PixelLearning,
34(6), 1-20.

669

Dehaene, S., Dupoux, E., & Mehler, J. (1990). Is numerical comparison digital? Analogical andsymbolic effects in two-digit number comparison. Journal of experimental Psychology: Human
Perception and performance, 16(3), 626.
Denham, A. R. (2019). Using the PCaRD digital game-based learning model of instruction in the
middle school mathematics classroom: A case study. British Journal of Educational Technology,
50(1), 415–427.
Eyimaya, A. O., & Irmak, A. Y. (2021). Relationship between parenting practices and children's screen
time during the COVID-19 Pandemic in Turkey. Journal of pediatric nursing, 56, 24-29.
Feigenson, L., Dehaene, S., & Spelke, E. (2004). Core systems of number. Trends in Cognitive
Sciences, 8, 307–314. https://doi-org.eproxy.lib.hku.hk/10.1016/j.tics.2004.05.002
Foster, A., & Shah, M. (2015). The Play Curricular Activity Reflection Discussion Model for GameBased Learning. Journal of Research on Technology in Education, 47(2), 71–88.
García-Bárcena, J., & García-Crespo, Á. (2007). Game based learning: a research on learning
content management systems. interaction, 9(11), 12.
Imbo, I., & Vandierendonck, A. (2008). Practice effects on strategy selection and strategy efficiency insimple mental arithmetic. Psychological Research, 72(5), 528–541.
https://doi.org/10.1007/s00426-007-0128-0
Jabbar, A. I. A., & Felicia, P. (2015). Gameplay Engagement and Learning in Game-Based Learning:
A Systematic Review. Review of Educational Research, 85(4), 740–779.
Jordan, N. C., & Montani, T. O. (1997). Cognitive arithmetic and problem solving: A comparison ofchildren with specific and general mathematics difficulties. Journal of learning disabilities, 30(6),624-634.
Ke, F. (2008). A case study of computer gaming for math: Engaged learning fromgameplay? Computers & Education, 51(4), 1609–1620.
Maedamuerk, K., Kikas, E., & Palu, A. (2018). Calculation and word problem-solving skill profiles:
Relationship to previous skills and interest. Educational Psychology, 38(10), 1239–1254.
https://doi.org/10.1080/01443410.2018.1495830
Mathrani, A., Christian, S., & Ponder-Sutton, A. (2016). PlayIT: Game based learning approach forteaching programming concepts. Journal of Educational Technology & Society, 19(2), 5-17.
McCloskey, M., Caramazza, A., & Basili, A. (1985). Cognitive mechanisms in number processing andcalculation: Evidence from dyscalculia. Brain and cognition, 4(2), 171-196.
Miller, D. J., & Robertson, D. P. (2011). Educational benefits of using game consoles in a primaryclassroom: A randomised controlled trial. British Journal of Educational Technology, 42(5), 850864.
Ministry of Education of the People's Republic of China. (2022). Curriculum standards for compulsory
education in mathematics (2022 edition). Beijing Normal University Press.
Núñez Castellar, E., Van Looy, J., Szmalec, A., & de Marez, L. (2014). Improving arithmetic skills
through gameplay: Assessment of the effectiveness of an educational game in terms of cognitive
and affective learning outcomes. Information Sciences, 264, 19ȸ31.
Núñez Castellar, E., All, A., de Marez, L., & Van Looy, J. (2015). Cognitive abilities, digital games andarithmetic performance enhancement: A study comparing the effects of a math game and paper
exercises. Computers & Education, 85, 123–133. https://doi.org/10.1016/j.compedu.2014.12.021
Tajudin, N. M., Ali, S. R., & Idris, N. (2015). Assessing Numeracy Thinking Strategy for Year FourPrimary School Pupils. Asian Journal of Assessment in Teaching and Learning, 5, 1–12.
Tsai, F., Yu, K., & Hsiao, H. (2012). Exploring the Factors Influencing Learning Effectiveness in Digital
Game-based Learning. J. Educ. Technol. Soc., 15, 240-250.
The effects of video game playing on attention, memory, and executive control—ScienceDirect. (n.d.).
Retrieved 5 February 2023
van der Ven, F., Segers, E., Takashima, A., & Verhoeven, L. (2017). Effects of a tablet gameintervention on simple addition and subtraction fluency in first graders. Computers in Human
Behavior, 72, 200–207. https://doi.org/10.1016/j.chb.2017.02.031
Vasiliou, A., & Economides, A. A. (2007, July). Game-based learning using MANETs. In Proceedingsof the 4th WSEAS/IASME International Conference on Engineering Education (pp. 24-26).
Wang, L.-H., Chen, B., Hwang, G.-J., Guan, J.-Q., & Wang, Y.-Q. (2022). Effects of digital gamebased STEM education on students’ learning achievement: A meta-analysis. International
Journal of STEM Education, 9(1), 26. https://doi.org/10.1186/s40594-022-00344-0
Young, M. F., Slota, S., Cutter, A. B., Jalette, G., Mullin, G., Lai, B., ... & Yukhymenko, M. (2012). Our
princess is in another castle: A review of trends in serious gaming for education. Review of
educational research, 82(1), 61-89.

670

" !   !" " "   #" !
#" !"$  #" ! #"          
   
               *$  *$  

$%6.591(49%*1.1)9'+,+0)13/$."0+7'34+59*+0$
%*11.1(&6%$5+10$.!'%*01.1)9$%6.591(&6%$5+10'++0)13/$."0+7'34+59*+0$
%*11.1(&6%$5+10$.!'%*01.1)9$%6.591(&6%$5+10'++0)13/$."0+7'34+59*+0$
:H4A:5AH 87H 6A  (()*."%!$."-'('%%'$)$#!./$""$#!.*()# "$!( #  *)$# # #)') ) ##( $ $"%'#(+'(' (##$!#-%!$'-%!###'
#' # +!*) ,( (!) $!!$,# ) 0##$!$.##'#)")(#)')$#1%%'$),(*()$+!$%*)$#)+)()')# ()*#)( # ) ) # ) '( $*(# $# 0.#"( # #'.
$#+'($##!'(#1'$*)''$*#())+#((#(!).$))+)(,'-"#()*."%!$.&*#)))+)$$!(#&*!))+#)'+,( #!*# )'( # ()*#)( )$ "%'$+ ) )# %'$(( #
#()'*)$#! '",$'   '(*!)( #) ))   *)$# )+)(+!$%*(#) $!(##)!.##()*#)(#)'()###'#(*)(,#%%!#)#%')  )#()'*)$#!)+)(#)+!. %%! # !(('$$"( )) #)') (#) #&*'. # ##'# %')(+')$*'($))''$*#()%')%)#)'("$#()')"%'$+
!(('$$"$#)'$!#)""#"#),!()*#)($'(###'#(#%'$)( ($, #$)! %'$'(( '$* ) $%)"/)$# $ ) )#%'$((#()'*)$#!'",$' #$)'(%)((*(")'!($%')$#(#"#"#))()*.*!)")!.+!$%!.%')!*)$#())#)')(##'#%')(#)$(##&*'.!(('$$"(((('+((''#$''$#)!#(#)'(#'(''(





  (##$!#)')*)$# )+).(#$)'%*)"%$,-&87H64G<BA,6<8A68-86;AB?B:LA:<A88E<A:4A7&4G;8@4G<6F<F4A<@CBEG4AG
@84AFGB6H?G<I4G8FGH78AGF<AABI4G<BA4A7CE46G<64?45<?<G<8F ?G;BH:;E8F84E6;8EF;4I8
7<998E8AGHA78EFG4A7<A:F4A7C8EFC86G<I8FBAG;86BA68CG6BAABG4G<BA4A76;4E46G8E<FG<6FB9,-&87H64G<BA!H58?54A>(?<I4 &8EF<A'4G<BA4?7H64G<BA<E86GBE4G8-HE>8L
.L:HE G;8L4??CB<AGGBG;86B@@BA:B4?B9<AG8E7<F6<C?<A4EL<AG8:E4G<BAJ<G;4
5EB478E4A7J<78EF6BC8B9<AG8:E4G<BA+47?B99 HM8L  /4E<BHFFGH7<8F;4I8F;BJA
G;4G4@BA:G;89BHE7<F6<C?<A8FB9,-4A7&@4G;8@4G<6F4A7F6<8A684E8G;8CE<@4EL
FH5=86GFJ;<?8BG;8EFF8EI84F6BAA86G<A:?<A>FE8FH?G<A:<AG;8@4E:<A4?<M4G<BAB9G;8
8A:<A88E<A:7<F6<C?<A8 "AE868AGL84EF@4AL6BHAGE<8F;4I8<@C?8@8AG878A:<A88E<A:
87H64G<BAE89BE@F<A$ ><A78E:4EG8AG;EBH:;GJ8?9G;:E478F6;BB?F&BBE88G4?  '4G<BA4?+8F84E6;BHA6<?  BE8K4@C?84E8CBEGG<G?87O)E8C4E<A:4A7"AFC<E<A:G;8
HGHE8,-&0BE>9BE68B9@8E<64$ ,6<8A68-86;AB?B:LA:<A88E<A:4A7
&4G;8@4G<6F,-&PFH5@<GG87GBG;8)E8F<78AGB9G;8.A<G87,G4G8F8@C;4F<M87G;8A887

671

GB6H?G<I4G8FGH78AGFJ<G;4FGEBA:,-&546>:EBHA7GBE8@4<A6B@C8G<G<I8<AG;8:?B54?
FB6<8GL'4G<BA4?+8F84E6;BHA6<?
 "AG8:E4G<A:F6<8AG<9<6<ADH<EL4A78A:<A88E<A:
CE46G<68F<AGBF6<8A686?4FFEBB@FGB9BE@"AG8:E4G87 "AG8:E4G<I8,-&7H64G<BA;4F
586B@8BA8B9G;8<@CBEG4AGGE8A7F<AG;878I8?BC@8AGB9,-&87H64G<BAJBE?7J<78
 HM8L8G4?   -;8E89BE8BHEE8F84E6;J<??9B??BJF<KFG8CF F88<:HE8 +)63' 8F<:A-846;<A:&B78? G;8&B78?
+8F84E6;8EF7<I<787G;8BI8E4??9E4@8JBE>B9<AG8:E4G87,-&87H64G<BA<AGB9BHE
6B@CBA8AGF:B4?FBHG6B@8FA4GHE84A7F6BC8B9<AG8:E4G<BA4A7 <@C?8@8AG4G<BA'4G<BA4?
+8F84E6;BHA6<?  -;<FFGH7L47BCGF4A<AG8:E4G874CCEB46;54F87BAF6<8AG<9<6
8KC?BE4G<BA4A78A:<A88E<A:78F<:AGBFG<@H?4G8FGH78AGF<AG8E8FG<A8KC?BE4G<BA8A6BHE4:8
G;8@GB78F<:A4A76E84G88A:<A88E<A:CEB=86GF 4A7 4CC?L<AG8E7<F6<C?<A4EL>ABJ?87:8-;8G846;<A:@B78?J4F78I8?BC875LG;8<B?B:<64?,6<8A68FHEE<6H?H@,GH7L
,,<AG;8  F<AG;8.A<G87,G4G8F54F87BAG;86BAFGEH6G<I<FG?84EA<A:C8EFC86G<I89BE
F6<8AG<9<6<ADH<EL "G<A6?H78F9<I8FG4:8F8A:4:8@8AG8KC?BE4G<BA8KC?4A4G<BA 8?45BE4G<BA
4A7 8I4?H4G<BAGB9BE@4AHA78EFG4A7<A:B9F6<8AG<9<66BA68CGF "G;4F588AJ<78?LHF87<A
5<B?B:LC;LF<6F6;8@<FGELF6<8A684A7 BHG B9 F6;BB?F6<8A6887H64G<BA46G<I<G<8FHE4A
HE4A  4EEL   CEBCBF87477<A:4AO8A:<A88E<A:P FG4:8GBG;8G846;<A:
@B78?E8FH?G<A:<AG;8&B78?A:4:8KC?BE8KC?4<AA:<A88EAE<6; 4A7I4?H4G8 
A:4:84<@FGBFG<@H?4G8<AG8E8FG4A78A;4A68?84EA<A:C4EG<6<C4G<BA8KC?BE8 CEBI<78F
BCCBEGHA<G<8F9BE6BAFGEH6G<A:F8?9 >ABJ?87:8HA78EFG4A7<A:4A4?LF<F4A7 ;4A7F BA
BC8E4G<BA8KC?4<A CEBI<78FBCCBEGHA<G<8F9BEE89?86G<BA8KC?4A4G<BA@B7<9<64G<BA4A7
8?45BE4G<BAB9>ABJ?87:88A:<A88ECEBI<78FBCCBEGHA<G<8FGB4CC?L?84EA87>ABJ?87:84A7
G86;AB?B:LGBCE46G<688AE<6;CEB@BG8F<A 78CG;8KC?BE4G<BAB9>ABJ?87:84A7 8I4?H4G8
6;86>F4A78I4?H4G8F?84EA<A:BHG6B@8F$;48EBA<A:GL4F8G4?   78I8?BC874,-&87H64G<BA46G<I<GLJ<G;G;8G;8@8B9
O-8@C8E4GHE84A7"GF;4A:8FP HF<A:G;8&B78? -;8L<@C?8@8AG87 4DH4F<
8KC8E<@8AG4?78F<:A 4A79BHA7G;4GG;88KC8E<@8AG4?:EBHC46;<8I87;<:;8EF6BE8F<AG;8
GBC<6B9G8@C8E4GHE84A7<GF6;4A:8F %4<;H   HG<?<M87G;8&B78? GB78I8?BC4,-&87H64G<BA46G<I<GLJ<G;G;8G;8@8B9O*H476BCG8EFF8@5?LP -;8?84EA<A:
C8E9BE@4A68B9@<77?8F6;BB?FGH78AGFF;BJ87F<:A<9<64AG<@CEBI8@8AG !F<4B8G4?   <@C?8@8AG874DH4F< 8KC8E<@8AG4?78F<:A9BHA7G;4GHF<A:EB5BG 54F87CE46G<68FGB
78I8?BC4A46G<I<GLG;4G<A6BECBE4G87G;8@B78?64A <@CEBI88?8@8AG4ELF6;BB?FGH78AGFQ
?84EA<A:@BG<I4G<BA?84EA<A:C8E9BE@4A686B@CHG4G<BA4?G;<A><A:45<?<GL4A7;4A7F BA
45<?<GL -;8,-&87H64G<BA46G<I<G<8FHF<A:G;8&B78?J8E8<@C?8@8AG87J<G;G;<E7:E478FGH78AGF<A-;4<?4A7 9G8E 6?4FF;BHEFG;8FGH78AGF9BHA7<@CEBI8@8AGF<A
46478@<6C8E9BE@4A68F6<8AG<9<66E84G<I<GL4A7<AABI4G<BA45<?<G<8F#BA:?H86;40BE4CHA  A7G;8&B78? <A4CE46G<64?46G<I<GL<AIB?I<A:G;878F<:AB98::CEBG86G<BA78I<68F
J4F9BHA7GB;4I84CBF<G<I8<@C46GBAG;8G86;AB?B:<64?4GG<GH784A7G86;AB?B:<64?<ADH<EL
45<?<G<8FB9G;8C4EG<6<C4G<A:F86BA74ELF6;BB?FGH78AGF%<A8G4?   

672

-JBFGH7L ;LCBG;8F8FJ8E8CEBCBF87HG<?<M<A:G;8&B78?GB78I8?BC,-&
87H64G<BA46G<I<G<8F64A5889986G<I8?L4CC?<87<AG846;<A:CE46G<68 4A7 ,-&87H64G<BA
46G<I<G<8F78I8?BC87HF<A:G;8&B78? 64A8A;4A68FGH78AGF<AG8E8FG<A8A:<A88E<A:
?84EA<A: -;<FFGH7L8@C?BL87 G;8 &B78?GB78I8?BC4A778F<:A<AG8:E4G87,-&
87H64G<BA46G<I<G<8F4A7I4?<74G87G;8<E89986G<I8A8FFG;EBH:;<@C?8@8AG4G<BAE8FH?G<A:<A4
6HEE<6H?H@78F<:A@8G;B7B9CE46G<64?F<:A<9<64A68

 )!%
4G4 J4F 6B??86G87 HF<A: DH4AG<G4G<I8 GBB?F 4A7 DH4?<G4G<I8 <AG8EI<8J @8G;B7F  *H4AG<G4G<I8
74G4<A6?H787F6BE8FB9FGH78AGF8A:<A88E<A:78F<:ACEB=86GF4A7FHEI8LFBAG;8<E<AG8E8FG<A
8A:<A88E<A: 4A7 F6<8A68 FH5=86GF  *H4?<G4G<I8 74G4 <A6?H787 <AG8EI<8JF J<G; G846;8EF 4A7
FGH78AGF -;8E8F84E6;CEB68FF<F4F9B??BJF F88<:HE8 

-;<FFGH7LE86EH<G87C4EG<6<C4AGFG;EBH:;6B??45BE4G<BAJ<G;BA8HA<I8EF<GL6B@@HA<GL4
GHGBE<A:BE:4A<M4G<BA 4A7 4C4E8AGF:EBHC GBG4?B9FGH78AGF <A :E478 <A:E4784A7BA8G846;8EJ;BJ4F4F6<8A6887H64G<BA<AFGEH6GBE9EB@G;8GHGBE<A:
BE:4A<M4G<BA

+)63'  +8F84E6;)EB68FF?BJ6;4EG

673

   !*' !05'3'45 637'9-;8,-&"AG8E8FG,HEI8L<F4A <G8@F8?9 E8CBEGF64?8J<G;946GBEFB9<AG8E8FG<A:<A
F6<8A68G86;AB?B:L@4G;8@4G<6F8A:<A88E<A:4A7,-&64E88EF-L?8E 0BB78G4? "G8@FJ8E84AFJ8E87BA4 CB<AG%<>8EGF64?8G;4GE4A:8F9EB@ ABG4G4??<@CBEG4AGGB8KGE8@8?L<@CBEG4AG -;8EBA546;F?C;4<F58GJ88A 4A7     56&'05#13- %13+0) *''5-;8G;8)8E9BE@4A68 4F87I4?H4G<BA)4E89BHE946GBEFB96B@C?8G<BA984F<5<?<GL
48FG;8G<6F4A7<AABI4G<BA#<A8G4?   -;8)HF8F4 CB<AGF64?8 J<G;;<:;
FG45<?<GL4A7E8?<45<?<GL   05'37+'865.+0'4
-;8FGH7L<A6?H78F<AG8EI<8JFJ<G;G846;8EF4A7FGH78AGF -846;8E<AG8EI<8JFCE<@4E<?L9B6HF
BAG;8F<GH4G<BAF4A7<FFH8F8A6BHAG8E877HE<A:46GH4?G846;<A:CEB68FF8F6BI8E<A:GBC<6F
FH6;4FG846;<A:@8G;B7F4A78I4?H4G<BA@8G;B7F BE8K4@C?87BLBH6BAF<78EG;<F
G846;<A:@8G;B7GB58F<@<?4EBE7<998E8AG9EB@CE8I<BHF@8G;B7F0;4G4E8G;847I4AG4:8F
4A77<F47I4AG4:8F,GH78AG<AG8EI<8JF4<@GB8KC?BE8G;84CC?<64G<BAB9F6<8AG<9<6CE<A6<C?8F
4A7@4G;8@4G<64?>ABJ?87:8<A8A:<A88E<A:CE46G<68F7HE<A:G;846G<I<GL4FJ8??4FG;8
7<99<6H?G<8F8A6BHAG8E87<AG;86?4FFEBB@4A7BI8E4??F4G<F946G<BAJ<G;G;86BHEF846G<I<G<8F BE8K4@C?8J;4G7<99<6H?G<8F7<7LBH8A6BHAG8E7HE<A:G;8?84EA<A:CEB68FFBE8K4@C?8
45FGE46G6BA68CGFBE6;4??8A:<A:;4A7F BABC8E4G<BAF )?84F8CEBI<78FC86<9<68K4@C?8F ('!)&( "  " 
"AG;878I8?BC@8AGB9G;8&B78?9<I84FC86GFJ8E86BAF<78E87A887F4A4?LF<F6BAG8AG
78F<:AG846;<A:E8FBHE68FCEB68FF78F<:A4A78I4?H4G<BAB9BHG6B@8F -;8A887F4A4?LF<F<AIB?I874A4?LM<A:87H64G<BA4?E8FBHE68F4A7?84EA8EF -;8FGH7L47BCG876?4FFEBB@ 54F87
46G<I<G<8FJ<G;4E8?4G<I8?LF;BEG7HE4G<BA4A7?BJE8FBHE68E8DH<E8@8AGF <9G;4A7F<KG; :E478
FGH78AGFJ8E86;BF8ABI8EG;<E74A79BHEG; :E478FGH78AGF7H8GBG;8<E;<:;8E?8I8?FB9F8?9@4A4:8@8AGF><??F;4A7F BACE46G<64?45<?<G<8F4A7CEB9<6<8A6L<A@4G;8@4G<64?BC8E4G<BAF
4A7<A78C8A78AGG;<A><A:  BAG8AG78F<:A<AIB?I876BA7H6G<A:FHEI8LF4@BA:9EBAG?<A8G846;8EF4A76HEE<6H?H@
78F<:A8A:<A88EF9B??BJ875LCE8?<@<A4EL4A4?LF<FB9CBG8AG<4?G;8@8F .?G<@4G8?L4A6<8AG
;<A8F8GE4AFCBEG4G<BAI8;<6?8FFC86<9<64??LG;8F4<? CBJ8E8764EGJ8E8F8?86G874FG;8
G;8@89BEG;86HEE<6H?H@78F<:A -846;<A:E8FBHE68F<A6?H787G;878I8?BC@8AGB9G846;<A:;4A75BB>FFGH78AG
;4A75BB>F4A746G<I<GL@4G8E<4?F><GF -;8G846;<A:;4A75BB>:H<787G846;8EF<A
HA78EFG4A7<A:G;86HEE<6H?H@78F<:A6BA68CGFG846;<A:CEB68FF8F4A7?8FFBACE8C4E4G<BA -;8FGH78AG;4A75BB>;8?C87FGH78AGFC?4AG;8<E?84EA<A:46G<I<G<8FE86BE7<@CBEG4AG6BHEF8<A9BE@4G<BA4A7E89?86GBAG;8<E8KC8E<@8AGF -;8@4G8E<4?F><G5HA7?87GB:8G;8E4??G;8
@4G8E<4?FHF87<AG;86HEE<6H?H@4A7J4F7<FGE<5HG87GBG;8FGH78AGF -;8CEB68FF78F<:A
FG4:86BAF<FG87B9G;89B??BJ<A:C;4F8F8A:4:88KC?BE88KC?4<A8A:<A88E8AE<6;4A7
8I4?H4G8 (HG6B@84FF8FF@8AGF<A6?H787@84FHE<A:<AFGH78AG<AG8E8FG<A8A:<A88E<A:
?84EA<A::E47<A:8A:<A88E<A:78F<:ACEB=86GF4A7<AG8EI<8J<A:G846;8E4A7FGH78AGF      " 
674

GBG4?B9G;E88EBHA7FB96BHEF8FJ8E86BA7H6G87J<G;BA8EBHA7C8EJ88>?4FG<A:   @<AHG8F846;G<@8 9G8E846;EBHA7B96BHEF8FG;8E8F84E6;8EF6BA7H6G87FG4G<FG<64?
4A4?LF<FBAG;874G44A76BA7H6G87<AG8EI<8JFJ<G;G;8G846;8EF4A7FGH78AGF9BE9HEG;8E
<@CEBI8@8AG -;878G4<?87CEB68FFB9G;89<EFGEBHA7B96BHEF8F<F4F9B??BJFF88-45?8  -45?8 !*''5$+.'&31%'44
);4F8
-<@8@<AF
A:4:8  

KC?BE8  A:<A88E<A:  AE<6;  

I4?H4G<BA  8G4<?87

-846;8EF8KC?4A4G<BA4A7FGH78AG7<F6HFF<BAF8A45?8
FGH78AGFGB?84EA45BHGG;8;<FGBELB9F4<?5B4GF4A7
G;8F6<8A684A74CC?<64G<BAFB9F4<?78F<:A ,GH78AGFC4EG<6<C4G8<A4F4<?5B4G @4><A:6B@C8G<G<BA
9B??BJ875LF;4E<A:G;8<E6E84G<BAF49G8EJ;<6;G;8
G846;8ECEBI<78F8KC?4A4G<BAFB9G;8HA78E?L<A:
CE<A6<C?8FFH6;4F9E<6G<BA4A7@4G8E<4?F8?86G<BA ,GH78AGFE878F<:AG;8<EF4<?5B4GFJ<G;G;8G846;8E
<AGEB7H6<A:G;86BA68CGFB98A:<A88E<A:4A7
:H<7<A:G;8@G;EBH:;G;88A:<A88E<A:78F<:A
CEB68FFGBBCG<@<M8G;8<E78F<:AF ,GH78AGFJBE><A:EBHCFB99BHEGB6B??45BE4G<I8?L
78F<:A4A75H<?7F4<?5B4GF9BE46B@C8G<G<BA -;8G846;8ECEBI<78F9887546>4A7FGH78AGFF;4E8
G;8<E8KC8E<8A68FJ<G;G;8G846;8E6BA6?H7<A:G;8
46G<I<GLJ<G;4FH@@4EL   /2317'/'051(!'$%*+0)31%'44"AG;8F86BA7EBHA7B946G<I<G<8FG;88A:4:8J4F6BA7H6G875L:EBHCG846;<A: -;8GBC<6
9B6HF87BA6B@@BAGBB?FG;4GHG<?<M8J<A78A8E:L4A7<AG8E46G<I87<F6HFF<BAFJ8E8;8?7J<G;
G;8FGH78AGFGB<AGEB7H68G;86BA68CGB9;4EA8FF<A:J<A78A8E:L "AG;88KC?BE4G<BAC;4F8
G;8FGE4JFJ8E8E8C?4687J<G;4@BE8E<:<74A784F<?L9<K45?8@4G8E<4?4A7G;8FGEH6GHE8
5H<?7<A:6B@CBA8AGJ4FF<@C?<9<87GBHF<A:9B4@5B4E7F9BEFHCCBEG -;88KC?4A4G<BA
E8C?4687G;8GE47<G<BA4??86GHE8 FGL?878?<I8ELB9>8LCB<AGFJ<G;<AG8E46G<I8@<A< :4@8F -;8
G846;8EHF87G;8C;LF<64?@B78?B9G;864EGGB4F>DH8FG<BAF4A78A:4:8G;8FGH78AGF<A
7<F6HFF<BAFJ;<?8E86BE7<A:G;8<EE8FCBAF8F "AG;88A:<A88E<A:@BE8G<@8J4F4??B64G879BE
G846;8E8KC?4A4G<BAF4A7FGH78AG;4A7F BA46G<I<G<8F HE<A:G;88AE<6;FGH78AGF9BE@87
G84@FB99BHEGBC4EG<6<C4G8<A46B@C8G<G<BA "AG;88I4?H4G<BA846;FGH78AG6BA7H6G874
5E<894A4?LF<F 
9G8EBCG<@<M<A:G;8F86;4A:8FG;8FH5F8DH8AGE86EH<G@8AG4A76?4FF8FJ8E8
6BA7H6G874A7G;8G846;8EFJ8E845?8GBF@BBG;?L6BAGEB?G;86?4FFEBB@ -;8G846;<A:
8KC8E<8A68<@CEBI874A7G;8E8J4F4A<A6E84F8<AG;86B@C?8G<BAE4G8B9FGH78AGCEB=86GF   /2317'/'051(!'$%*+0)3$/'813-

9G8EG;89<EFGEBHA7B946G<I<G<8F8KC?BE4G<BA4A78A:<A88E<A:6B@CBA8AGF;4I8588A47787
GBCEBI<78E898E8A688KC8E<@8AG4?CEB687HE8F9BEFGH78AGFJ;B@4L8A6BHAG8E7<99<6H?G<8F<A
68EG4<ABC8E4G<BAF&BE8F884CC8A7<K 4CC8A7<K 0<G;G;8<A6?HF<BAB98KC8E<@8AG4?
CEB687HE8F<AG;8F86BA7EBHA7B946G<I<G<8FBI8E B9FGH78AGFFH668FF9H??L6B@C?8G87
 BE@BE8B9G;86BAG8AG4A778F<:AB9G;8<ECEB=86GF4FBHG?<A87<AG;8?84EA<A:@4AH4? 
BEG;8G;<E7EBHA7B946G<I<G<8FG;8,&)+,H5FG<GHG8B@5<A874CG&B7<9L &4:A<9L
)HECBF8?<@<A4G84A7+84EE4A:8 +8I8EF85E4<AFGBE@<A:6B:A<G<I8GBB?J4FHF87GB:H<78
FGH78AGFG;EBH:;G;8CEB5?8@ FB?I<A:CEB68FF,8EE4G,8EE4G  77<G<BA4??LG;8
E8F84E6;8EF8@C?BL87G;8  *H<6>FF8FF@8AG&8G;B7  6B@@BA?LHF87<AG;8<ADH<EL 54F87G846;<A:@B78?GB8A6BHE4:8FGH78AGFGBF;4E8G;E88G4>84J4LFGJB
6;4??8A:8F4A7BA84FC86GG;8L9BHA7C4EG<6H?4E?L89986G<I87HE<A:G;88AG<E846G<I<GL 

675

B??BJ<A:G;8F8<@CEBI8@8AGFG;8FH668FFE4G8B9FGH78AG8KC?BE4G<BA8KC8E<@8AGF4A7G;8
6B@C?8G<BA?8I8?B9G;8<ECEB=86GFF<:A<9<64AG?L<A6E84F87<AG;8G;<E7EBHA7B9<AFGEH6G<BA   5*'3/2317'/'054

<EFG?L<AG8E@FB9C8EFBAA8?6BBE7<A4G<BAG;8E8J4F8A;4A6876B@@HA<64G<BA4@BA:
E8F84E6;8EFE86EH<G@8AGC8EFBAA8?4A7G846;8EF -;8E86EH<G@8AGCEB68FFJ4F6BBE7<A4G87
4A7C?4AA87GB4IB<75?<A78AEB??@8AG ,86BA7?L<AG8E@FB96BHEF8@4G8E<4?F64AI4F
G;<AA8EJBB74A7477<G<BA4?F@4??JBB78AFG<6>FJ8E847787GB@88GG;8FGH78AGFA887F
4A78AG;HF<4F@9BE945E<64G<BA4A778F<:A  0<G;G;8F8<@CEBI8@8AGFG;8E86EH<G@8AGB9FGH78AGF5864@8@BE8G4E:8G87
4IB<7<A:G;8<AG8E98E8A68B9<A89986G<I8F4@C?8F -;8HF8B9477<G<BA4?@4G8E<4?F5LFGH78AGF
?87GB@BE8E89<A874A76B@C?8K78F<:ACEB=86GFE8FH?G<A:<A<@CEBI87C8E9BE@4A687HE<A:
G8FG<A:  $#,("(%'$"$ $)'()
     $! -;EBH:;G;E88EBHA7FB9E86EH<G@8AG4A76BHEF8<@C?8@8AG4G<BA4GBG4?B9I4?<7,-&<AG8E8FGFHEI8L9BE@FJ8E86B??86G879BE@F<AG;89<EFGEBHA7 9BE@F<AG;8F86BA7EBHA7
4A7 9BE@F<AG;8G;<E7EBHA7 -;84A4?LF<FE8FH?GF4E84F9B??BJFF88-45?8 
-45?8  !'$30+0)05'3'450$.94+4'46.54


"











 

  " !"!



"


 





 














 

 

 



"

 



 



 







 

 



 



 





 



"


      






 

 



 





 !"
#
# 
#$  "G64A58B5F8EI87G;4GG;8E8J8E8ABF<:A<9<64AG7<998E8A68F<AFGH78AGF<AG8E8FG<A
F6<8A68G86;AB?B:L@4G;8@4G<6F4A7,-&64E88EF !BJ8I8EG;8E8J4F4F<:A<9<64AG
7<998E8A68<A8A:<A88E<A:?84EA<A:<AG8E8FG49G8E<AFGEH6G<BA4?<AG8EI8AG<BA  $!    -;8E8F84E6;6BA7H6G874F6BE<A:4A7E86BE7<A:B9G;88A:<A88E<A:78F<:ACEB=86GF<A846;
EBHA7 -;8E8J8E8CEB=86GF<AG;89<EFGEBHA7 CEB=86GF<AG;8F86BA7EBHA74A7 
CEB=86GF<AG;8G;<E7EBHA7 "A846;EBHA7G;89<A4?F6BE8F9BE846;7<@8AF<BAB9G;8CEB=86GF

676

J8E864?6H?4G874FG;84I8E4:8F6BE8F -;84A4?LF<F9B6HF87BA6B@C4E<A:J;8G;8EG;8E8
J4F4A<@CEBI8@8AG<AG;8F6BE8FB9FGH78AGCEB=86GF<AG8E@FB96B@C?8G8A8FF984F<5<?<GL
48FG;8G<6F4A7<AABI4G<BAF88<:HE8  +)63' A4?LF<F+8FH?GFB9A:<A88E<A:8F<:A)EB=86G HE<A:G;8G;E88EBHA7FG;8E8J4F4A<@CEBI8@8AG<AG;86B@C?8G8A8FF984F<5<?<GL
4A7<AABI4G<BAB9FGH78AG8A:<A88E<A:78F<:ACEB=86GF -;848FG;8G<6F4?FBF;BJ87
<@CEBI8@8AG<AG;8G;<E7EBHA74?G;BH:;G;87<998E8A68J4FABGF<:A<9<64AG &BE8BI8EG;8E8
J4F4ABG<6845?8<@CEBI8@8AG<AG;86B@C?8G8A8FF58GJ88AG;89<EFG4A7F86BA7EBHA7F4A7
G;8<AABI4G<BAF<:A<9<64AG?L<@CEBI8749G8EG;8F86BA7EBHA7&BE8F884CC8A7<K
4CC8A7<K4CC8A7<K   "#$! 
EB@4ABI8E4??C8EFC86G<I849G8EG;89<EFGEBHA7B9G;846G<I<G<8FFGH78AGFJ;BJ8E8
<AG8EI<8J875BG;G;BF8J;BFH668FF9H??L6B@C?8G87G;88A:<A88E<A:78F<:ACEB=86GF4A7
G;BF8J;B7<7ABG6BH?7BA?LE864??G;874G4@8AG<BA875LG;8G846;8E<A6?4FF4A7CEBI<784
F<@C?88KC?4A4G<BAJ<G;BHG58<A:45?8GB4EG<6H?4G8;BJGB4CC?L<GGBG;878F<:AB9G;8F4<?
64E !BJ8I8E<AG;8F86BA74A7G;<E7EBHA7FFB@8FGH78AGFJ8E845?8GB4FF8FF4A7
8I4?H4G8E8?8I4AG8A:<A88E<A:>ABJ?87:84A76BA68CGF "AG;8G;<E7EBHA7FB@8FGH78AGF
8I8ACEBCBF87<A6BECBE4G<A:>ABJ?87:8B95HBL4A6LGB@4>8G;8F4<?64E4@C;<5<BHF HEG;8E@BE8<AG8E@FB9F4G<F946G<BAJ<G;G;86BHEF84A7<AG8E8FG<AG;846G<I<G<8F4C4EG9EB@
FB@8FGH78AGF<AG;89<EFGEBHA78KCE8FF<A:4F;BEG4:8B9@4G8E<4?FBI8E4??F4G<F946G<BAJ4F
;<:; 


"(*(("%$)+)*%$)$#&(%,#$*)/%+,&(%,($*$%((%$*"$*()$
.*($" )$ +*%() *% %$)( ()*"/ /%+ (%##$ #!$ *%(%+ &(&(*%$)-$#&"#$*$*,*)+)$* %"$*$/%+$%($0*+((+"+#
(%+$ ("" $$($ &(%"#) ** $ )%",$ $ )$ $'+(/ .&(#$*) -*#%(* +"*/ *% &(%, $))(/ )%"$ %( )*+$*) *(/ )*#+"*$ *(%$$ $ $)%$ )$ $ * .&"$*%$ *() $ %$+* $'+(/ .&(#$*)

677

.&"$ $ *" * )$* %$&*) &($&") $ %( !$%-" ('+( *% )%",
$$($&(%"#)/((($*%(",$*#*(")$"+$$*()&"$(/)+ *))+) )$ )*%(/ *$%"%" ,"%&#$* )%" )$) #*#*) * +($ *$$($*()$.&"$*%$&*)$)$&(%)))$,%",$$$($$+*"0 (*, *$!$ *%%") "!   ($)*%(#$ *% )))* )*+$*) $ %($0$ $
%$&*+"0$*()$)+(*(#%(*())%+"&()%$""/*)*$(**&(%+*)**)*+$*)(.&**%#!$"))$$*(*,*)*$$&(%,*#"/!$)+)*%$)$*,"+*%$&)#&"%/,()))))#$*#*%),%
)%""/*(","+*%$$%$)(+)$*%%"))+)*  *%""%-)*+$*)*%)(*( *( *!-/) *-%'+)*%$) $ %$ &%)*, )&* $"$ *# *% ("* %$ *("($$%+*%#)$)+##(0*%+()%$*$*%$"//%+#&)0*#&%(*$% *#-%(! $ %""%(*%$ $ * #&"#$**%$ &(%)) *% $$ -%(! $/ $&(,$* ((%() %( .#&" $ * ) % * )#"" )" ( *,*/ * *$ $)*(+*%()
.&(*)$&/))$$$($!$%-""%$-**(.*$),*$.&($(*"/*()(($#&(%,$*$)*(+*%$")$
77<G<BA4??LLBH;<:;?<:;GG;86;4??8A:8F8A6BHAG8E877HE<A:G;8<@C?8@8AG4G<BA
<A6?H7<A:7<99<6H?G<8F<AFGH78AGE86EH<G@8AG4A7?<@<G4G<BAF6BA68EA<A:8KC8E<@8AG4?@4G8E<4?F
GBB?F4A7946<?<G<8F -846;8EF@4L4??B64G84F<:A<9<64AGCBEG<BAB9G;8<EG846;<A:G<@8GB
@4<AG4<ABE78E7HE<A:G;8CEB68FFB9FGH78AGF;4A7F BA6BAFGEH6G<BAJ;<6;E8FH?GF<A?8FF
G<@8787<64G87GB8KC?4<A<A:G;8<AFGEH6G<BA4?6BAG8AG 2BHFH::8FGG;4GG;<F<FFH8@4L4E<F8
9EB@<AFH99<6<8AG6BAF<78E4G<BAB9G;846GH4?G846;<A:F<GH4G<BA4A7CBG8AG<4?CEB5?8@F7HE<A:
G;8<A<G<4?E8F84E6;C;4F8 -;8E89BE8J;8A78F<:A<A:46G<I<G<8FJ<G;8A:<A88E<A:G;8@8FLBH
E86B@@8A7G;4GG846;8EFC4L4GG8AG<BAGBCE8C4E4GBELJBE>FH6;4F6B??86G<A:@4G8E<4?F
6BA7H6G<A:@B78?G8FG<A:8I4?H4G<A:89986GF4A7CEBI<7<A:45HA74AG<AFGEH6G<BA4?
F6499B?7<A:GB@4<AG4<AFGH78AGF6BA9<78A68<A4HGBAB@BHF78F<:A  %$#*("%$4F87BAG;8E8F84E6;9<A7<A:F4A78KC8E<8A68F<AG8:E4G<A:6B@CE8;8AF<I8,-&
87H64G<BAJ<G;$ 8A:<A88E<A:87H64G<BA4A8KC?BE4G<BAB9<A6BECBE4G<A:8A:<A88E<A:
8?8@8AGF<AGBF6<8A686?4FFEBB@FJ4F6BA7H6G87 ,-&87H64G<BA64F8FGH7LJ4F
78F<:A874A778I8?BC87J<G;49B6HFBAG;878F<:AB94A4A6<8AG;<A8F8GE4AFCBEG4G<BA
I8;<6?864??87G;8F4<?64E "GJ4F7<F6BI8E87G;4GJ;8A,-&87H64G<BA46G<I<G<8F78I8?BC87
HF<A:G;8&B78?J8E84CC?<87<AG846;<A:CE46G<68FG;8LF<:A<9<64AG?L8A;4A687FGH78AGF
<AG8E8FG<AG;89<8?7B98A:<A88E<A: -;8<AFGEH6G<BA4?46G<I<G<8FCEBI8789986G<I89BE<AG8:E4G<A:
F6<8AG<9<6<ADH<ELJ<G;8A:<A88E<A:CE46G<68F<AG;86?4FFEBB@ -;EBH:;BCG<@<M<A:G;8
G846;<A:CEB68FF<AFGEH6G<BA4?F6499B?7<A:4A7BG;8E4FC86GFFH6;4F@4G8E<4?F4A7
BC8E4G<BAF@4A4:8@8AG4;<:;?LCE46G<64?,-&87H64G<BA64F8G;4G<AG8:E4G8F8A:<A88E<A:
CE46G<68F<AGBF6<8A68<ADH<EL6?4FFEBB@FJ4F78I8?BC87B998E<A:4A8J4CCEB46;9BE
<AG8EA4G<BA4?,-&87H64G<BA6HEE<6H?H@78I8?BC@8AG '%(&)(

-;8?<@<G4G<BAFB9G;8FGH7L<A6?H789B6HF<A:BA?LBA@86;4A<6F4A7OG86;AB?B:L4A7
8A:<A88E<A:PG;8@8FJ<G;BHG9BE@<A:46B@C?8G8F8E<8FB96BHEF8F4F@4??F4@C?8F<M84A7
4F<A:?88I4?H4G<BA@8G;B7 "A9HGHE8E8F84E6;G;88AE<6;@8AG6BH?7588KC4A7875L
<A6BECBE4G<A:G;8HF8B97<:<G4?GBB?FGB:H<78FGH78AGF<A477<A:F8AFBEF4A774G44A4?LF<F
GBB?FG;4GCEBI<78E84? G<@8@BA<GBE<A:4A77<FC?4LB9G;864EF@BG<BAFG4GHFGBG;88K<FG<A:
F4<?64E@B78? "AG8E@FB96HEE<6H?H@GBC<6FBE:4A<M<A:<AFGEH6G<BA4?6BAG8AG4EBHA7
G86;AB?B:L4A78A:<A88E<A:BE5EB478EG;8@8F6BH?7586BAF<78E87GB78I8?BC4F8E<8FB9
6BHEF8F 





678

'$(4EEL'   -;8"-?84EA<A:5L8," '@B78? !'%*01.1)9$0&0)+0''3+0)!'$%*'3  N  
HE4A%  HE4A   -;8<AFGEH6G<BA4?@B78??84EA<A:6L6?8
4CCEB46;9BE<ADH<EL 54F87F6<8A68G846;<A:  %+'0%'&6%$5+10'7+'8 N 
HM8L, , &BBE8- # !4EJ8??& &BE8AB&   ,-&"AG8:E4G<BA<A
&<77?8,6;BB?%<98,6<8A68,GH78AG%84EA<A:4A7GG<GH78F 1630$.1( %+'0%'
&6%$5+10$0&!'%*01.1)9  N ;GGCF 7B< BE: F    K!F<4B! , %<A2 0 %<A$ 2 %<A 2 ;8A# ! ;8A#    .F<A:
EB5BG 54F87CE46G<68FGB78I8?BC4A46G<I<GLG;4G<A6BECBE4G87G;8@B78?GB<@CEBI88?8@8AG4ELF6;BB?FGH78AGFQ?84EA<A:C8E9BE@4A68F 05'3$%5+7''$30+0)
07+310/'054  N ;GGCF 7B< BE:     !H58?54A># (?<I4-    !#13%'45'31.95'%*0+%045+565'
#13%'45'3 #<A, ! ,BA:$ !LBHA:, ,;<A,   C8E9BE@4A68 54F878I4?H4G<BA
EH5E<69BE4FF8FF<A:4A78A;4A6<A:8A:<A88E<A:78F<:AF><??F<A<AGEB7H6GBEL
8A:<A88E<A:78F<:A6BHEF8F 05'30$5+10$.1630$.1(0)+0''3+0)&6%$5+10 
N  
#BA:?H86;4) 0BE4CHA0   8I8?BC<A: E478,GH78AG,6<8A68%84EA<A:6;<8I8@8AG4A7,6<8AG<9<6E84G<I<GL.F<A:G;8&B78?<A,-&7H64G<BA 
1630$.1(&6%$5+10$.446'4  ;GGCF 7B< BE: =8< I<  $;48EBA<A:GL4F' )8E@4A4F4E< !4@<74;"   ,G8@?84EA<A:<A@4G8E<4?
B9G8@C8E4GHE84A7<GF6;4A:8GB<@CEBI8F6<8AG<9<6?<G8E46LB9=HA<BE;<:;F6;BB? 
630$.'0&+&+-$00&10'4+$  N
%4< ! ;H &   8I8?BC@8AG4A7I4?H4G<BAB9,-&4F87
"AFGEH6G<BA4?8F<:AAK4@C?8B9*H476BCG8EBHEF8 "A- - 0H+  8AA4E<
2 & !H4A:! 1<82 4B7F /'3)+0)!'%*01.1)+'4(13&6%$5+10/B? CC  N   ,CE<A:8E"AG8EA4G<BA4?)H5?<F;<A: ;GGCF 7B< BE:      3 %<A$ 2 !F<4B! , 0<??<4@F) # ;8A2 !    9986GFB9 BE<8AG87
,-&CE46G<64?46G<I<G<8F<A6H?G<I4G<A:@<77?8F6;BB?FGH78AGFQ4GG<GH78FGBJ4E7
G86;AB?B:L4A7G86;AB?B:<64?<ADH<EL45<?<GL '4'$3%*+0 %+'0%'!'%*01.1)+%$.
&6%$5+10   N  ;GGCF 7B< BE:      &8EF<A'4G<BA4?7H64G<BA<E86GBE4G8-HE>8L.L:HE&   ,-& 4F87
BHEF88F<:A04LGB8I8?BCGG<GH78FGBJ4E7F,-&4A7,6<8A68BHEF8 %+'0%'&6%$5+1005'30$5+10$. N ;GGCF 7B< BE: F8< I < 
&BBE8- #  ?4A6L 0 -4A>$ & $8EFG8A#  ,@<G;$  ,GB;?@4AA& ,   9E4@8JBE>9BEDH4?<GL$ 8A:<A88E<A:87H64G<BA+8F84E6;4A7
78I8?BC@8AG 1630$.1(3'1..')'0)+0''3+0)&6%$5+10'4'$3%*  '4G<BA4?+8F84E6;BHA6<?   0)+0''3+0)+0 '&6%$5+10"0&'345$0&+0)5*'
45$564$0&+/2317+0)5*'23142'%54 '4G<BA4?6478@<8F)E8FF '4G<BA4?+8F84E6;BHA6<?   6%%'44(6.  !'&6%$5+10&'05+(9+0)
'(('%5+7'$2231$%*'4+04%+'0%'5'%*01.1)9'0)+0''3+0)$0&/$5*'/$5+%4 '4G<BA4?6478@<8F)E8FF '4G<BA4?+8F84E6;BHA6<?    !+05')3$5+10+0 '&6%$5+10 5$564
23142'%54$0&$0$)'0&$(133'4'$3%* '4G<BA4?6478@<8F)E8FF +47?B99#  HM8L,   "AI8FG<:4G<A:)E8F8EI<68,-&-846;8EBA68CG<BAF
B9,-&7H64G<BA 1630$.1( %+'0%'&6%$5+10$0&!'%*01.1)9 N ;GGCF 7B< BE: F    ,8EE4G( ,8EE4G(   -;8,&)+G86;A<DH8 018.'&)' 1.65+104
!11.4'5*1&4$0&2231$%*'4513+7'3)$0+:$5+10$.'3(13/$0%' N  679

-L?8E 0BB7- $A8M8> ;E<FG8AF8A+  "AFGEH@8AGF9BE4FF8FF<A:
<AG8E8FG<A,-&6BAG8AG4A764E88EF 1630$.1(!'%*01.1)9$0&!'$%*'3
&6%$5+10 N 






680

&&$"+
CC8A7<K KC?BE847787



681




CC8A7<KA:<A88E<A:477



682



683




CC8A7<K-;89<EFGEBHA7B9FGH78AGJBE>FC4EG<4?

684

CC8A7<K -;8F86BA7EBHA7B9FGH78AGJBE>FC4EG<4?


CC8A7<K -;8G;<E7EBHA7B9FGH78AGJBE>FC4EG<4?

685

6KLK -/ HW DO (GV   3URFHHGLQJV RI WKH VW ,QWHUQDWLRQDO &RQIHUHQFH RQ &RPSXWHUV LQ (GXFDWLRQ$VLD3DFLILF 6RFLHW\ IRU &RPSXWHUV LQ (GXFDWLRQ

7KH ,PSDFW RI *DPLILHG $VVHVVPHQW RQ WKH/HDUQLQJ %XUQRXW RI 8QGHUJUDGXDWH&RPSXWLQJ 6WXGHQWV D 4XDVLH[SHULPHQWDO
5HVHDUFK%HLOHL =+$1*D  $OH[ :LQJ &KHXQJ 76(E7KH 8QLYHUVLW\ RI +RQJ .RQJ +RQJ .RQJ 6$5
DX#FRQQHFWKNXKN EDZFWVH#KNXKN$EVWUDFW /HDUQLQJ EXUQRXW UHIHUV WR WKH QHJDWLYH DWWLWXGH DQG EHKDYLRU RI EHLQJ ERUHGZLWK OHDUQLQJ GXH WR OHDUQLQJ SUHVVXUH RU ODFN RI OHDUQLQJ LQWHUHVWV ,W DIIHFWV QRW RQO\
VWXGHQWV¶ DFDGHPLF SHUIRUPDQFH EXW DOVR WKHLU SK\VLFDO DQG PHQWDO KHDOWK &XUUHQWO\WKH UHVHDUFK RQ OHDUQLQJ GHVLJQ WR DOOHYLDWH OHDUQLQJ EXUQRXW KDV RQO\ EHHQ WKH VXEMHFW
RI D IHZ VWXGLHV &RQVLGHULQJ WKDW JDPLILHG DVVHVVPHQW WKHRUHWLFDOO\ JDYH LW SRWHQWLDO WR
DIIHFW VWXGHQWV¶ OHDUQLQJ EXUQRXW  XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV DW D
XQLYHUVLW\ LQ PDLQODQG &KLQD SDUWLFLSDWHG LQ WKLV VWXG\ WR LQYHVWLJDWH ZKHWKHU VXFK DOHDUQLQJ SURFHVV ZDV FRQGXFLYH WR DIIHFWLQJ OHDUQLQJ EXUQRXW 7KH UHVHDUFK ZDV DTXDVLH[SHULPHQWDO GHVLJQ ZLWK SUHWHVW DQG SRVWWHVW )RU WKH H[SHULPHQWDO JURXSQ   JDPLILFDWLRQ HOHPHQWV ZHUH LQWHJUDWHG LQWR WZR VHVVLRQV LQFOXGLQJ WKH TXHVWLRQDQG DQVZHU VHVVLRQV DQG DIWHUFODVV TXL]]HV E\ 4XL]L]] IRU VL[ ZHHNV  KRXUV SHU
ZHHN  'XULQJ WKH GDWD FROOHFWLRQ DOO SDUWLFLSDQWV LQYROYHG FRPSOHWHG WKH SUHWHVW DQG
SRVWWHVW RI WKH 0DVODFK /HDUQLQJ %XUQRXW ,QYHQWRU\6WXGHQW 6XUYH\ YLD DQ HOHFWURQLF
TXHVWLRQQDLUH 7KH TXDQWLWDWLYH GDWD ZHUH DQDO\]HG WKURXJK $1&29$ 7KLV VWXG\
IRXQG WKDW VWXGHQWV¶ OHDUQLQJ ZLWK JDPLILHG DVVHVVPHQW IRU VL[ ZHHNV GLG FRQWULEXWH WR
DOOHYLDWLQJ VWXGHQWV¶ OHDUQLQJ EXUQRXW SDUWLFXODUO\ LQ WHUPV RI F\QLFLVP DQG ORZ HIILFDF\
7KH ILQGLQJV FDQ EH D UHIHUHQFH IRU H[SORULQJ WKH LQWHJUDWLRQ RI JDPLILHG DVVHVVPHQW
LQWR WHUWLDU\ HGXFDWLRQ.H\ZRUGV *DPLILHG DVVHVVPHQW OHDUQLQJ EXUQRXW JDPLILFDWLRQ TXDVLH[SHULPHQWDOGHVLJQ

 ,QWURGXFWLRQ:LWK WKH ZLGHVSUHDG XVH RI LQIRUPDWLRQ WHFKQRORJ\ LQ WHUWLDU\ HGXFDWLRQ PHQWDO KHDOWK ULVNVHVSHFLDOO\ WKRVH UHODWHG WR OHDUQLQJ EXUQRXW DPRQJ XQGHUJUDGXDWH VWXGHQWV DUH SDUWLFXODUO\REYLRXV =KDR HW DO   $IIHFWHG E\ OHDUQLQJ EXUQRXW VWXGHQWV¶ FKDUDFWHULVWLFV RI SRRUOHDUQLQJ SHUIRUPDQFH EHFRPH PRUH FRPPRQ LQFOXGLQJ LQDWWHQWLYHQHVV ORZ OHDUQLQJHIILFLHQF\ KLJK OHDUQLQJ DQ[LHW\ DQG HYHQ UHVLVWDQFH WR OHDUQLQJ 6RPH VFKRROV KDYH SXWIRUZDUG FRUUHVSRQGLQJ SV\FKRORJLFDO FRXQVHOOLQJ VXJJHVWLRQV DLPLQJ DW HIIHFWLYHO\ DOOHYLDWLQJ
WKH HPRWLRQDO SUREOHPV RI OHDUQLQJ $WHV  ,Q DGGLWLRQ WR SV\FKRORJLFDO JXLGDQFH WU\LQJ QHZ OHDUQLQJ DQG WHDFKLQJ PHWKRGV FDQ
EH DQRWKHU ZD\ WR GHDO ZLWK OHDUQLQJ EXUQRXW ,Q RUGHU WR SUHYHQW VWXGHQW EXUQRXW GXULQJ WKHOHDUQLQJ SURFHVV LW LV FUXFLDO WR NHHS WKHLU HQWKXVLDVP IRU WKH VXEMHFW DOLYH 2QH ZD\ WR
PRELOL]H WKHLU LQWHUHVW LQ OHDUQLQJ LV WKURXJK JDPLILHG DVVHVVPHQW +XDQJ HW DO  *DPLILFDWLRQ FDQ EH GHILQHG DV WKH XVH RI JDPH HOHPHQWV LQ QRQJDPH HQYLURQPHQWV'HWHUGLQJ HW DO   :KHQ JDPLILFDWLRQ LV DSSOLHG WR WKH HGXFDWLRQDO DUHD LW PRWLYDWHV DQG
HGXFDWHV OHDUQHUV WKURXJK DSSURSULDWH JDPH HOHPHQWV *DPLILFDWLRQ KDV EHHQ H[KLELWHG DV DQ
HIIHFWLYH DSSURDFK IRU WHDFKLQJ FRPSXWHU FRXUVHV LQ WHUWLDU\ HGXFDWLRQ $KPDG HW DO  
7KH XVH RI JDPLILFDWLRQ LQ HGXFDWLRQDO FRQWH[WV KDV OHG WR WKH FUHDWLRQ RI QHZ DVVHVVPHQW
PHWKRGV 0RUH VSHFLILFDOO\ JDPH HOHPHQWV VXFK DV EDGJHV DQG OHYHOV DUH DGGHG WR VWXGHQW

686

DVVHVVPHQWV 3DVW UHVHDUFK KDV VKRZQ WKDW JDPLILFDWLRQ KDV EHHQ HIIHFWLYHO\ LPSOHPHQWHG LQ
D QXPEHU RI OHDUQLQJ HQYLURQPHQWV &KXQJ /LQ  6DOHHP HW DO   +RZHYHU WKH
XVH RI JDPLILFDWLRQ WR DVVHVV VWXGHQW OHDUQLQJ KDV RQO\ EHHQ WKH VXEMHFW RI D IHZ VWXGLHV=KDQJ )DQJ   1RWDEO\ OLWWOH UHVHDUFK KDV EHHQ GRQH WR LQYHVWLJDWH WKH HIIHFWV RIJDPLILHG DVVHVVPHQW LQ WKH FRQWH[W RI WHUWLDU\ HGXFDWLRQ
7KLV VWXG\ DLPV WR H[SORUH ZKHWKHU JDPLILHG DVVHVVPHQW FRQWULEXWHV WR DOOHYLDWLQJXQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV¶ OHDUQLQJ EXUQRXW KRSLQJ WR SURYLGH D UHIHUHQFH IRU
H[SORULQJ WKH LPSURYHPHQW RI VXFK D SUREOHP $ PRUH WKRURXJK DQDO\VLV RI KRZ JDPLILHG
DVVHVVPHQW DIIHFWV PXOWLSOH GLPHQVLRQV RI OHDUQLQJ EXUQRXW KDV EHHQ FRQGXFWLQJ LQ WKH
VHFRQG VWDJH RI WKH UHVHDUFK SURMHFW EDVHG RQ WKH TXDOLWDWLYH ILQGLQJV ZKLFK ZLOO EH UHSRUWHG LQ
DQRWKHU SDSHU
7KH UHVHDUFK TXHVWLRQ RI WKH SUHVHQW VWXG\ 'RHV JDPLILHG DVVHVVPHQW DIIHFW WKHOHDUQLQJ EXUQRXW RI XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV" /LWHUDWXUH 5HYLHZ *DPLILFDWLRQ LQ (GXFDWLRQ*DPLILFDWLRQ LV WKH XVH RI JDPH HOHPHQWV RU JDPH GHVLJQ WHFKQLTXHV LQ QRQJDPH FRQWH[WVZLWK WKH DLP RI LQWURGXFLQJ JDPH HOHPHQWV ZLWKRXW GHWDFKLQJ WKHP IURP UHDOLVWLF FRQWH[WV:HUEDFK HW DO   *DPLILFDWLRQ KDV EHHQ VKRZQ WR HQKDQFH OHDUQLQJ $QGULDPLDULVRD
 21HLOO HW DO   ,W LV DQ HIIHFWLYH ZD\ RI ERRVWLQJ OHDUQHUV WR DFKLHYH OHDUQLQJ
RXWFRPHV PRWLYDWLQJ OHDUQHUV DQG IDFLOLWDWLQJ OHDUQLQJ *|NVQ *UVR\   0DF.LQQRQ
HW DO  FRQGXFWHG D TXDVLH[SHULPHQWDO VWXG\ WR LQYHVWLJDWH WKH LPSDFW RI JDPLILFDWLRQ RQ
VWXGHQWV¶ PRWLYDWLRQ DQG WKHLU FRQFOXVLRQV LQGLFDWHG WKDW JDPLILFDWLRQ VLJQLILFDQWO\ LPSURYHG
OHDUQLQJ RXWFRPHV 7KH JDPLILFDWLRQ HOHPHQWV FDQ EH SURJUHVV EDUV OHDGHUERDUGV DQG
EDGJHV HPSOR\HG DV DOWHUQDWLYH RU VXSSOHPHQWDU\ DVVHVVPHQW PHWKRGV WR VFRUHV 7VD\ HW
DO   1XPHURXV HGXFDWLRQDO UHVHDUFK KDV XVHG JDPLILFDWLRQ IRU WKH ODVW GHFDGH WR
VWLPXODWH OHDUQLQJ PRWLYDWLRQ LQFUHDVH HQJDJHPHQW DQG DFFHOHUDWH OHDUQLQJ 'LFKHYD HW DO  7KHUHIRUH IXUWKHU H[SORUDWLRQ DQG UHVHDUFK LQWR WKH LQWHJUDWLRQ RI JDPLILFDWLRQ LQ
HGXFDWLRQ LV QHFHVVDU\ WR IXOO\ UHDOL]H LWV SRWHQWLDO LQ HQKDQFLQJ WKH LQVWUXFWLRQDO HIIHFWLYHQHVV,Q WKLV VWXG\ JDPLILFDWLRQ ZDV LQWHJUDWHG LQWR WKH DVVHVVPHQW *DPLILHG $VVHVVPHQW:KHQ WKH WHUP ³JDPLILHG´ LV DSSOLHG WR DVVHVVPHQW LW GHVFULEHV D GHVLJQ SURFHVV WKDWLQFRUSRUDWHV JDPLQJ DVSHFWV WR LPSURYH DQ H[LVWLQJ DVVHVVPHQW PHWKRG *HRUJLRX
1LNRODRX   *DPLILHG DVVHVVPHQW FDQ FUHDWLYHO\ DSSO\ JDPLILFDWLRQ HOHPHQWV WR UHODWHG
WDVNV ,W JLYHV WKH SRWHQWLDO WR SURPRWH WKH HIILFLHQF\ RI VWXGHQWV¶ LQIRUPDWLRQ H[FKDQJH WRDFKLHYH WKH SXUSRVH RI SURPRWLQJ OHDUQLQJ 6KXWH  
9DULRXV JDPLILFDWLRQ HOHPHQWV FDQ EH LQFRUSRUDWHG LQWR H[LVWLQJ DVVHVVPHQWVLQFOXGLQJ SURJUHVV EDUV SRLQWV DQG EDGJHV /DQGHUV HW DO   7KH LPSOHPHQWDWLRQ RI
JDPLILHG DVVHVVPHQW LV XVXDOO\ FDUULHG RXW E\ WKH WHDFKHU XVLQJ VRPH JDPLILHG DVVHVVPHQW
V\VWHPV 'HODFUX]   VXFK DV .DKRRW 6RFUDWLYH DQG 4XL]L]] 7HDFKHUV FDQ UDSLGO\ DQG
HIIHFWLYHO\ DVVHVV VWXGHQWV¶ OHDUQLQJ ZLWK WKH DLG RI JDPLILFDWLRQ V\VWHPV DQG VWXGHQWV FDQ
EHQHILW IURP LPPHGLDWH IHHGEDFN $ UHFHQW HPSLULFDO VWXG\ KDV IXUWKHU FRQILUPHG WKDWWHFKQRORJ\HQKDQFHG JDPLILHG DVVHVVPHQW ZDV PRUH PRWLYDWLQJ WKDQ SDSHU WHVWV DQG
LPSURYHG DFDGHPLF SHUIRUPDQFH =DLQXGGLQ HW DO   8VLQJ JDPLILFDWLRQ LQWHUDFWLRQ FRXOGLPSURYH FRXUVH FRPSOHWLRQ LQWHUDFWLRQ DQG PRWLYDWLRQ UDWH +DVVDQ HW DO   'LDV IRXQG WKDW JDPLILHG DVVHVVPHQW LQFUHDVHG WKH SURSRUWLRQ RI VWXGHQWV SDUWLFLSDWLQJ LQ WKHFODVVURRP $ OLWHUDWXUH UHYLHZ FRQFOXGHG WKDW WKH PDMRULW\ RI UHODWHG SDSHUV KDYH FRQFOXGHG
WKDW JDPLILHG DVVHVVPHQW KDV SRVLWLYH HIIHFWV RQ OHDUQLQJ PRWLYDWLRQ OHDUQLQJ DFKLHYHPHQWDQG OHDUQLQJ VDWLVIDFWLRQ :DQJ 7DKLU  ,Q VXPPDU\ JDPLILHG DVVHVVPHQW LV DQ LQQRYDWLYH DSSURDFK WR DVVHVVPHQW WKDWLQFRUSRUDWHV JDPH HOHPHQWV LQWR HGXFDWLRQDO FRQWH[WV WR SRVLWLYHO\ PRWLYDWH DQG HQJDJH

687

VWXGHQWV LQ OHDUQLQJ *DPLILHG DVVHVVPHQW KDV WKH SRWHQWLDO WR DIIHFW WKH OHDUQLQJ EXUQRXW RI
XQGHUJUDGXDWH VWXGHQWV DQG HQKDQFH WKHLU OHDUQLQJ HIIHFWLYHQHVV /HDUQLQJ %XUQRXW/HDUQLQJ EXUQRXW UHIHUV WR WKH QHJDWLYH DWWLWXGH DQG EHKDYLRU RI EHLQJ ERUHG ZLWK OHDUQLQJ GXHWR OHDUQLQJ SUHVVXUH RU ODFN RI OHDUQLQJ LQWHUHVWV 0HLHU 6FKPHFN   %XUQRXW DPRQJVWXGHQWV FDQ KDYH D QHJDWLYH LPSDFW RQ WKHLU DFDGHPLF DFKLHYHPHQW DQG SHUVRQDO ZHOOEHLQJ
5LEHLUR HW DO   7KHVH ILQGLQJV VXJJHVW WKDW DGGUHVVLQJ OHDUQLQJ EXUQRXW LV FUXFLDO IRUVWXGHQWV¶ RYHUDOO KHDOWK DV ZHOO DV IRU WKHLU DFDGHPLF DFKLHYHPHQW$ VWXG\ UHSRUWHG WKDW OHDUQLQJ PRWLYDWLRQ DQG EXUQRXW KDYH D VLJQLILFDQW QHJDWLYH
FRUUHODWLRQ &D]DQ   &XUUHQW UHOHYDQW UHVHDUFK IRFXVHV RQ H[SORULQJ WKH LQIOXHQFLQJIDFWRUV RI OHDUQLQJ EXUQRXW IURP D SV\FKRORJLFDO SHUVSHFWLYH DQG VHHNLQJ LQWHUYHQWLRQPHDVXUHV $Q HPSLULFDO VWXG\ IRXQG WKDW WHDFKHUV¶ HPRWLRQDO VXSSRUW KDV D PLWLJDWLQJ HIIHFWRQ OHDUQLQJ EXUQRXW =KDR HW DO   %DVHG RQ H[LVWLQJ UHOHYDQW UHVHDUFK D ODFN RIPRWLYDWLRQ LV RQH RI WKH FDXVHV RI EXUQRXW )HOD]D HW DO   *DPLILHG DVVHVVPHQW KDV
EHHQ VKRZQ WR VWLPXODWH PRWLYDWLRQ WKURXJK WKH LQFRUSRUDWLRQ RI JDPLILFDWLRQ HOHPHQWVLPPHGLDWH IHHGEDFN SURJUHVV WUDFNLQJ DQG KHDOWK\ FRPSHWLWLRQ DPRQJ VWXGHQWV 2OLYHU
 3LWR\R $VLE   +HQFH WKH JDPLILHG DVVHVVPHQW JLYHV LW SRWHQWLDO WR
DOOHYLDWH OHDUQLQJ EXUQRXW FRQVLGHULQJ OHDUQLQJ PRWLYDWLRQ /HDUQLQJ DQ[LHW\ LV DQ HVVHQWLDOIDFWRU LQIOXHQFLQJ OHDUQLQJ EXUQRXW *XDQJZHQ HW DO   7KH DSSOLFDWLRQ RI JDPLILHG
DVVHVVPHQW LQ WUDGLWLRQDO FODVVURRP WHDFKLQJ FDQ HIIHFWLYHO\ DOOHYLDWH OHDUQHUV¶ OHDUQLQJ
DQ[LHW\ /HH HW DO   +HQFH JDPLILHG DVVHVVPHQW PD\ LQGLUHFWO\ KHOS WR DOOHYLDWHOHDUQLQJ EXUQRXW 3UHYLRXV UHVHDUFK KDV GHPRQVWUDWHG WKDW JDPLILFDWLRQ FDQ JLYH OHDUQHUVSOHDVDQW SV\FKRORJLFDO H[SHULHQFHV OLNH LPPHUVLRQ DQG IORZ H[SHULHQFHV ZKLFK FRXOG
SURPRWH OHDUQLQJ 3LWR\R  :DWVRQ HW DO   *DPLILFDWLRQ DOVR SODFHV OHDUQHUV LQ D
SRVLWLYH SV\FKRORJLFDO VWDWH RI OHDUQLQJ ZKLFK KDV D SRVLWLYH LPSDFW RQ OHDUQHUV¶ DFDGHPLF
DFKLHYHPHQW DQG HQJDJHPHQW WKXV DOOHYLDWLQJ OHDUQLQJ EXUQRXW *|NVQ *UVR\  
7KH DERYH OLWHUDWXUH UHYLHZ KLJKOLJKWV WKH QHJDWLYH UHODWLRQVKLS EHWZHHQ OHDUQLQJPRWLYDWLRQ DQG EXUQRXW JLYLQJ JDPLILHG DVVHVVPHQW WKH SRWHQWLDO WR DIIHFW OHDUQLQJ EXUQRXW7KH UHYLHZ DOVR VXJJHVWV WKDW JDPLILHG DVVHVVPHQW KDV WKH SRWHQWLDO WR DOOHYLDWH OHDUQLQJ
EXUQRXW E\ VWLPXODWLQJ PRWLYDWLRQ DQG SURPRWLQJ SRVLWLYH SV\FKRORJLFDO H[SHULHQFHV 7KLVSURYLGHV D EDVLV IRU H[SORULQJ LI JDPLILHG DVVHVVPHQW FDQ DIIHFW WKH OHDUQLQJ EXUQRXW RIVWXGHQWV 7KLV VWXG\ LQYHVWLJDWHG VXFK DQ LVVXH DPRQJ WKH FRPSXWLQJ XQGHUJUDGXDWHV

 0HWKRGV 7KHRUHWLFDO )UDPHZRUN ,QVWUXPHQW DQG ,QWHUYHQWLRQ/HDUQLQJ EXUQRXW FDQ EH PHDVXUHG ZLWK WKH 0DVODFK %XUQRXW ,QYHQWRU\6WXGHQW 6XUYH\LQFOXGLQJ WKUHH GLPHQVLRQV HPRWLRQDO H[KDXVWLRQ F\QLFLVP DQG ORZ HIILFDF\ 6FKDXIHOL HW DO  7KH VFDOH LV DGPLQLVWHUHG RQ D VHYHQSRLQW /LNHUW VFDOH UDQJLQJ IURP  DEVROXWHO\LQDSSURSULDWH WR  DEVROXWHO\ DSSURSULDWH  +LJKHU VFRUHV LQGLFDWH KLJKHU OHYHOV RI OHDUQLQJ
EXUQRXW 7KH VWXG\ NHSW WKH VWUXFWXUH RI WKH RULJLQDO VFDOH E\ UHIHUULQJ WR WKH 0DVODFK %XUQRXW,QYHQWRU\6WXGHQW 6XUYH\ 7KH TXHVWLRQQDLUH FRQVLVWV RI  LWHPV GLYLGHG LQWR WKUHH
GLPHQVLRQV DERYH )LYH LWHPV DUH GHVLJQHG WR PHDVXUH HPRWLRQDO H[KDXVWLRQ ZKLFKGHVFULEHV IHHOLQJV RI HPRWLRQDO GHSOHWLRQ GXH WR VWXG\LQJUHODWHG VWUHVV &\QLFLVP LVPHDVXUHG ZLWK IRXU LWHPV WKDW DVVHVV QHJDWLYH DWWLWXGHV DQG GHWDFKPHQW WRZDUGV VWXG\LQJ
FODVVPDWHV RU WHDFKHUV /DVWO\ VL[ LWHPV DUH HPSOR\HG WR HYDOXDWH ORZ HIILFDF\ ZKLFK UHIHUVWR D GLPLQLVKHG VHQVH RI FRPSHWHQFH DQG DFFRPSOLVKPHQW LQ WKH UHDOP RI DFDGHPLFV 7KH/HDUQLQJ %XUQRXW 6FDOH ZDV WUDQVODWHG LQWR &KLQHVH E\ D SURIHVVLRQDO ZLWK H[SHUWLVH LQ
HGXFDWLRQDO SV\FKRORJ\ 3DUWLFLSDQWV ZHUH SURYLGHG ZLWK LQVWUXFWLRQV LQ WKHLU QDWLYH ODQJXDJH
DQG HQFRXUDJHG WR VHHN FODULILFDWLRQ LI QHHGHG $ SLORW VWXG\ ZDV FRQGXFWHG WR HVWDEOLVK WKHYDOLGLW\ DQG UHOLDELOLW\ RI WKH WUDQVODWHG YHUVLRQ688

(PSLULFDO YDOLGDWLRQ LV QHFHVVDU\ WR FRQILUP WKH K\SRWKHVL]HG SRWHQWLDO EHQHILWV RIJDPLILHG DVVHVVPHQWV LQ UHGXFLQJ OHDUQLQJ EXUQRXW 1HYHUWKHOHVV WKH PDMRULW\ RI UHVHDUFK RQ
OHDUQLQJ EXUQRXW KDV EHHQ IRFXVHG RQ PHGLFDO VWXGHQWV ZKLFK OLPLWV WKH SRWHQWLDO IRU
JHQHUDOL]DWLRQ RI WKH REVHUYHG ILQGLQJV $OPXWDLUL HW DO  )LRULOOL HW DO  *UDGLVNL HWDO   $OO WKHVH SRLQWV UHYHDO WKH LPSRUWDQFH RI WKLV UHVHDUFK %DVHG RQ WKH GLVFXVVLRQDERYH WKH LQWHUYHQWLRQ LQ WKLV VWXG\ ZDV JXLGHG E\ WKH WKHRUHWLFDO IUDPHZRUN SUHVHQWHG LQ)LJXUH  'LFKHYD HW DO  6FKDXIHOL HW DO  *DPLILHG 4XHVWLRQ
DQG $QVZHU*DPLILHG$VVHVVPHQW*DPLILHG $IWHUFODVV
4XL]]HV(PRWLRQDO([KDXVWLRQ/HDUQLQJ%XUQRXW)UDPHZRUN&\QLFLVP/RZ (IILFDF\([SHULPHQWDO JURXSƔ *DPLILHG LQVWUXFWLRQ:HHNV Ɣ /HFWXUHV  PLQXWHV
Ɣ *DPLILHG DVVHVVPHQW RITXHVWLRQ DQG DQVZHU,QTXLU\EDVHG OHDUQLQJDFWLYLWLHV GLVFXVVLRQ DQG
SUHVHQWDWLRQ  PLQXWHV
Ɣ 6KRUW OHFWXUHV PLQXWHV
Ɣ *DPLILHG DVVHVVPHQW RIDIWHUFODVV TXL]]HV 
PLQXWHV
Ɣ (YDOXDWLRQ IHHGEDFNV PLQXWHV)LJXUH  $OLJQPHQW EHWZHHQ WKH 7KHRUHWLFDO )UDPHZRUN DQG ,QWHUYHQWLRQ
7KH WDUJHWHG SDUWLFLSDQWV ZHUH UHFUXLWHG IURP  VRSKRPRUHV ZKR PDMRUHG LQ
FRPSXWLQJ DW D XQLYHUVLW\ LQ PDLQODQG &KLQD 7KHUH DUH WZR JURXSV LQFOXGLQJ WKH H[SHULPHQWDOQ  DQG FRQWURO JURXSV Q   $ ZHHNO\ WZRKRXU -DYD SURJUDPPLQJ FRXUVH ZDV SURYLGHG
ZLWK RU ZLWKRXW JDPLILHG DVVHVVPHQW IRU VL[ ZHHNV 7KH VDPH FRPSXWLQJ LQVWUXFWRU WDXJKW WZRFODVVHV XQGHU D VLPLODU LQVWUXFWLRQDO VFKHGXOH IRU WKH SXUSRVH RI FRQWUROOLQJ WKH FRQGLWLRQV+RZHYHU WKH -DYD SURJUDPPLQJ FXUULFXOXP IRU WKH WZR JURXSV KDG GLIIHUHQW LQVWUXFWLRQDO
GHVLJQV 7KH LQVWUXFWRU WDXJKW WKH FRQWURO JURXS EDVHG RQ WKH WUDGLWLRQDO SDSHUDQGSHQ
DVVHVVPHQW :KHUHDV IRU WKH H[SHULPHQWDO JURXS VWXGHQWV OHDUQHG XQGHU JDPLILHG
DVVHVVPHQW GXULQJ FODVV DQG ILQLVKHG JDPLILHG DIWHUFODVV TXL]]HV E\ 4XL]L]] 2YHUDOO WKH
H[SHULPHQWDO JURXS ZDV GHOLYHUHG LQVWUXFWLRQV ZLWK WKH JDPLILHG DVVHVVPHQW ZKLOH WKH FRQWUROJURXS FRQWLQXHG WR XVH WKH WUDGLWLRQDO DVVHVVPHQW 7KH DVVHVVPHQW WDVNV ZHUH EDODQFHG LQ
WHUPV RI FRQWHQW DQG OHYHO RI GLIILFXOW\ 7KRXJK OHDUQLQJ ZLWK GLIIHUHQW DSSURDFKHV SDUWLFLSDQWV
LQ ERWK WKH FRQWURO DQG H[SHULPHQWDO JURXSV UHFHLYHG WKH VDPH W\SH RI IHHGEDFN ZKLFKLQFOXGHG FRUUHFW DQVZHUV DQG H[SODQDWLRQV IRU HDFK LWHP DIWHU FRPSOHWLQJ WKH DVVHVVPHQW
2QH RI WKH UHVHDUFKHUV ZDV LQ FKDUJH RI WKH JDPLILHG OHDUQLQJ HQYLURQPHQW VXSSRUW JDPLILHG
OHDUQLQJ GHVLJQ DQG WHDFKHUV WUDLQLQJ
7KH UHVHDUFK DFWLYLWLHV WRRN SODFH RYHU D VHYHQZHHN SHULRG DQG LQFOXGHG SUHSDUDWLRQLQWHUYHQWLRQ DQG DVVHVVPHQW DFWLYLWLHV :HHNO\KRXUVDFWLYLWLHV RI FODVVHV ZLWK JDPLILHG
DVVHVVPHQW IRU VL[ ZHHNV ZHUH GHVLJQHG E\ UHIHUULQJ WR WKH WKHRUHWLFDO IUDPHZRUN )LJXUH  )RU WKH H[SHULPHQWDO JURXS WKH VWXGHQWV ZHUH LQWURGXFHG WR WKH FRQFHSW RI JDPLILFDWLRQ WKH
UHOHYDQW SODWIRUPV DQG WKH JDPH HOHPHQWV LQ WKH ILUVW ZHHN )URP ZHHN WZR WR ZHHN VHYHQ DQ
LQWHUYHQWLRQ RI WHDFKLQJ DQG OHDUQLQJ DFWLYLWLHV WRRN SODFH DQG LQVWUXFWLRQV ZHUH JLYHQ RQ
GLIIHUHQW WRSLFV RI -DYD SURJUDPPLQJ 7KH HQWLUH FODVV ZDV FRQGXFWHG LQ D JDPLILHG
HQYLURQPHQW :KLOH WKH FRQWURO JURXS ZDV WDXJKW WUDGLWLRQDOO\ ZLWKRXW JDPLILHG DVVHVVPHQW $W
WKH HQG RI HDFK OHVVRQ VWXGHQWV ZRXOG WDNH DQ DIWHUFODVV TXL]

 5HVHDUFK 'HVLJQ DQG 'DWD &ROOHFWLRQ 3URFHGXUHV689

$ TXDQWLWDWLYH DSSURDFK ZLWK D SUHWHVWSRVWWHVW TXDVLH[SHULPHQWDO GHVLJQ ZDV XVHG LQ WKLVVWXG\ )LJXUH   7KH LPSOHPHQWDWLRQ RI JDPLILHG DVVHVVPHQW FRQVLVWHG RI WZR FRPSRQHQWVJDPLILHG TXHVWLRQ DQG DQVZHU VHVVLRQV GXULQJ FODVV DQG JDPLILHG TXL] VHVVLRQV DIWHU FODVV
4XDQWLWDWLYH GDWD RQ EXUQRXW ZHUH FROOHFWHG E\ WKH /HDUQLQJ %XUQRXW 6FDOH 6FKDXIHOLHW DO   6KRUWO\ EHIRUH DQG DIWHU WKH LQWHUYHQWLRQ WKH H[SHULPHQWDO DQG FRQWURO JURXSVWRRN SUHWHVWV DQG SRVWWHVWV RI OHDUQLQJ EXUQRXW $YHUDJHO\ LW WRRN DERXW  PLQXWHV IRU
SDUWLFLSDQWV WR FRPSOHWH WKH TXHVWLRQQDLUHV RQFH ZKLFK ZHUH SUHVHQWHG LQ HOHFWURQLF IRUPV'XULQJ WKH GDWD FROOHFWLRQ VWXGHQWV ZHUH JLYHQ WKH HTXHVWLRQQDLUH DQG DOO WKH GDWD ZDVLPSRUWHG LQWR 6366 DV WH[W ILOHV IRU GDWD DQDO\VLV
3UHWHVW RI OHDUQLQJ EXUQRXW([SHULPHQWDO JURXS&RQWURO JURXS/HFWXUHV ZLWK JDPLILHG
DVVHVVPHQW/HFWXUHV ZLWK WUDGLWLRQDO
DVVHVVPHQW

3RVWWHVW RI OHDUQLQJ EXUQRXW)LJXUH  7KH 2YHUDOO 5HVHDUFK 'HVLJQ *DPLILHG $VVHVVPHQW ,QVWUXFWLRQDO 'HVLJQ*DPLILHG 4XHVWLRQ DQG $QVZHU 6HVVLRQV

7KH GHVLJQ RI JDPLILHG TXHVWLRQ DQG DQVZHU VHVVLRQV LV VKRZQ LQ 7DEOH  EDVHG RQ WKH GHVLJQRI JDPLILHG DVVHVVPHQW E\ :HUEDFK HW DO :HUEDFK HW DO   5HJDUGLQJ WKH SRVVLELOLW\ RI
VWXGHQW FKRLFH WKH SULQFLSOH RI ³IUHHGRP RI FKRLFH´ ZDV DSSOLHG LQ WKLV VWXG\ 'LFKHYD HW DO  7KH SULQFLSOH RI ³VRFLDO HQJDJHPHQW´ LQFOXGLQJ LQGLYLGXDO DQG WHDP FRPSHWLWLRQV /L HWDO  ZDV DSSOLHG ZKHQ GHVLJQLQJ JDPLILHG TXHVWLRQ DQG DQVZHU VHVVLRQV 6WXGHQWVFRXOG ILQG LQIRUPDWLRQ RQ DOO WKHVH JDPLILHG FRPSRQHQWV LQ 0RRGOH DOORZLQJ WKHP WR WUDFN WKHLUSHUIRUPDQFH DQG SURJUHVV RYHU WLPH
7DEOH  7KH 'HVLJQ RI *DPLILHG 4XHVWLRQ DQG $QVZHU 6HVVLRQV :HUEDFK HW DO '\QDPLFV(PRWLRQV

3URJUHVVLRQ
5HODWLRQVKLSV0HFKDQLFV5HZDUG&RPSHWLWLRQ&RRSHUDWLRQ
5HVRXUFH $FTXLVLWLRQ)HHGEDFN
5HZDUG5HVRXUFH $FTXLVLWLRQ)HHGEDFN&RRSHUDWLRQ&RPSRQHQWV%DGJHV%DGJHV /HDGHU %RDUG
7HDPV
3RLQWV
3RLQWV %DGJHV /HDGHU %RDUG /HYHOV%DGJHV
3RLQWV
3RLQWV %DGJHV /HDGHU %RDUG /HYHOV
7HDPV

7KH ILYH W\SHV RI EDGJHV ³$FWLYH EDGJH´ ³&RXUDJH EDGJH´ ³:HOO GRQH EDGJH´
³&KDOOHQJH EDGJH´ DQG ³&RRSHUDWLRQ EDGJH´ ZHUH DZDUGHG WR VWXGHQWV EDVHG RQ WKHLUSHUIRUPDQFH LQ OHDUQLQJ VHH 7DEOH  IRU GHWDLOHG GHVLJQ  (DFK EDGJH UHSUHVHQWHG D
SDUWLFXODU DFKLHYHPHQW VWXGHQWV FRXOG FROOHFW WKHP DV WKH\ SURJUHVVHG WKURXJK WKH
DVVHVVPHQW 7KH JDPLILHG DVVHVVPHQW V\VWHP LPSOHPHQWHG D SRLQWEDVHG UHZDUG VWUXFWXUHSURYLGLQJ LPPHGLDWH IHHGEDFN IRU FRUUHFW DQVZHUV DQG WKXV SURPRWLQJ DFFXUDF\ 7KLVLQFHQWLYL]DWLRQ FRPELQHG ZLWK WKH RSSRUWXQLW\ IRU VWXGHQWV WR IRUP FROODERUDWLYH WHDPV

690

IDFLOLWDWHG QRW RQO\ LQGLYLGXDO OHDUQLQJ EXW DOVR FROOHFWLYH DFKLHYHPHQW WKURXJK WKHDFFXPXODWLRQ RI SRLQWV DQG EDGJHV 7KH YLVLEOH OHDGHUERDUG IRVWHUHG D FRPSHWLWLYH
HQYLURQPHQW DQG IXUWKHU PRWLYDWHG VWXGHQWV WR H[FHO 7KH PXOWLWLHUHG VWUXFWXUH RI WKH
DVVHVVPHQW ZKLFK IHDWXUHG HVFDODWLQJ OHYHOV RI GLIILFXOW\ DQG UHZDUG ZDV DLPHG DW
HQJHQGHULQJ VWXGHQWV D VHQVH RI DFFRPSOLVKPHQW WKHUHE\ LQVSLULQJ FRQWLQXHG OHDUQLQJSURJUHVVLRQ
7DEOH  7\SHV DQG 'HVFULSWLRQV RI %DGJHV 8VHG LQ WKH 4XHVWLRQ DQG $QVZHU 6HVVLRQV
%DGJH

7\SH

'HVFULSWLRQ$FWLYH EDGJH:KHQ D VWXGHQW WDNHV WKH LQLWLDWLYH WR DQVZHU D TXHVWLRQKHVKH ZRXOG EH JLYHQ DQ ³$FWLYH EDGJH´&RXUDJH EDGJH:KHQ D VK\ VWXGHQW DQVZHUV D TXHVWLRQ IRU WKH ILUVW
WLPH WKH\ ZRXOG EH JLYHQ D ³&RXUDJH EDGJH´:HOO GRQH EDGJH:KHQ D VWXGHQW JLYHV D SHUIHFW DQVZHU D ³:HOO GRQH
EDGJH´ ZRXOG EH DZDUGHG

&KDOOHQJH EDGJH

4XHVWLRQV ZRXOG EH FODVVLILHG DV HDV\ PHGLXP DQG
GLIILFXOW OHYHOV 7KH VWXGHQW ZRXOG UHFHLYH D ³&KDOOHQJH
EDGJH´ ZKHQ KHVKH FRPSOHWHV D GLIILFXOW TXHVWLRQ 2QH³&KDOOHQJH EDGJH´ LV HTXLYDOHQW WR  ³$FWLYH EDGJH´

&RRSHUDWLRQ EDGJH'XULQJ JURXS WDVNV LI D JURXS PHPEHU VKDUHV D JRRG
VXJJHVWLRQ IRU VROYLQJ D SUREOHP KHVKH ZRXOG UHFHLYH D³&RRSHUDWLRQ EDGJH´

$OO WKHVH HOHPHQWV VWDWHG LQ 7DEOH  KDG WKH SRWHQWLDO WR PLWLJDWH VWXGHQW EXUQRXW E\
SURYLGLQJ LPPHGLDWH IHHGEDFN DQG LQFHQWLYHV WR PRWLYDWH DQG HQJDJH VWXGHQWV 6DLOHU 6DLOHU
  7KURXJK WHDPZRUN DQG FRPSHWLWLRQ WKH LQFRUSRUDWLRQ RI JDPLILFDWLRQ HOHPHQWVSURPRWHG LQWHUDFWLRQ DQG VXSSRUW DPRQJ VWXGHQWV LQFUHDVLQJ WKHLU VHOIFRQILGHQFH DQG DELOLW\WR ZRUN WRJHWKHU +XDQJ HW DO   3RLQWV OHYHOV DQG GLIIHUHQW W\SHV RI EDGJHV FDQ
GHFUHDVH OHDUQLQJ EXUQRXW E\ SURYLGLQJ FODULW\ RI SXUSRVH DQG D VHQVH RI SURJUHVVLYHDFKLHYHPHQW 0HNOHU HW DO  

*DPLILHG 4XL]]HV 6HVVLRQV*DPLILHG TXL]]HV ZHUH WKH SURFHVV RI WUDQVIHUULQJ WKH DIWHUFODVV TXL]]HV IURP WKH HOHDUQLQJ
SODWIRUP WR WKH JDPLILHG DVVHVVPHQW V\VWHP 'HODFUX]   ,W PHDQW XVLQJ WKH H[LVWLQJJDPLILHG DVVHVVPHQW V\VWHP WR FRQGXFW WKH DIWHUFODVV TXL]]HV 7KH LPSOHPHQWDWLRQRI JDPLILHG DVVHVVPHQW LQYROYHG VHOHFWLQJ D VXLWDEOH V\VWHP FUHDWLQJ TXHVWLRQV PDQDJLQJFODVV GDWD DQG VHWWLQJ HYDOXDWLRQ UXOHV $Q DSSURSULDWH JDPLILHG DVVHVVPHQW V\VWHP ZDVVHOHFWHG EDVHG RQ IDFWRUV OLNH IXQFWLRQDOLW\ DQG FRVW $VVHVVPHQW TXHVWLRQV ZHUH FUHDWHGZKLFK LQFOXGHG D PL[ RI PXOWLSOH FKRLFH DQG MXGJHPHQW TXHVWLRQV DQG VFRUHG RXW RI  &ODVVGDWD ZDV HIIHFWLYHO\ PDQDJHG WR HQVXUH WKH VXFFHVVIXO LPSOHPHQWDWLRQ RI JDPLILHG
DVVHVVPHQW DQG HYDOXDWLRQ UXOHV ZHUH HVWDEOLVKHG WR HQVXUH IDLU DQG DFFXUDWH DVVHVVPHQW*DPLILHG TXL]]HV VHVVLRQV XVHG 4XL]L]] DV D WRRO 4XL]L]] LQFRUSRUDWHV WZR GLVWLQFW
JDPH PRGHV DQ DVVLJQPHQW PRGH ZKLFK RSHUDWHV DW D VHOISDFHG WHPSR DQG D FODVVURRP
PRGH ZKLFK IROORZV D UHDOWLPH SUHGHWHUPLQHG SDFH 7KH DVVLJQPHQW PRGH ZDV VHOHFWHG IRUWKLV VWXG\ WR DFFRPPRGDWH WKH OHDUQLQJ QHHGV RI VWXGHQWV DIWHU FODVV )XUWKHUPRUH YDULRXVJDPLILFDWLRQ HOHPHQWV ZHUH LQWHJUDWHG LQFOXGLQJ SRLQWV OHDGHUERDUGV FXVWRPL]DEOH DYDWDUVDXGLWRU\ FXHV FRXQWGRZQ WLPHUV DQG PRUH

691

 'DWD $QDO\VLV
8VLQJ ,%0 6366  WKH TXDQWLWDWLYH GDWD ZHUH DQDO\]HG &URQEDFK¶V DOSKD ZDV LQLWLDOO\
GHWHUPLQHG XVLQJ WKH SUHWHVW GDWD ZLWK WKH JRDO RI JXDUDQWHHLQJ LQWHUQDO FRQVLVWHQF\ IRU HYHU\
GLPHQVLRQ RI WKH /HDUQLQJ %XUQRXW ,QYHQWRU\6WXGHQW 6XUYH\ LQ WKH FRQWH[W RI WKH UHVHDUFK
7KHQ XWLOL]LQJ ERWK SUHWHVW DQG SRVWWHVW GDWD GHVFULSWLYH DQDO\VLV DQG DQDO\VLV RI FRYDULDQFH
ZHUH FDUULHG RXW WR SURYLGH FRQWH[W DV ZHOO DV DQDO\]H WKH RYHUDOO VWXGHQW OHDUQLQJ EXUQRXW DQGHDFK VSHFLILF GLPHQVLRQ EHWZHHQ WKH H[SHULPHQWDO DQG FRQWURO JURXSV 7KH SUHWHVW VFRUHV RIOHDUQLQJ LQ WZR JURXSV ZHUH FRQVLGHUHG LQ WKLV VWXG\ DV IDFWRUV WKDW PD\ KDYH DQ LPSDFW RQ WKHSRVWWHVW VFRUHV EXW DUH XQFRQWUROODEOH 7KHUHIRUH $1&29$ ZDV XVHG WR PRGLI\ SRVWWHVWUHVXOWV IRU YDULDEOHV DQG UHPRYH WKHVH XQGHVLUHG FRQVHTXHQFHV ZLWK D FRQILGHQFH LQWHUYDO RI
 3DUWLFLSDQWV¶ RYHUDOO OHDUQLQJ EXUQRXW OHYHO LQ WKH SRVWWHVW DQG HDFK GLPHQVLRQ RI LW ZHUH
WKH GHSHQGHQW YDULDEOHV DQG WKH LQGHSHQGHQW YDULDEOH ZDV ³JURXS´ WKH FRQWURO JURXSJURXS  WKH H[SHULPHQWDO JURXS JURXS   $OO WKH EDVLF $1&29$ DVVXPSWLRQV ZHUH
FRQILUPHG SULRU WR WKH DQDO\VLV 7R H[DPLQH WKH KRPRJHQHLW\ RI YDULDQFH DFURVV WKH WZRJURXSV /HYHQH¶V WHVW ZDV DSSOLHG

 5HVXOWV&URQEDFK¶V DOSKD ZDV GHWHUPLQHG XVLQJ WKH SUHWHVW GDWD WR JXDUDQWHH LQWHUQDO FRQVLVWHQF\ IRU
HYHU\ GLPHQVLRQ RI WKH /HDUQLQJ %XUQRXW ,QYHQWRU\ SULRU WR WKH SRVWWHVW $OO GLPHQVLRQV¶&URQEDFK¶V DOSKDV ZHUH PRUH WKDQ  αEX  αCY  αLE  αD  
LQGLFDWLQJ WKDW WKH /HDUQLQJ %XUQRXW ,QYHQWRU\6WXGHQW 6XUYH\ ZDV D WUXVWZRUWK\ LQVWUXPHQWZLWK LQWHUQDO FRQVLVWHQF\ 7KH KRPRJHQHLW\ RI YDULDQFH ZDV DVVHVVHG XVLQJ /HYHQH¶V WHVW
DQG WKH ILQGLQJV )(;  S !  )&<  S !  )/(  S   ) 
S !  LQGLFDWHG WKDW H[FHSW IURP WKH GLPHQVLRQ RI ORZ HIILFDF\ WKHUH ZDV QR YDULDWLRQ LQ
WKH YDULDQFHV DFURVV JURXSV 7KHQ DIWHU FRQGXFWLQJ WKH .UXVNDO:DOOLV WHVW IRU ORZ HIILFDF\ LQ
WZR JURXSV 3    WKHUH ZDV DOVR D VLJQLILFDQW GLIIHUHQFH LQ ORZ HIILFDF\ 7KH OHDUQLQJ
EXUQRXW SUHWHVW DQG PRGLILHG SRVWWHVW ILQGLQJV DUH DQDO\]HG E\ GHVFULSWLYH VWDWLVWLFV DQG$1&29$ DV VKRZQ LQ 7DEOH  7KH UHVXOWV LQGLFDWHG WKDW WKH RYHUDOO OHDUQLQJ EXUQRXW RI WKH
H[SHULPHQWDO JURXS GHFUHDVHG VLJQLILFDQWO\ DIWHU WKH LQWHUYHQWLRQ
7DEOH  'HVFULSWLYH 6WDWLVWLFV DQG $1&29$ 5HVXOWV RI /HDUQLQJ %XUQRXW'LPHQVLRQ

3UHWHVW
0HDQ
6'2YHUDOO /HDUQLQJ %XUQRXW([SHULPHQWDO *URXS
&RQWURO *URXS
(PRWLRQDO ([KDXVWLRQ (;([SHULPHQWDO *URXS
&RQWURO *URXS
&\QLFLVP &<([SHULPHQWDO *URXS
&RQWURO *URXS
/RZ (IILFDF\ /(([SHULPHQWDO *URXS
&RQWURO *URXS
1RWHV ([SHULPHQWDO *URXS Q
EDVHG RQ WKH YDULDEOHV

3RVWWHVW
0HDQ
6'

)

$1&29$
3

𝜂2


D
D




 






D
D













D
D




 






D
D




 

  &RQWURO *URXS Q  7KH SRVWWHVW KDV EHHQ PRGLILHG

7R GHWHUPLQH ZKHWKHU WKHUH ZHUH DQ\ VLJQLILFDQW YDULDWLRQV LQ RYHUDOO OHDUQLQJ EXUQRXWOHYHO DQG GLPHQVLRQV RI OHDUQLQJ EXUQRXW EHWZHHQ WKH H[SHULPHQWDO DQG FRQWURO JURXSV LQ WKH

692

SRVWWHVW UHSHDWHG PHDVXUHV RI $1&29$ ZHUH FDUULHG RXW 2YHUDOO VWXGHQWV LQ WKH
H[SHULPHQWDO JURXS KDG D ORZHU OHYHO RI OHDUQLQJ EXUQRXW WKDQ WKDW RI WKH FRQWURO JURXSDFFRUGLQJ WR WKH UHVXOWV 7DEOH  )  3   η2   ,Q SDUWLFXODU WKH SRVWWHVWUHYHDOHG VLJQLILFDQW GLIIHUHQFHV EHWZHHQ WKH WZR JURXSV RQ F\QLFLVP )  3   𝜂2 DQG ORZ HIILFDF\ )  3   𝜂2   7KH ILUVW FRQFOXVLRQ LV WKDWLQVWUXFWLRQV ZLWK JDPLILHG DVVHVVPHQW LQ FRPSXWLQJ FXUULFXOXP KDYH REYLRXVO\ DOOHYLDWHG WKH
H[SHULPHQWDO JURXS¶V RYHUDOO OHDUQLQJ EXUQRXW HVSHFLDOO\ LQ WHUPV RI F\QLFLVP HJ VWXGHQWVKDYH EHFRPH OHVV LQWHUHVWHG LQ VWXGLHV VLQFH HQUROOPHQW LQ WKH XQLYHUVLW\ DQG ORZ HIILFDF\
HJ VWXGHQWV FDQ HIIHFWLYHO\ VROYH WKH SUREOHPV WKDW DULVH LQ VWXGLHV  +RZHYHU WKHUH ZDV QR
VLJQLILFDQW GLIIHUHQFH LQ HPRWLRQDO H[KDXVWLRQ EHWZHHQ WKH H[SHULPHQWDO DQG FRQWURO JURXSVDFFRUGLQJ WR WKH SRVWWHVW VFRUHV 7KH VHFRQG FRQFOXVLRQ LV WKDW JDPLILHG DVVHVVPHQW GLG QRWDIIHFW WKH OHDUQLQJ EXUQRXW RI XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV LQ WKH HPRWLRQDO H[KDXVWLRQ
GLPHQVLRQ 2YHUDOO WKH VWXG\ IRXQG WKDW JDPLILHG DVVHVVPHQW DIIHFWV WKH OHDUQLQJ EXUQRXW RI
XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV H[FHSW WKH HPRWLRQDO H[KDXVWLRQ GLPHQVLRQ 'LVFXVVLRQ DQG &RQFOXVLRQ
7KLV VWXG\ ZDV D SUHWHVWSRVWWHVW TXDVLH[SHULPHQWDO GHVLJQ WR LQYHVWLJDWH WKH LPSDFW RI
JDPLILHG DVVHVVPHQW RQ OHDUQLQJ EXUQRXW 7KH ILQGLQJV RI WKH VWXG\ GHPRQVWUDWHG WKDWJDPLILHG DVVHVVPHQW KDG D VLJQLILFDQW HIIHFW RQ DOOHYLDWLQJ WKH OHDUQLQJ EXUQRXW RI
XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV SDUWLFXODUO\ UHJDUGLQJ F\QLFLVP DQG ORZ HIILFDF\ WKRXJKQR VLJQLILFDQW GLIIHUHQFH ZDV PDQLIHVWHG LQ HPRWLRQDO H[KDXVWLRQ 7KH EHQHILFLDO HIIHFW RI
JDPLILHG DVVHVVPHQW LQ XQGHUJUDGXDWH HGXFDWLRQ LV EHWWHU UHFRJQL]HG 7KURXJK LQVWUXFWLRQDO
SUDFWLFHV LW DLGHG WHDFKHUV LQ UHDOL]LQJ WKH SURPLVLQJ EHQHILWV RI JDPLILHG DVVHVVPHQW WRDOOHYLDWH RYHUDOO DQG WZR RXW RI WKUHH GLPHQVLRQV RI VWXGHQW OHDUQLQJ EXUQRXW
7KH DIRUHPHQWLRQHG UHVXOWV DUH LQ OLQH ZLWK VRPH UHVHDUFK RQ JDPLILHG DVVHVVPHQW LQ
HGXFDWLRQ 'LDV  +DVVDQ HW DO  :DQJ 7DKLU  =DLQXGGLQ HW DO  
VXJJHVWLQJ WKDW JDPLILHG DVVHVVPHQW FRXOG DOOHYLDWH EXUQRXW LQ WKH F\QLFLVP DQG ORZ HIILFDF\,Q RUGHU WR VHH KRZ JDPLILHG DVVHVVPHQW DIIHFWV VWXGHQWV¶ OHDUQLQJ EXUQRXW SRVLWLYHO\ RU
QHJDWLYHO\ LQ FHUWDLQ DVSHFW V  WKH VXEVHTXHQW VWDJH RI WKLV SURMHFW V TXDOLWDWLYH UHVHDUFK KDV
EHHQ DQDO\]LQJ WKH GDWD IURP VWXGHQW LQWHUYLHZV $FDGHPLF SHUIRUPDQFH SHUVRQDOLW\ WUDLWVVWXG\ KDELWV DQG IDPLO\ EDFNJURXQG KDYH DOO EHHQ SURSRVHG DV SRWHQWLDO PRGHUDWRUV WKDWFRXOG DIIHFW VWXGHQWV¶ OHDUQLQJ EXUQRXW $QRWKHU SDSHU ZLOO SUHVHQW DQG GLVFXVV WKH UHODWHGILQGLQJV DQG FRQFOXVLRQV
7KH VWXG\ LQFOXGHG VHYHUDO OLPLWDWLRQV )LUVWO\ LW LV VWLOO XQFOHDU KRZ OHDUQLQJ EXUQRXWYDULHV LQ GLIIHUHQW HGXFDWLRQDO FRQWH[WV )XWXUH VWXGLHV DUH H[SHFWHG WR WDNH GLIIHUHQW VXEMHFWV¶LQVWUXFWLRQDO DQG VWXGHQWV¶ LQGLYLGXDO IDFWRUV LQWR FRQVLGHUDWLRQ WR GUDZ D ZLGHU UDQJH RI
FRQFOXVLRQV %D\GDV &LFHN   $GGLWLRQDOO\ EHFDXVH WKH LQWHUYHQWLRQ RQO\ ODVWHG VL[ZHHNV LWV LPSDFW RQ OHDUQLQJ EXUQRXW PD\ KDYH EHHQ UDWKHU OLPLWHG :LWK ORQJLWXGLQDO
H[SHULPHQWV IXUWKHU UHVHDUFK FDQ EH GRQH 0RUHRYHU WKLV VWXG\¶V VDPSOH VL]H RI VWXGHQWV ZDV VPDOO ZKLFK PLJKW UHVWULFW WKH JHQHUDOL]DELOLW\ RI UHVHDUFK ILQGLQJV %\ XVLQJPRUH VXEVWDQWLDO VDPSOH VL]HV WKH UHOLDELOLW\ FDQ EH LPSURYHG ,Q DGGLWLRQ WR WKH SUHYLRXVO\
PHQWLRQHG OLPLWDWLRQV LW LV FUXFLDO WR UHFRJQL]H WKDW ODQJXDJH EDUULHUV PD\ KDYH FDXVHG
GLIILFXOWLHV IRU VRPH SDUWLFLSDQWV LQ XQGHUVWDQGLQJ WKH VFDOH LWHPV 7R DGGUHVV WKLV SRWHQWLDOLVVXH IXWXUH VWXGLHV FRXOG FRQVLGHU LQYROYLQJ ELOLQJXDO SDUWLFLSDQWV RU FRQGXFWLQJ VHSDUDWH
VWXGLHV LQ GLIIHUHQW ODQJXDJH FRQWH[WV )XUWKHUPRUH WKH GDWD FROOHFWLRQ PDLQO\ UHOLHG RQVHOIUHSRUWLQJ ZKLFK FRXOG PDNH UHVXOWV UHODWLYHO\ VXEMHFWLYH DQG LPSUHFLVH ,W LV SRVVLEOH WR
FRQGXFW H[WHQVLYH VWXG\ XWLOL]LQJ D YDULHW\ RI WRROV VXFK DV OHYHUDJLQJ PDFKLQH OHDUQLQJDOJRULWKPV WR DVVLVW LQ FODVVURRP REVHUYDWLRQV DQG LQYROYLQJ SV\FKRORJLVWV LQ DVVHVVLQJ WKHOHYHO RI VWXGHQW EXUQRXW 'HVSLWH WKH IDFW WKDW VLJQLILFDQW GLIIHUHQFHV ZHUH IRXQG LQ WZR RXW RIWKUHH GLPHQVLRQV RI OHDUQLQJ EXUQRXW WKHLU HIIHFW VL]HV ZHUH PLQLPDO HJ 𝜂2 &<  
VXJJHVWLQJ WKDW WKH GLIIHUHQFHV ZHUH QRW VXEVWDQWLDOO\ VLJQLILFDQW &RKHQ   7KHUH LV VWLOO D
QHHG IRU PRUH HPSLULFDO VWXGLHV LQYHVWLJDWLQJ WKH HIIHFWV RI JDPLILHG DVVHVVPHQW

693

7KLV LV WKH ILUVWVWDJH TXDQWLWDWLYH VWXG\ LQYHVWLJDWLQJ WKH LPSDFW RI JDPLILHG
DVVHVVPHQW RQ WKH OHDUQLQJ EXUQRXW RI XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV 7KH ILQGLQJV RI WKH
VHFRQGVWDJH TXDOLWDWLYH VWXG\ ZLOO H[SODLQ KRZ JDPLILHG DVVHVVPHQW DIIHFWV WKH OHDUQLQJ
EXUQRXW RI XQGHUJUDGXDWH FRPSXWLQJ VWXGHQWV 7KH IXUWKHU ILQGLQJV ZLOO EH UHSRUWHG LQ DQRWKHUSDSHU $IWHU WKDW PRUH FRPSUHKHQVLYH SURSRVDOV IRU WKH LQFRUSRUDWLRQ RI JDPLILHG DVVHVVPHQW
LQWR WHUWLDU\ HGXFDWLRQ FDQ EH VXJJHVWHG

$FNQRZOHGJHPHQWV:H ZRXOG OLNH WR H[SUHVV RXU JUDWLWXGH WR HYHU\RQH ZKR SDUWLFLSDWHG LQ WKH VWXG\ DQG KHOSHG WR
UHYLHZ WKH SDSHU

5HIHUHQFHV$KPDG $ =HVKDQ ) .KDQ 0 6 0DUULDP 5 $OL $ 6DPUHHQ $   7KH LPSDFW RI
JDPLILFDWLRQ RQ OHDUQLQJ RXWFRPHV RI FRPSXWHU VFLHQFH PDMRUV $&0 7UDQVDFWLRQV RQ&RPSXWLQJ (GXFDWLRQ 72&(     $OPXWDLUL + $OVXEDLHL $ $EGXOMDZDG 6 $OVKDWWL $ )HNLK5RPGKDQH ) +XVQL 0 -DKUDPL +  3UHYDOHQFH RI EXUQRXW LQ PHGLFDO VWXGHQWV $ V\VWHPDWLF UHYLHZ DQG PHWDDQDO\VLV,QWHUQDWLRQDO -RXUQDO RI 6RFLDO 3V\FKLDWU\    $QGULDPLDULVRD 5   ,PSDFW RI JDPLILFDWLRQ RQ VWXGHQW HQJDJHPHQW LQ JUDGXDWH PHGLFDO VWXGLHV'RFWRUDO GLVVHUWDWLRQ :DOGHQ 8QLYHUVLW\ $WHV %   (IIHFW RI 6ROXWLRQ )RFXVHG *URXS &RXQVHOLQJ IRU +LJK 6FKRRO 6WXGHQWV LQ 2UGHU WR
6WUXJJOH ZLWK 6FKRRO %XUQRXW -RXUQDO RI (GXFDWLRQ DQG 7UDLQLQJ 6WXGLHV    
%D\GDV 2 &LFHN 0   7KH H[DPLQDWLRQ RI WKH JDPLILFDWLRQ SURFHVV LQ XQGHUJUDGXDWH
HGXFDWLRQ D VFDOH GHYHORSPHQW VWXG\ 7HFKQRORJ\ 3HGDJRJ\ DQG (GXFDWLRQ    
&D]DQ $0   /HDUQLQJ PRWLYDWLRQ HQJDJHPHQW DQG EXUQRXW DPRQJ XQLYHUVLW\ VWXGHQWV
3URFHGLD6RFLDO DQG %HKDYLRUDO 6FLHQFHV  
&KXQJ & + /LQ < <   2QOLQH ' JDPLILFDWLRQ IRU WHDFKLQJ D KXPDQ UHVRXUFH GHYHORSPHQWFRXUVH -RXUQDO RI &RPSXWHU $VVLVWHG /HDUQLQJ    &RKHQ -   6WDWLVWLFDO SRZHU DQDO\VLV IRU WKH EHKDYLRUDO VFLHQFHV $FDGHPLF SUHVV'HODFUX] * &   *DPHV DV )RUPDWLYH $VVHVVPHQW (QYLURQPHQWV ([DPLQLQJ WKH ,PSDFW RI([SODQDWLRQV RI 6FRULQJ DQG ,QFHQWLYHV RQ 0DWK /HDUQLQJ *DPH 3HUIRUPDQFH DQG +HOS
6HHNLQJ &5(667 5HSRUW  1DWLRQDO &HQWHU IRU 5HVHDUFK RQ (YDOXDWLRQ 6WDQGDUGV DQG
6WXGHQW 7HVWLQJ &5(667 'HWHUGLQJ 6 6LFDUW 0 1DFNH / 2 +DUD . 'L[RQ '   *DPLILFDWLRQ XVLQJ JDPHGHVLJQ
HOHPHQWV LQ QRQJDPLQJ FRQWH[WV ,Q &+,  H[WHQGHG DEVWUDFWV RQ KXPDQ IDFWRUV LQ FRPSXWLQJ
V\VWHPV SS  'LDV -   7HDFKLQJ RSHUDWLRQV UHVHDUFK WR XQGHUJUDGXDWH PDQDJHPHQW VWXGHQWV 7KH UROH RIJDPLILFDWLRQ 7KH ,QWHUQDWLRQDO -RXUQDO RI 0DQDJHPHQW (GXFDWLRQ    
'LFKHYD ' 'LFKHY & $JUH * $QJHORYD *   *DPLILFDWLRQ LQ HGXFDWLRQ $ V\VWHPDWLFPDSSLQJ VWXG\ -RXUQDO RI HGXFDWLRQDO WHFKQRORJ\ VRFLHW\    
)HOD]D ( )LQG\DUWLQL $ 6HW\RULQL ' 0XVWLND 5   +RZ 0RWLYDWLRQ &RUUHODWHV ZLWK $FDGHPLF%XUQRXW 6WXG\ &RQGXFWHG LQ 8QGHUJUDGXDWH 0HGLFDO 6WXGHQWV (GXFDWLRQ LQ 0HGLFLQH -RXUQDO  
)LRULOOL & %DUQL ' 5XVVR & 0DUFKHWWL 9 $QJHOLQL * 5RPDQR /   6WXGHQWV¶ %XUQRXW DW
8QLYHUVLW\ 7KH 5ROH RI *HQGHU DQG :RUNHU 6WDWXV ,QWHUQDWLRQDO -RXUQDO RI (QYLURQPHQWDO
5HVHDUFK DQG 3XEOLF +HDOWK    *HRUJLRX . 1LNRODRX ,   $UH DSSOLFDQWV LQ IDYRU RI WUDGLWLRQDO RU JDPLILHG DVVHVVPHQW
PHWKRGV" ([SORULQJ DSSOLFDQW UHDFWLRQV WRZDUGV D JDPLILHG VHOHFWLRQ PHWKRG &RPSXWHUV LQ+XPDQ %HKDYLRU  
*|NVQ ' 2 *UVR\ *   &RPSDULQJ VXFFHVV DQG HQJDJHPHQW LQ JDPLILHG OHDUQLQJH[SHULHQFHV YLD .DKRRW DQG 4XL]L]] &RPSXWHUV (GXFDWLRQ  *UDGLVNL , 3 %RURYHFNL $ ûXUNRYLü 0 6DQ0DUWtQ 0 'HOJDGR %ROWRQ 5 & 9LYDQFR /  %XUQRXW LQ LQWHUQDWLRQDO PHGLFDO VWXGHQWV FKDUDFWHUL]DWLRQ RI SURIHVVLRQDOLVP DQG ORQHOLQHVV DV
SUHGLFWLYH IDFWRUV RI EXUQRXW ,QWHUQDWLRQDO -RXUQDO RI (QYLURQPHQWDO 5HVHDUFK DQG 3XEOLF+HDOWK    694

*XDQJZHQ 6 :DQMLH % :HQ +   /HDUQLQJ EXUQRXW DQG LWV UHODWLRQVKLS ZLWK IXWXUH WLPHSHUVSHFWLYH DQG DFKLHYHPHQW JRDO RULHQWDWLRQ DPRQJ PLGGOH VFKRRO VWXGHQWV 6WXGLHV RI
3V\FKRORJ\ DQG %HKDYLRU    +DVVDQ 0 $ +DELED 8 0DMHHG ) 6KRDLE 0   $GDSWLYH JDPLILFDWLRQ LQ HOHDUQLQJ EDVHG
RQ VWXGHQWV¶ OHDUQLQJ VW\OHV ,QWHUDFWLYH /HDUQLQJ (QYLURQPHQWV    
+XDQJ % +HZ . ) /R & .   ,QYHVWLJDWLQJ WKH HIIHFWV RI JDPLILFDWLRQHQKDQFHG IOLSSHGOHDUQLQJ RQ XQGHUJUDGXDWH VWXGHQWV¶ EHKDYLRUDO DQG FRJQLWLYH HQJDJHPHQW ,QWHUDFWLYH/HDUQLQJ (QYLURQPHQWV    
/DQGHUV 5 1 %DXHU . 1 &DOODQ 5 & $UPVWURQJ 0 %   3V\FKRORJLFDO WKHRU\ DQG WKH
JDPLILFDWLRQ RI OHDUQLQJ *DPLILFDWLRQ LQ HGXFDWLRQ DQG EXVLQHVV 
/HH && +DR < /HH . 6 6LP 6 & +XDQJ &&   ,QYHVWLJDWLRQ RI WKH HIIHFWV RI DQ
RQOLQH LQVWDQW UHVSRQVH V\VWHP RQ VWXGHQWV LQ D PLGGOH VFKRRO RI D UXUDO DUHD &RPSXWHUV LQ+XPDQ %HKDYLRU  
/L : *URVVPDQ 7 )LW]PDXULFH *   &$'DPHQW D JDPLILHG PXOWLSOD\HU VRIWZDUH WXWRULDO
V\VWHP 3URFHHGLQJV RI WKH 6,*&+, &RQIHUHQFH RQ +XPDQ )DFWRUV LQ &RPSXWLQJ 6\VWHPV
0HLHU 6 7 6FKPHFN 5 5   7KH EXUQHGRXW FROOHJH VWXGHQW $ GHVFULSWLYH SURILOH -RXUQDO RI
FROOHJH VWXGHQW SHUVRQQHO0HNOHU ( ' %UKOPDQQ ) 7XFK $ 1 2SZLV .   7RZDUGV XQGHUVWDQGLQJ WKH HIIHFWV RILQGLYLGXDO JDPLILFDWLRQ HOHPHQWV RQ LQWULQVLF PRWLYDWLRQ DQG SHUIRUPDQFH &RPSXWHUV LQ +XPDQ%HKDYLRU  
2OLYHU (   *DPLILFDWLRQ DV WUDQVIRUPDWLYH DVVHVVPHQW LQ KLJKHU HGXFDWLRQ +76 7KHRORJLFDO
6WXGLHV    
21HLOO . 5REE 0 .HQQHG\ 5 %KDWWDFKDU\D $ 'RPLQLFL 1 5 0XUSK\ $   0RELOH
WHFKQRORJ\ MXVWLQWLPH OHDUQLQJ DQG JDPLILFDWLRQ ,QQRYDWLYH VWUDWHJLHV IRU D &$87, HGXFDWLRQSURJUDP 2Q/LQH -RXUQDO RI 1XUVLQJ ,QIRUPDWLFV   
3LWR\R 0 '   *DPLILFDWLRQ EDVHG DVVHVVPHQW $ WHVW DQ[LHW\ UHGXFWLRQ WKURXJK JDPH HOHPHQWV
LQ 4XL]L]] SODWIRUP ,-(5 ,QGRQHVLDQ -RXUQDO RI (GXFDWLRQDO 5HVHDUFK     
3LWR\R 0 ' $VLE $   *DPLILFDWLRQ%DVHG $VVHVVPHQW 7KH :DVKEDFN (IIHFW RI 4XL]L]] RQ
6WXGHQWV /HDUQLQJ LQ +LJKHU (GXFDWLRQ ,QWHUQDWLRQDO -RXUQDO RI /DQJXDJH (GXFDWLRQ   

5LEHLUR , - 3HUHLUD 5 )UHLUH , 9 GH 2OLYHLUD % * &DVRWWL & $ %RHU\ ( 1   6WUHVV DQG
TXDOLW\ RI OLIH DPRQJ XQLYHUVLW\ VWXGHQWV $ V\VWHPDWLF OLWHUDWXUH UHYLHZ +HDOWK 3URIHVVLRQV(GXFDWLRQ    
6DLOHU 0 6DLOHU 0   *DPLILFDWLRQ RI LQ澻FODVV DFWLYLWLHV LQ IOLSSHG FODVVURRP OHFWXUHV %ULWLVK-RXUQDO RI (GXFDWLRQDO 7HFKQRORJ\    
6DOHHP $ 1 1RRUL 1 0 2]GDPOL )   *DPLILFDWLRQ DSSOLFDWLRQV LQ (OHDUQLQJ $ OLWHUDWXUH
UHYLHZ 7HFKQRORJ\ .QRZOHGJH DQG /HDUQLQJ    
6FKDXIHOL : % 0DUWLQH] , 0 3LQWR $ 0 6DODQRYD 0 %DNNHU $ %   %XUQRXW DQG
HQJDJHPHQW LQ XQLYHUVLW\ VWXGHQWV $ FURVVQDWLRQDO VWXG\ -RXUQDO RI FURVVFXOWXUDO SV\FKRORJ\   
6KXWH 9 -   )RFXV RQ IRUPDWLYH IHHGEDFN 5HYLHZ RI HGXFDWLRQDO UHVHDUFK    
7VD\ & ++ .RILQDV $ /XR -   (QKDQFLQJ VWXGHQW OHDUQLQJ H[SHULHQFH ZLWKWHFKQRORJ\PHGLDWHG JDPLILFDWLRQ $Q HPSLULFDO VWXG\ &RPSXWHUV (GXFDWLRQ  
:DQJ $ , 7DKLU 5   7KH HIIHFW RI XVLQJ .DKRRW IRU OHDUQLQJ±$ OLWHUDWXUH UHYLHZ &RPSXWHUV(GXFDWLRQ  
:DWVRQ : 5 0RQJ & - +DUULV & $   $ FDVH VWXG\ RI WKH LQFODVV XVH RI D YLGHR JDPH IRU
WHDFKLQJ KLJK VFKRRO KLVWRU\ &RPSXWHUV (GXFDWLRQ    
:HUEDFK . +XQWHU ' 'L[RQ :   )RU WKH ZLQ +RZ JDPH WKLQNLQJ FDQ UHYROXWLRQL]H \RXU
EXVLQHVV 9RO   :KDUWRQ GLJLWDO SUHVV 3KLODGHOSKLD=DLQXGGLQ = 6KXMDKDW 0 +DUXQD + &KX 6 . :   7KH UROH RI JDPLILHG HTXL]]HV RQ
VWXGHQW OHDUQLQJ DQG HQJDJHPHQW $Q LQWHUDFWLYH JDPLILFDWLRQ VROXWLRQ IRU D IRUPDWLYH
DVVHVVPHQW V\VWHP &RPSXWHUV (GXFDWLRQ  
=KDQJ + )DQJ /   3URMHFWEDVHG OHDUQLQJ IRU VWDWLVWLFDO OLWHUDF\ $ JDPLILFDWLRQ DSSURDFK'LJLWDO 7XUQ LQ 6FKRROV²5HVHDUFK 3ROLF\ 3UDFWLFH 3URFHHGLQJV RI ,&(0  &RQIHUHQFH
=KDR & /L + -LDQJ = +XDQJ <   $OOHYLDWLQJ RQOLQH OHDUQHUV EXUQRXW D VWXG\ RQ WKHLQIOXHQFH RI WHDFKHUV HPRWLRQDO VXSSRUW &KLQD (GXFDWLRQDO 7HFKQRORJ\  =KDR * :DQJ 4 :X / 'RQJ <   ([SORULQJ WKH VWUXFWXUDO UHODWLRQVKLS EHWZHHQ XQLYHUVLW\VXSSRUW VWXGHQWV¶ WHFKQRVWUHVV DQG EXUQRXW LQ WHFKQRORJ\HQKDQFHG OHDUQLQJ 7KH$VLD3DFLILF (GXFDWLRQ 5HVHDUFKHU    

695

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Enhancing Learner Satisfaction in
Simulation-Based Learning: The Impact ofLearner Characteristics and Expectancy
Steven CK NGa*, Li Fern TANb , & Poh Nguk LAUc
aTemasek Polytechnic, SingaporebTemasek Polytechnic, Singapore
cTemasek Polytechnic, Singapore*steven_ck_ng@tp.edu.sg

Abstract: Digital game-based learning, facilitated by immersive virtual reality
technology, has become integral to modern education. This study investigates learner
satisfaction within the context of marketing simulations, exploring the roles of
Performance Expectancy (PE) and Effort Expectancy (EE), their interplay with learnercharacteristics, and their alignment with the Self-Determination Theory (SDT). The
study's theoretical framework draws from the Technology Acceptance Model (TAM),
Expectancy-Confirmation Theory, and SDT. Through factor analyses and hierarchicalregression, the research reveals the significance of Performance and Effort Expectancy
in influencing Learner Satisfaction. Notably, Performance Expectancy emerges as astronger driver than Effort Expectancy. The study also explores the impact of learnercharacteristics, such as gender and game-playing motivations rooted in SDT's
principles of intrinsic motivation, on Learner Satisfaction, identifying Striving as apotential deterrent to satisfaction. Despite limitations like sample specificity and selfreported data, this research provides valuable insights into learner engagement intechnology-mediated educational contexts, taking into account both cognitive andmotivational aspects. The findings underscore the importance of considering individual
motives and characteristics in optimizing educational technology interventions.
Keywords: simulation-based learning, game-based learning, learner satisfaction,
learner characteristics, game-playing motives, Self-Determination Theory (SDT),
Technology Acceptance Model (TAM), Expectancy-Confirmation Theory, gamified
learning environments, educational gamification, higher education

1. Introduction
Digital game-based learning, facilitated by immersive virtual reality technology (Merchant etal., 2014), is integrated into education for engaging and realistic experiences that promotecritical thinking (Byun & Joung, 2018; Tsai & Tsai, 2018). Simulations within serious games
offer contextualized cognition, skill acquisition, and attitudinal changes (Batko, 2016),
fostering interactive and social learning (Buzzard et al., 2011). In STEM fields, simulationbased learning fosters critical thinking, problem-solving, and decision-making skills (Cook etal., 2013; Hegland et al., 2017), while marketing simulations enhance strategic managementand decision-making (Hall, 2014; Caruana et al., 2016). Learner satisfaction and user
motives are pivotal in simulation effectiveness (Vos & Brennan, 2010; Kolb, 1984), driving
the need to explore factors influencing user satisfaction (Caruana et al., 2016).

1.1 Business and Marketing Simulation: Bridging Theory and PracticeMarketing simulations like "Hubro Marketing Simulation (HMS)" and "Markstrat" engage
students in decision-making processes (Hall, 2014; Caruana et al., 2016). These simulations
offer immediate feedback and objective data on market behaviors (Tompson & Dass, 2000),
enabling longitudinal analysis (Kietzmann & Pitt, 2016). Complexity challenges decisionmaking, with the acceptance and effectiveness of simulations impacting learning outcomes(Vos & Brennan, 2010). User satisfaction is vital, considering (dis)confirmation and specificaspects of the product (Giese & Cote, 2000).

696

1.2 User Satisfaction as an Outcome Variable
Building on Caruana et al. (2016), this study examines learner satisfaction (Caruana et al.,2016) and explores influencing variables such as gender and age (Caruana et al., 2016).
The study integrates the Expectancy-Confirmation Theory and the Unified Theory ofAcceptance and Use of Technology (UTAUT) (Caruana et al., 2016). Learner motives,
crucial in simulation effectiveness (Deci & Ryan, 1985), are explored alongside user
satisfaction, crucial for educational tool optimization (Bartle, 1996; Yee, 2006; Kahn et al.,2015).

1.3 Learner Game-Playing Motivations as a Crucial Part of Learner Characteristics
Learner motives have been identified as critical factors in the context of simulation games,
encompassing intrinsic motivation, extrinsic rewards, social interaction, skill development,
and personal interest (Deci & Ryan, 1985). Different motives, such as the application oftheoretical concepts, hands-on experience, and competition, significantly impact learnerengagement, effort, and overall satisfaction with the learning experience. The diversity ofstudent motivation, prior knowledge, and background has been found to have the greatest
influence on the effectiveness of teaching interventions (Hattie, 2015). Thus, understanding
learner motives is essential for identifying the most effective educational methods.

1.4 Aims of study and research questions
The main focus of this research is to determine the contribution of Performance Expectancy(PE) and Effort Expectancy (EE) on Learner Satisfaction (LS) when the potential impact of
Course Performance and Course Type are considered. On the basis of the above
discussion, we, therefore, hypothesize that:
1. Hypothesis 1: Performance Expectancy (PE) has a positive influence on Learner
Satisfaction with the simulation game.
2. Hypothesis 1a: The effect of Performance Expectancy on Learner Satisfaction ismoderated by Learner Characteristics like Gender, Past Course Performance,
Course Type, and Learner Game-Playing Motivation
3. Hypothesis 2: Effort Expectancy (EE) has a positive influence on Learner
Satisfaction with the simulation game.
4. Hypothesis 2a: The effect of Effort Expectancy on Learner Satisfaction is
influenced by Learner Characteristics like Gender, Past Course Performance,
Course Type, and Learner Game-Playing Motivation

2. Method
2.1 Participants and simulation deployment
Scholtz and Hughes (2018) emphasized pedagogical alignment, thus meticulously crafting
the deployment of the "Hubro Marketing Simulation (HMS)" to optimize connections between
Learner Satisfaction (LS), Performance Expectancy (PE), Effort Expectancy (EE), andLearner Motives. Integrated as a group project in a foundational marketing course during
October 2022 semester over 12 weeks, teams submitted comprehensive reports detailingmarketing choices made over eight rounds, incentivizing competition and sustained decisionmaking. Eleven tutors, spanning 28 classes, ensured intervention uniformity. Rigoroustraining aligned with Chaurasia (2017), tutor toolkits, and a practice round mitigated tutor
effects for student preparation. Out of 538 students, 227 participated in the survey, including
a 37-item questionnaire covering Learner Satisfaction, Performance Expectancy, Effort
Expectancy, Learner Motives, and demographic variables. Notably, 66% of respondents
were female, reflecting typical gender distribution in Singapore's business-related diploma
courses. Figure 1 illustrates the HMS interface and decision-making guidelines.
2697

Figure 1: HMS interface and how decisions are made. Tutors were briefed to aid
learners with Steps 1,2, and 3 to “connect the dots” in their group decision-making.

2.2 Variables and data analysis
Factor analyses explored the Learner Game-Playing Motivation construct (Jaskari & Henna
Syrjälä, 2022) and Learner Satisfaction, Performance Expectancy, and Effort Expectancy
constructs. A two-stage hierarchical multiple regression examined hypothesized
relationships with 227 respondents, surpassing recommended subject-to-item ratio(Nannally, 1978) and minimum requirement for hierarchical regression (Soper, 2016).
2.2.1

Game-playing motivation construct

Employing established steps (Luomala et al., 2017; Vahlo et al., 2017; Jaskari & Syrjälä,
2022), factor analysis assessed game-playing motivations. Three fixed factors –,PPHUVLYLW\&RPSOHWLQJ Į  6RFLDOL]LQJ Į  6WULYLQJ Į  – explained
78.1% variance (Table 1).
Table 1. Mean and Standard Deviation of Items Together With Pattern Matrix From Factor
Analysis for Game-playing Motivation Construct
Items
When thinking about your own learning, howimportant do you find the following items?

Mean

Immersivity Completing
1. To advance well in the subject
2. To be immersed in learning
3. To feel that you can control your learning
4. To feel the joy of research in learning
5. To figure out the idea of marketing and thelogic of the subject
6. To learn the stories and principles related tothe topics of the subject
7. To notice that I have progressed in learning
8. To set goals and achieve them

45.74 10.24
6.00
1.20
5.67
1.24
5.79
1.23
5.59
1.4
5.64
1.31

0.74
0.73
0.78
0.79
0.85

5.54

1.31

0.83

5.82
5.69

1.27
1.28

0.87
0.69

3

698

SD

Loadings

Cronbach’s
Alpha

0.95

Socializing
9. To chat with other students
10. To compete with other students
11. To feel that you belong to the community ofsubject participants
12. To feel that you belong to the group

20.56
5.12
4.84
5.16

6.04
1.44
1.67
1.50

0.87
0.63
0.72
0.77

5.44

1.43

0.68

10.25 3.11
0.91
Striving
5.28
1.46
0.83
13. To be one of the most skilled students
14. To be the best student in the subject
4.97
1.65
0.83
Note. Extraction method: Principal components analysis. Rotation method: Varimax. Loading
coefficients from the factor analysis below .5 are not shown.
2.2.2

Learner Satisfaction, Performance Expectancy, and Effort Expectancy Constructs

Assessed through 7-point Likert scales (1–7), three constructs – LS, PE, EE – underwent
factor analysis, aligned with Caruana et al. (2016). The .02  DQG%DUWOHWW VWHVW Ȥ 5812, p < .001) supported validity. Principal components factor analysis with Oblimin rotation
indicated reliability via Cronbach's alpha (.97), surpassing .7 (Nunnally, 1978). (Table 2).
Table 2. Mean and Standard Deviation of Items Together With Pattern Matrix From Factor
Analysis for Learner Satisfaction, Performance Expectancy, and Effort Expectancy
constructs
Cronbach’s
Items
Mean
SD
Loadings
Alpha
Learner Satisfaction (LS)92.72 26.86
0.97
The use of the simulation game in the subject was 5.16
1.37
0.85
one of the best experiences I could have had.
I am satisfied with having used the simulation
5.31
1.44
0.80
game in the subject.
The choice of the simulation game for the subject
5.28
1.52
0.89
was a wise one.
I am certain it was the right thing that the
5.11
1.42
0.52
simulation game was included as an integral partof the subject.
Playing the simulation game engaged me more in
5.18
1.58
0.79
the course than a regular teaching method does.
I will recommend the use of simulation game to
5.22
1.67
0.85
learn and apply marketing principles and theories.
Performance Expectancy (PE)Using the simulation game enabled me toaccomplish tasks in the subject more quickly.
Using the simulation game improved my study
performance in this subject.
Using the simulation game improved myunderstanding of marketing.
Using the simulation game enhanced myeffectiveness in my study of marketing.
Using the simulation game made it easier todevelop marketing competencies.
I found the simulation game useful for addressing
marketing-related issues.
4

699

30.73
5.06

8.93
1.52

0.61

0.97

4.82

1.56

0.72

5.30

1.48

0.84

5.08

1.52

0.92

5.21

1.41

0.93

5.26

1.44

0.94

Effort Expectancy (EE)29.75 9.28
0.97
Learning to use the simulation game was easy for 4.89
1.61
0.98
me.
It was easy to get the simulation game to do what I 4.94
1.56
0.88
wanted it to do.
Interfacing with the simulation game was clear and 4.99
1.55
0.78
understandable.
I found the simulation game flexible to interact
5.07
1.44
0.74
with.
It was easy for me to become skillful at using the
4.82
1.57
0.77
simulation game.
I found the simulation game easy to use.
5.04
1.55
0.86
Note. Extraction method: Principal components analysis. Rotation method: Varimax. Loading
coefficients from the factor analysis below .5 are not shown.

2.3 Relationship between Variables
Following Caruana et al. (2016), a two-stage hierarchical regression method was applied. In
Stage 2, two independent variables were added beyond Stage 1 variables, ensuring
normality and linearity assumptions. Harman's single-factor test revealed no significant
common method bias (Podsakoff et al., 2003).
Stage 1 of the hierarchical regression involved Gender, Course Type, Past GPA
Attainment, and game-playing motivations (Immersity Completing, Socializing, Striving) as
independent variables predicting Learner Satisfaction. The results (Table 3) explained 41%
YDULDQFH6LJQLILFDQWSUHGLFWRUVZHUH*HQGHU ȕ S ,PPHQVLW\&RPSOHWLQJ ȕ 
0.53, p <  6RFLDOL]LQJ ȕ S DQG6WULYLQJ ȕ -0.27, p < .001). (Table 3)
Table 3. Results From the Two-Stage Hierarchical Multiple Regression (N = 227).
Stage 1
Stage 2
Variable
GenderCourse Type
Past GPA Attainment
Immersivity Completing
Socializing
Striving
Performance Expectancy
Effort Expectancy
R
R2
Adjusted R2
Change R2*p<.05 **p<.01 ***p<.001

6WGȕ
0.24
0.01
0.15
0.53
0.30
-0.27

t
2.16*
0.28
1.48
6.76***
3.36***-3.67***

sr2
0.01
0.00
0.00
0.12
0.03
0.04

0.66
0.43
0.41
0.41

6WGȕ
0.04
0.01
0.02
0.05
0.05
-0.09
0.52
0.41
0.92
0.85
0.84
0.43

t
0.644
0.242
0.82
1.12
1.09
-2.41*
10.67***
8.98***

sr2
0.00
0.00
0.00
0.00
0.00
0.00
0.08
0.06

Aligned with Caruana et al. (2016), a robust Performance Expectancy-Effort Expectancy
correlation drove regression score computation via exploratory factor analysis. The model
YDULDQFHH[SODLQHG)   S3HUformance Expectancy, Effort
Expectancy, and Striving contributed 14% to Learner Satisfaction variance post adjustment
for Gender, Past GPA Attainment, and Course Type. In the final model, Performance([SHFWDQF\ ȕ S VXUSDVVHG(IIRUW([SHFWDQF\ ȕ S DQG
6WULYLQJ ȕ -0.09, p < .05) was also notable (Table 3).
5700

3. Results and Discussion
The analysis of research hypotheses and their corresponding outcomes, as presented inTable 4, furnishes a comprehensive comprehension of the intricate interplay between
Performance and Effort Expectancy and their influence on Learner Satisfaction. The
meticulous assessment revealed a substantial explanatory capacity of 84.4% concerning
Learner Satisfaction. Importantly, both Performance and Effort Expectancy emerged as
significant drivers, aligning harmoniously with established research within the Technology
Acceptance Model (TAM) framework (Caruana et al., 2016; Davis, 1989; Shen & Eder, 2009;
Wu & Gao, 2011). Noteworthy is the prominence of Performance Expectancy, surpassing
Effort Expectancy, thereby accentuating its pivotal role.
An intriguing facet of the findings pertains to the partial validation of Striving as agame-playing motivation and its impact on the model's outcomes, diverging from Jaskari and
Syrjälä's (2022) findings. Methodological distinctions are likely contributors; our study
examined individual motives, whereas earlier research concentrated on aggregated clusters.
The counterintuitive effect of Striving may arise from learners perceiving a curtailed sense ofcontrol, particularly in competitive scenarios where their academic performance is at stake,an aspect particularly pertinent in the Singaporean context where grades hold paramountimportance for university admission. In response, our team aimed to experiment with gameelements like awarding "Most Valuable Player" among high achievers within the teams to
mitigate the potential negative impact of Striving.
Additionally, Gender emerged as a determinant, notably evident in Stage 2, wherefemale learners exhibited relatively lower satisfaction compared to males, aligning withtechnology adoption and simulation-based educational games literature (Caruana et al.,2016; Venkatesh et al., 2003; Fenwick & Neal, 2001), underscoring the importance ofaddressing gender-based disparities in educational technology interventions.
To summarize, the systematic exploration of Performance and Effort Expectancy,
coupled with the nuanced influence of game-playing motivation and gender, imparts valuable
insights into the multifaceted dynamics underpinning Learner Satisfaction. These findings
contribute to a broader comprehension of learner engagement within technology-mediatededucational contexts, thereby offering implications for instructional design and suggesting
avenues for further research in the field.
Table 4. Summary of Hypotheses and Conclusions on Performance and Effort Expectancy
Effects on Learner Satisfaction
Hypothesis Conclusion
1
Accepted: Strongly supported. Results from a two-stage hierarchicalregression confirm a positive correlation between Performance Expectancy(PE) and Learner Satisfaction, in alignment with established models (TAM)(Caruana et al., 2016; Davis, 1989).
1a
Accept with Consideration: The impact of Performance Expectancy on
Learner Satisfaction is affirmed, while the significance of Gender becomesapparent. Female participants exhibit lower satisfaction levels, indicating amoderated effect of Performance Expectancy by Gender. Further
investigation into game-playing motivations and course-related attributes is
crucial for a nuanced comprehension.
2
Accepted: Strongly supported. Substantial evidence confirms that Effort
Expectancy (EE) is a significant predictor of Learner Satisfaction, aligningwith established theoretical frameworks (TAM).
2aAccept with Reservation: Partial support The influence of Effort
Expectancy on Learner Satisfaction is validated; however, the interplay
between Learner Characteristics such as Gender, Past Course
Performance, Course Type, and Learner Game-Playing Motivation warrants
deeper exploration.
6701

4. Limitations and Directions for Further ResearchThis study responds to the call for deeper cross-cultural investigations, as advocated by
Caruana et al. (2016) and Jaskari & Henna Syrjälä (2022), but is confined to a specificpolytechnic business school in Singapore, limiting generalizability across diverse settings. To
enhance applicability, future research should explore various educational contexts.
The reliance on self-reported data introduces potential bias and subjective
interpretations. Incorporating objective measures or observational data would enhance result
reliability. The cross-sectional design precludes causal inferences and temporal insights;
adopting longitudinal or experimental approaches could address these limitations.
While this study focuses on Performance and Effort Expectancy, other variables like
self-efficacy and engagement contribute to Learner Satisfaction. Future studies should
consider broader variable inclusion to refine understanding. In conclusion, while shedding
light on Performance and Effort Expectancy's impact, this study acknowledges contextual
boundaries, self-report reliance, and the need for variable expansion. Overcoming theselimitations would refine comprehension of technology-mediated education engagement.

Acknowledgements
We thank Mr Matthew Chua and the team of Marketing Fundamentals at Temasek
Polytechnic, School of Business, for their assistance in launching the simulation unit.

References
Batko, M. (2016). Business management simulations: A detailed industry analysis as well asrecommendations for the future. International Journal of Serious Games, 3(2), 47-65.
Bartle, R. (1996). Hearts, clubs, diamonds, spades: Players who suit MUDs. Journal of MUD
Research, 1(1), 19.
Buzzard, C., Crittenden, V., Crittenden, W., & McCarty, P. (2011). Motivating student participation in
virtual worlds: An examination of 3D technology acceptance in Second Life. Journal of Interactive
Marketing, 25(3), 143-157.
Byun, J., & Joung, E. (2018). Digital game-based learning for K–12 mathematics education: A metaanalysis. School Science and Mathematics, 118(3-4), 113-126.
Caruana, A., La Rocca, A., & Snehota, I. (2016). Learner satisfaction in marketing simulation games:
Antecedents and influencers. Journal of Marketing Education, 38(2), 107-118.
Chaurasia, V. (2017). Immersive virtual reality (VR) in pedagogical practices: Exploring learners’
acceptance and emotional responses. Education and Information Technologies, 22(4), 14111427.
Cook, D. A., Brydges, R., Zendejas, B., Hamstra, S. J., & Hatala, R. (2013). Technology-enhanced
simulation to assess health professionals: A systematic review of validity evidence, researchmethods, and reporting quality. Academic Medicine, 88(6), 872–883.
Davis, F. D. (1993). User acceptance of information technology: System characteristics, user
perceptions and behavioral impacts. International Journal of Man-Machine Studies, 38, 475-487.
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior.
Springer.
Fenwick, T. J., & Neal, D. M. (2001). Learning from experience: Minority access to simulation-based
education. The Journal of Negro Education, 70(1/2), 90-101.
Giese, J. L., & Cote, J. A. (2000). Defining consumer satisfaction. Academy of Marketing Science
Review, 2000(1).
Hall, J.J. (2014). Quality assurance in business simulation design. In Developments in Business
Simulation and Experiential Learning: Proceedings of the Annual ABSEL conference (Vol. 41).
Hattie, J. (2015). The applicability of visible learning to higher education. Scholarship of Teaching and
Learning in Psychology, 1(1), 79-91.
Hegland, P. A., Aarlie, H., Strømme, H., & Jamtvedt, G. (2017). Simulation-based training for nurses:
Systematic review and meta-analysis. Nurse Education Today, 54(1), 6–20.
Jaskari, Minna-Maarit & Syrjälä (Jyrinki), Henna. (2022). A Mixed-Methods Study of Marketing
Students’ Game-Playing Motivations and Gamification Elements. Journal of Marketing Education.
45. 027347532210832. 10.1177/02734753221083220.
7702

Kahn, A. S., Shen, C., Lu, L., Ratan, R. A., Coary, S., Hou, J., Meng, J., Osborn, J., & Williams, D.
(2015). The Trojan player typology: A cross-genre, cross-cultural, behaviorally validated scale ofvideo game play motivations. Computers in Human Behavior, 49, 354–361.
Kietzmann, J. H., & Pitt, L. F. (2016). Transformative consumer research and gamification inmarketing education. Journal of Marketing Education, 38(3), 166-179
Kolb, D. A. (1984). Experiential learning: Experience as a source of learning and development.
Englewood Cliffs, NJ: Prentice Hall.
Larréché, J-C. (1987). On simulations in business education and research. Journal of BusinessResearch, 15, 559-571.
Luomala, H. T., Sihvonen, J., Syrjälä, H., Mäkilä, T., Könnölä, K., Liukkonen, T., Lundén, S., &Sandell, M. (2017). Linking digital game-playing motivations to food consumption. In Proceedingsof the 1st International GamiFIN Conference, CEUR-WS, Vol 1857 (pp. 111–119).
Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., & Davis, T. J. (2014). Effectivenessof virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A
meta-analysis. Computers & Education, 70, 29-40.
Nunnally, J. C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases inbehavioral research: A critical review of the literature and recommended remedies. Journal ofApplied Psychology, 88(5), 879-903.
Scholtz, J., & Hughes, C. (2018). Considering characteristics of well-designed game elements. In
Proceedings of the European Conference on Games Based Learning (Vol. 1, p. 575). Academic
Conferences International Limited.
Shen, J., & Eder, L. (2009). Intentions to use virtual words in education. Journal of Information
Systems Education, 20, 225-233.
Soper, D. S. (2016). A-priori sample size calculator for hierarchical multiple regression [Software].
Retrieved from http://www.danielsoper.com/statcalc
Tompson, G. H., & Dass, M. (2000). Student attitudes toward a marketing simulation: A longitudinalassessment. Journal of Marketing Education, 22(1), 28-35.
Tsai, C. W., & Tsai, M. J. (2018). Using augmented reality technology to improve students' English
vocabulary learning and retention. Educational Technology & Society, 21(3), 153-165.
Vahlo, J., Luomala, H. T., Karinkanta, S., & Leppänen, V. (2017). Human factors in big data: A
hospital emergency department case study. International Journal of Human Factors and
Ergonomics, 5(2), 182-196.
Venkatesh, V., Morris, M. G., Davis, F. D., & Davis, G. B. (2003). User acceptance of information
technology: Toward a unified view. MIS Quarterly, 27, 425-478.
Vos, L., & Brennan, R. (2010). Marketing simulation games: Student and lecturer perspectives.
Marketing Intelligence & Planning, 28, 882-897.
Wu, X., & Gao, Y. (2011). Applying the extended technology acceptance model to the use of clickers
in student learning: Some evidence from macroeconomics classes. American Journal ofBusiness Education, 4(7), 43-50.
Yee, N. (2006). Motivations for play in online games. CyberPsychology & Behavior, 9(6), 772-775.

8703

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring the Impact on Student Reading
Preferences in Gamified Reading Portfolio
Hsiao-Tung YANG1*, Chang-Yen LIAO2, Ciao-Min SYU1 & Tak-Wai CHAN1
Department of Network Learning Technology, National Central University, Taiwan
2
Department of Hakka Language and Social Sciences, National Central University, Taiwan*jessica4625@gmail.com
1

Abstract: Extensive reading enhances horizons, deepens understanding, and nurtures
critical thinking. This study aims to gamify reading goals, turning teacher-recommended
books into engaging game levels. Books are categorized and distributed by genre anddepth. And we will design achievement medals, allowing students to earn medals upon
meeting specific criteria. The goal is to boost reading interest, fostering a passion forlearning through accomplishments. This research addresses selective reading,encouraging exploration of varied genres while nurturing a genuine love for reading.
Keywords: Gamification, Achievement Goals, Reading Interest, Extensive Reading

1. Introduction
Reading initiatives like the "morning reading program" and "reading courses" are wellestablished in Taiwan. Since 2009, our team has researched Modeled Sustained Silent
Reading (MSSR) and introduced the "morning reading campaign" to schools. Promoting
MSSR actively nurtures a reading habit and enjoyment. However, as students mature, achallenge arises: they develop "reading preferences", sticking to specific genres and avoiding
diverse books. Some also resist advancing to complex levels, staying in their "comfort zone"
and limiting knowledge expansion through reading.
Therefore, the "Gamified Reading Performance System" designed by this instituteincorporates a list of books recommended by several language teachers for grades 1 to 6 into
100 levels in order to help students read different types and depths of books by completing
different levels, so that students can choose the level that suits their reading according to theirown interests or abilities, and then record the books using the system after finishing thereading. After reading the books, use the system to record the books. After reading the books
and completing the levels, students can receive corresponding rewards to keep themmotivated to continue the challenge.
This study investigates how the system can help teachers and students in schools to
make reading activities more attractive and interesting. Through the design of this system,
reading is no longer just "reading a book", but can become an activity that can interact withothers and express oneself, thus increasing the interest and motivation of reading, which is
the goal of this study.

2. Literature Review
2.1 Gamification
Gamification involves integrating game elements into non-game contexts to enhance
engagement and motivation. By employing techniques like points, badges, and leaderboards,
it aims to make tasks more enjoyable. (Deterding et al., 2011). In education, gamification
keeps learners immersed and time-conscious, impacting their flow experience (Mak, Wang &Chu, 2019). Effective design involves tasks aligned with learners' interests and autonomousgoal-setting (Deci & Ryan, 2000).

704

2.2 Goal Setting
The theory of goal setting, originating from Locke and Latham's seminal work (2002),
underscores the potency of establishing distinct and demanding objectives to amplify
motivation, concentration, and competence. The lucidity and challenge of goals significantly
influence their accomplishment. Well-defined targets contribute to positive sentiments,
contentment, and self-efficacy. A clear self-assessment process further bolsters self-efficacy.
Proficient goal setting, characterized by quantifiable milestones and time-sensitive
benchmarks, not only bolsters productivity but also fosters personal development and triumph
across diverse facets of life (Locke et al., 1981).

3. System Design
Our research team introduced a reading system to schools in Taiwan in tandem with reading
programs. This system enables students to record their reading progress, motivations for bookchoices, and recommendations. Notably, many students display reading habits but lack longterm goals. Thus, we aim to employ Goal Setting and Gamified Learning theories, integratingan "Achievement Medal Mechanism" and "Data Visualization." This aids students in pursuing
personalized reading interests, guided by level coordination. A "Reading Dashboard" (Figure1) monitors indicators like level progress and book categories. The main display features 100
levels, each containing varying books. Passing a level requires reading a set number of books,
marked by changing button color and "rockets" earned, encouraging challenge and sustained
reading motivation.

Figure 1. Reading Dashboard

Figure 2. Level content page

Upon entering a level (Figure 2, you can access basic information about the books,
including titles, categories, and depth. Three book categories (story fiction, humanities andsociety, science and technology) with four sub-categories each (e.g., warm and inspirational,
adventure and reasoning) are featured. Book depth is rated by stars: 1 star for picture books,
2 stars for bridge books, 3 stars for beginner texts, 4 stars for intermediate texts, and 5 stars
for advanced texts. Each level's top displays the total "stars" earned by "read books." A
checkmark symbolizes completed books, while an unread lock indicates locked books. By
equitably distributing various book types and depths, students are exposed to a broader
reading spectrum, fostering a balanced reading habit and preventing preferences.
The achievement medal collection (Figure 3 presents medals based on book types anddepths, categorized as bronze, silver, and gold. These medals can be sorted by type, and aclick on "Your Achievements" reveals all earned medals. By earning these, students enhanceself-confidence, fostering diverse reading preferences and expanding reading horizons.
Finally, Figure 4 resents the "Reading Leaderboard" showcasing top students in eachindex, encouraging competition and improvement. "My Reading Performance" displays
weekly detailed reading status, aiding students in self-review, comparison, and refining
reading plans.

705

Figure 3 Achievement Badge List

Figure 4 weekly record section

4. ResultThis study involves 30 students from a primary school in Taiwan, specifically the second andthird grades. They engage in daily Modeled Sustained Silent Reading (MSSR) activities during
an eight-week experiment. The system was developed in alignment with the school's reading
program after understanding the requirements from teachers and students. Using the DesignBased Research (DBR) approach, bi-weekly interviews were conducted after implementing
the system to collect feedback and make gradual adjustments. Log files were also gathered
for user behavior analysis.
Two interview rounds focused on students' system usage, feedback, and its impact onreading behavior. Initial findings indicate that about 40% of participants explore unfamiliar
book genres through the level arrangement, while around 30% intend to read more diverse
books to complete additional levels. These preliminary interviews highlighted persisting"reading preferences" among students. Therefore, we plan to enhance the system further. For
example, we aim to introduce personalized prompts on the dashboard, offering individualized
adjustments and suggestions based on reading progress. Future research will include
alternative methods to assess comprehension. Beyond setting reading quantity goals, we
contemplate integrating activities like recording impressive content, sharing thoughts throughvideos after completing a book, and recommending good books to peers. These could serve
as level completion criteria or achievement objectives, enhancing reading motivation. The
incorporation of diverse activities to amplify enjoyment and challenges in reading could provide
insights into their motivational impact on students.

Acknowledgements
This study was funded by the National Science and Technology Council of Taiwan (NSTC112-2410-H-008-020-MY3) and by Research Center for Science and Technology for Learning,
National Central University, Taiwan.

References
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements togamefulness: Defining "gamification". In Proceedings of the 15th International Academic
MindTrek Conference: Envisioning Future Media Environments (pp. 9-15).
Locke, E. A., Shaw, K. N., Saari, L. M., & Latham, G. P. (1981). Goal setting and task
performance: 1969–1980. Psychological Bulletin, 90(1), 125–152.
Locke, E. A., & Latham, G. P. (2002). Building a practically useful theory of goal setting andtask motivation: A 35-year odyssey. American Psychologist, 57(9), 705–717.
Mak, M.T.F., Wang, M., & Chu, S.K.W. (2019). Effects of a Gamified Learning Platform on
Elementary School Students’ Flow Experiences in Leisure Reading. Paper presented at
the ASIS&T 2019 Annual Meeting, Melbourne, Australia.
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic
motivation, social development, and well-being. American psychologist, 55(1), 68.

706

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Improving Teamwork through a
Decision-Theoretic Coach in a Minecraft
Search-and-Rescue GameDavid V. PYNADATHa*, Nikolos GURNEYa, Sarah KENNYa, Rajay KUMARa, Stacy C.
MARSELLAb, Haley MATUSZAKb, Hala MOSTAFAc, Pedro SEQUEIRAd, Vokan USTUNa& Peggy WUeaInstitute for Creative Technologies, University of Southern California, USAb
Northeastern University, USAc
Bitsight Technologies, USAd
SRI International, USAe
Raytheon Technologies Research Center , USA*pynadath@usc.edu
Game-based learning provides a promising methodology for teaching skills such as
teamwork (Martín-Hernández et al., 2021; Riivari et al., 2021; Syynimaa et al., 2022).
Studies of human teamwork have identified various team processes that underlie good jointtask performance (e.g., Marks et al., 2001). The multiagent research community has
operationalized many of these findings into domain-independent algorithms that separatetask-specific knowledge from reusable task-independent teamwork knowledge (e.g., Tambe,
1997). These frameworks were successfully deployed in facilitating teamwork in real-world
human teams when supplied with task-specific knowledge (e.g., Chalupsky et al., 2001).
We present here an agent that focuses exclusively on teamwork-level variables in
deciding what interventions to use in coaching a human team in a game-based
search-and-rescue environment in Minecraft (Corral et al., 2021). The game is an urban
search and rescue (USAR) task that involves clearing and avoiding hazards while rescuing
victims of a disaster. The task is performed by distributed teams of three participants,
possibly with a human or agent as a coach. The coaches are not acting team members, butthey are able to communicate, via text chat, with the human team members, who themselves
can directly engage each other in high-volume, unconstrained voice communication.
Our agent does not directly observe the game environment or the people in it, butinstead relies on input from analytic components (ACs) (developed by other research teams)
that process environmental information and output only teamwork-relevant measures. Our
agent models teamwork variables and updates its beliefs over them using PsychSim, which
provides reusable AI technology for agents capable of populating game environments(Pynadath & Marsella, 2005). PsychSim integrates two AI technologies: recursive models(Gmytrasiewicz & Durfee, 1995) and decision-theoretic reasoning in the form of POMDPs(Kaelbling et al., 1998). PsychSim has been used to build interactive training games for avariety of soft skills, such as negotiation (Kim et al., 2009) and avoiding risky behavior undersocial pressure (Miller et al., 2011). For the current search-and-rescue task, we built anagent that assesses the team it is currently observing and chooses interventions to best
assist them. Details on our agent and some objective measures of its purposes are available
in an earlier publication (Pynadath et al., 2023). In this paper, we look deeper into somespecific observations of human behavior when interacting with our agent in the game.
Our agent used team-process variables (Marks et al., 2001) whose measures wererecently validated (Mathieu et al., 2020), but only those relevant to the game environment:● Affect management: "activities that foster emotional balance, togetherness, andeffective coping with stressful demands and frustration."

707

●
●

Coordination: "the process of synchronizing or aligning the members’ actions."
Motivating and confidence building: "activities that develop and maintainmembers’ motivation and confidence while working toward team goals."● Systems monitoring: "activities such as tracking team resources (e.g., money) andfactors in the team environment (e.g., inventories) to ensure that the team has what itneeds to accomplish its goals and objectives."● Team monitoring: "members assisting others in the performance of their tasks (byproviding feedback or coaching or assisting with the task itself)."
None of these processes are directly observable. We thus model them as hidden variables
that our agent seeks to improve through the following candidate interventions:● Reflection: Between trials, the agent prompts the team with a reminder about asituation in the first trial where a player was stuck on a threat plate for an inordinate
amount of time.
● Cheerleading: The agent congratulates a player on successfully achieving a goal
(more specifically, moving a victim to a triage area).
● Report drop: The agent reports on a noteworthy lack of performance by one player(more specifically, failure to respond to outstanding requests by a teammate).
● Notify phase (early): The agent reminds the team that it is still early in the game, so
exploration should be valued more.
● Notify phase (late): The agent reminds the team that it is getting late in the game,so exploration should be valued less.
● Remind of best practices: The agent suggests that someone help a player who has
a number of outstanding, but unaddressed, requests.
● Prompt activity: The agent asks about any possible issues upon observing that theteam has been predominantly idle for a period of time.
Data for thirteen teams were collected at Arizona State University. One immediate
question was whether the teams even noticed our agent’s interventions. For one team, fiveof eight interventions were verbally acknowledged; for two others, none were. For all otherteams, one or two interventions were explicitly acknowledged. Breaking down these counts
by intervention type, none of the cheerleading interventions were acknowledged. The two
notify phase interventions were the most commonly acknowledged (14 of 38), while thevarious request/activity based interventions were less frequently acknowledged (8 of 38).
The post-trial survey asked a few questions about the participants' opinions of theircoach. Not surprisingly, the human coaches received much higher trust scores than theagent coaches. All of the agent coaches were distrusted, but ours was distrusted the least.
Beyond these aggregate results, we found it necessary to examine the actual
behaviors of individual teams, in addition to listening to the dialogue within those teams. The
relatively small number of teams makes this in-depth examination feasible, although it also
potentially limits the generalizability of the observations made.
One recurring issue was a disconnect between what our agent perceived as requests
vs. what the team members actually intended as requests. Participants could mark blocks in
Minecraft (e.g., for victims, for obstructed rooms), making an implicit request (e.g., toevacuate/triage the victim inside, to clear the obstruction). However, marking these blocks
was sometimes used as implicit communication or even just a mnemonic device, not arequest. This difference of perspective led to occasional confusion when our agent
intervened to stimulate satisfaction of “requests”. For example, in one trial, our agent
performed a remind of best practices intervention regarding Red's requests, but Red thenexpressed no awareness of any requests when asked by another team member to clarify.
As a more positive example, there was another team where our agent also choseremind of best practices for Red and where a team member also asked for clarification.
However, in this case, Red responded by explicitly identifying a victim, its type, and its
location. The other team member then saw this victim and triaged it. Our agent's intervention
thus triggered the desired shared situational awareness and subsequent task completion.
A more ambiguous example starts off just as these first two, with a team member
asking for clarification of Red's requests after an agent intervention. In this case, Red admits708

to accidentally forgetting to pick up a marker. Despite our agent’s mistaken reaction to anonexistent request, the communication prompted by its intervention preempted themisunderstanding that would have occurred if the discussion had not taken place.
As another example of the evaluation challenge, our agent performed the late versionof its “notify phase” interventions while the team was in the middle of discussing its strategy.
There was a pause while the speaker considered the agent's message, but then the teamresumed discussing and executing its current plan, which was in conflict with our agent's
recommendation. The team was clearly not complying, but it is not clear whether they were
better or worse off as a result. Thus, while the lack of compliance is objectively measurable,
any wider-scope learning outcomes are harder to evaluate.

Acknowledgements
This material is based upon work supported by the Defense Advanced Research Projects
Agency (DARPA) under Contract No. W911NF-20-1-0011 and by the U.S. Army ResearchLaboratory (ARL) under contract number W911NF-14-D-0005. Any opinions, findings, orrecommendations do not necessarily reflect the views of DARPA or the US Government.

References
Chalupsky, H., Gil, Y., Knoblock, C. A., Lerman, K., Oh, J., Pynadath, D. V., Russ, T. A., Tambe, M., etal. (2001). Electric elves: Applying agent technology to support human organizations.
Proceedings of the Innovative Applications of Artificial Intelligence Conference, 1, 51–58.
Corral, C. C., Tatapudi, K. S., Buchanan, V., Huang, L., & Cooke, N. J. (2021). Building a synthetic
task environment to support artificial social intelligence research. Proceedings of the Human
Factors and Ergonomics Society Annual Meeting, 65(1), 660–664.
Gmytrasiewicz, P. J., & Durfee, E. H. (1995). A rigorous, operational formalization of recursivemodeling. Proceedings of the International Conference on Multiagent Systems, 125–132.
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable
stochastic domains. Artificial intelligence, 101(1-2), 99–134.
Kim, J. M., Hill Jr, R. W., Durlach, P. J., Lane, H. C., Forbell, E., Core, M., Marsella, S., Pynadath, D.V.
& Hart, J. (2009). BiLAT: A game-based environment for practicing negotiation in a culturalcontext. International Journal of Artificial Intelligence in Education, 19(3), 289-308.
Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy ofteam processes. Academy of Management Review, 26(3), 356–376.
Martín-Hernández, P., Gil-Lacruz, M., Gil-Lacruz, A. I., Azkue-Beteta, J. L., Lira, E. M., & Cantarero, L.
(2021). Fostering University Students’ Engagement in Teamwork and Innovation Behaviors
through Game-Based Learning (GBL). Sustainability, 13(24), 13573.
Mathieu, J. E., Luciano, M. M., D’Innocenzo, L., Klock, E. A., & LePine, J. A. (2020). The development
and construct validity of a team processes survey measure. Organizational Research Methods,
23(3), 399–431.
Miller, L. C., Marsella, S., Dey, T., Appleby, P. R., Christensen, J. L., Klatt, J., & Read, S. J. (2011).
Socially optimized learning in virtual environments (SOLVE). In Interactive Storytelling: FourthInternational Conference on Interactive Digital Storytelling, 4, 182-192.
Pynadath, D. V., Gurney, N., Kenny, S., Kumar, R., Marsella, S. C., Mostafa, H., Sequeira, P., Ustun,V., and Wu, P. (2023). “Effectiveness of teamwork-level interventions through decision-theoretic
reasoning in a Minecraft search-and-rescue task”. In Proceedings of the International
Conference on Autonomous Agents and Multiagent Systems.
Pynadath, D. V., & Marsella, S. C. (2004). Fitting and compilation of multiagent models throughpiecewise linear functions. Proceedings of the International Joint Conference on Autonomous
Agents and Multiagent Systems, 1197–1204.
Syynimaa, K., Lainema, K., & Lainema, T. (2022). Identifying Challenges in Virtual Teams: A Case
Study of Teamwork in a Game-Based Learning Environment. International Association for
Development of the Information Society.
Riivari, E., Kivijärvi, M., & Lämsä, A. M. (2021). Learning teamwork through a computer game: for thesake of performance or collaborative learning? Educational Technology Research andDevelopment, 69, 1753-1771.
Tambe, M. (1997). Towards flexible teamwork. Journal of Artificial Intelligence Research, 7, 83–124.

709

TELL

710

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Using Flipped Classroom Approach toIntegrate SRL Instruction in ClassicalChinese Reading Instruction: Insights fromthe First-Year Results
Kit Ling LAUa
The Chinese University of Hong Kong, Hong Kong
*dinkylau@cuhk.edu.hkaAbstract: This study adopted a flipped classroom (FC) approach to combine theinstructional principles of self-regulated learning (SRL) and eLearning activities in a
two-year intervention program to facilitate Hong Kong students’ learning of classicalChinese (CC). Using a quasi-experimental design, students from three Hong Kong
secondary schools were assigned to three different treatment groups: SRL instruction
plus out-of-class eLearning, SRL instruction only, and control. Pre- and post-treatment
measures of CC reading comprehension tests and student questionnaires were used
to evaluate the effectiveness of the program on enhancing students’ strategy use,
motivation, and performance in CC reading. This paper reports the findings from thequantitative data collected in the first-year of the study. The findings indicated that while
all treatment groups showed significant improvements in the CC reading post-test, only
students in the two experimental groups significantly increased their use of CC reading
strategies, self-efficacy, and intrinsic motivation. The findings offer promisingpreliminary evidence supporting the viability and advantages of integrating SRL
instruction and FC as an effective instructional approach to enhance student learning
within a conventional academic domain.
Keywords: Classical Chinese reading, eLearning, flipped classroom, motivation, selfregulated learning instruction

1. Introduction
Self-regulated learning (SRL), involving metacognition, motivation, and strategic actions, hasbeen widely advocated as an effective form of learning (Winne & Perry, 2000; Zimmerman,
2000). Schunk and Zimmerman (1997) proposed four developmental levels through whichlearners progress from other-regulation to self-regulation. At the observational level, learners
observe the key aspects of strategies by watching a proficient model. Moving to theemulation level, learners attempt to apply the observed strategies by imitating the patterns
while being supervised by the model. In the self-controlled level, learners gain the ability to
use the strategies independently. Finally, in the self-regulation level, learners cansystematically regulate their behaviors and performance in response to varying conditions.
Based on previous SRL research, the author proposed a "TSAE" framework (Lau, 2013; 2020)
to guide teachers in fostering students' progression towards higher levels of SRL. The
framework consists of four major instructional principles: Task nature (T), where teachers
provide direct instruction on subject knowledge and learning strategies while designing open
tasks to facilitate knowledge transfer. Teacher support (S), where various scaffolding techniques
are employed to assist students in executing self-regulatory skills. Student autonomy (A), wherestudent-directed activities are designed to enable practice of self-regulatory skills. Evaluation
practices (E), which involve students in self- and peer-evaluation to enhance metacognitive
knowledge and skills (Cousins et al., 2022; Dignath et al., 2008; Dignath & Veenman, 2021;
Jayawardena et al., 2019; Lombaerts et al., 2009; Mohammadi et al., 2020; Neitzel &Connor, 2017; Perry & Rahim, 2011).

711

Using the TSAE framework, the author developed an intervention program to improve
Hong Kong students' classical Chinese (CC) reading, traditionally dominated by teachercentered instruction (Lau, 2020). While the findings of that study indicated that SRL
instruction was more effective than the traditional teacher-centered instruction in enhancing
students’ prior CC knowledge and comprehension, no significant changes in their strategy
use and motivation were found. Given the potential limitations of the intervention effects due
to students' weak foundation in CC reading and the short duration of the previous study, thisstudy aimed to expand upon the author's 2020 study by incorporating a flipped classroom(FC) approach to add an eLearning component into SRL instruction and extending theintervention duration to two years. FC is a widely adopted blended pedagogical approachthat comprises two major components: in-class instruction and out-of-class eLearning. By
having students study content before class through online learning to free in-class time formore interactive and higher-level learning activities, FC reverses the traditional teachercentered classroom into student-centered learning (Berrett, 2012; Fulton, 2012; Strelan etal., 2020). Compared with solely online learning, FC put more emphasis on teachers’ role asa facilitator in face-to-face teaching to enable students to perform SRL (Öztürk & Çakıroğlu,2021; Sletten, 2017), which is consistent with the Principle S of SRL instruction. At the sametime, the out-of-class eLearning component of FC allows students to practice various SRL
skills independently, which is better than the traditional classroom instruction in
implementing the Principle A of SRL instruction (Al Mulhim, 2021; Rasheed et al., 2020).
Specifically, the study aimed to examine (1) whether SRL instruction was more
effective than the traditional teacher-centered instruction on enhancing students’ strategy
use, motivation, and performance in CC reading and (2) whether the inclusion of out-of-classeLearning activities produced more positive effects on enhancing students’ learning of CC
reading than when only SRL instruction was used.
Compared with many short-term SRL studies, the study adopted a longitudinal
design to guide students’ development of SRL through the four developmental levels.
Moreover, considering that CC reading is very difficult for Chinese students and has long
been dominated by teacher-centered instruction, it was particularly interested for the study toverify whether applying SRL instruction and FC in CC reading instruction could achieve
similar positive effects as in other subject areas. The findings should provide valuable
insights into the applicability of SRL instruction and FC in traditional subject areas.

2. Methodology
2.1 Participants
The study adopted a quasi-experimental design. A total of 352 Grade 8 students (172 boys;
179 girls; 1 unreported) between the ages of 12 and 14 years (mean = 12.94 years, SD =.63) from three secondary schools in Hong Kong participated in the study on a voluntarybasis. To ensure the comparability of students from different schools, all the three schools
mainly admit moderate-achieving students, use the same Chinese language textbook, andhave a similar number of Chinese language lessons each week. Students from each school
were assigned to one of the three treatment groups: SRL instruction plus out-of-classeLearning (EG-I+E), SRL instruction only (EG-I), and control (CG). Students in EG-I+E and
EG-I had the same number of Chinese language lessons, CC reading materials, and in-class
SRL instruction. While students in EG-I+E were assigned out-of-class eLearning activities
using a FC approach, EG-I students received post-class paper-based SRL tasks. Teachersof the CG used traditional teacher-centered instruction and assigned the same CC texts totheir students in the form of traditional comprehension exercises.

2.2 Design of the intervention program
The design of the intervention program was based on the TSAE framework and the fourstage SRL development model. The intervention program comprised four phases. One CC712

reading module was designed for each phase and implemented in one of the semesters of
the two academic years. Each module consists of twenty 40-minutes lessons. Different CC
texts were organized based on a humanistic theme of ancient Chinese culture and various
types of CC knowledge and reading strategies were included in each module to enhance
students’ CC reading ability. Interesting materials and interactive in-class learning activities
were designed to enhance student motivation. Pre- and post-class eLearning activities were
provided for students to make pre-class preparation and practice the knowledge andstrategies in new tasks. The importance of the TSAE instructional principles was adjusted
according to students’ developmental levels of self-regulatory skills. The detailed descriptionof the instructional design of each phase can be found in Lau (2021).

2.3 Measures
2.3.1 CC Reading Comprehension Tests
Two CC reading comprehension tests were designed to assess students’ CC reading ability
in each academic year. Each test comprised one narrative and one argumentative text. Five“word interpretation” questions and nine “text comprehension” questions were designed foreach type of text to assess students’ word- and text-level reading performance. The internalconsistency reliabilities of the pre- and post-reading test were .66 and .69, respectively.
2.3.2 Student Questionnaires
Two validated student questionnaires were adopted from the author’s studies (Lau, 2019;2020) to assess students’ strategy use and motivation in CC reading. The CC reading
strategy questionnaire comprised two subscales, “text comprehension strategies” and “wordinterpretation strategies” subscales, with 26 items to measure students’ use of different
strategies during their reading of CC texts. The internal consistency reliabilities of these two
subscales in pre- and post-test measures ranged from .85 to .91. The CC reading motivationquestionnaire comprised three six-item subscales measuring students’ self-efficacy, intrinsic
motivation, and extrinsic motivation in CC reading. The internal consistency reliabilities ofthese three subscales in pre- and post-test measures ranged from .86 to .92. All items in thetwo questionnaires were rated on a 5-point Likert scale.

2.4 Procedures
The instructional design of the program and the CC reading tests were reviewed by two
scholars and all EG teachers. The reading tests were piloted in a school with a similar
background to the experimental schools and refinements were made based on reviewers’comments and the pilot study’s results. The intervention started from the academic year of2021/2022 and ended at the academic year of 2022/2023. Teacher workshops and meetings
were held for EG teachers to support their preparation and evaluation of implementing theexperimental materials. All pre-tests were administered in September 2021 and the first andsecond wave of post-tests were administered in July 2022 and June 2023, respectively.

3. Results of the First Year
4.1 Student changes on CC Reading performance
Repeated measures ANOVA using time as the within-subject factor and treatment as the
between-group factor were performed to compare changes among different treatmentgroups. The findings indicated a significant time effect on reading comprehension test scores(F = 99.33, p < .001, Partial K2 = .227) but no significant time × group interaction effects were
found (F = 1.53, p = .218, Partial K2 = .009). The results of follow-up paired samples t-tests
indicated that all treatment groups significantly increased their reading comprehension testscores at the end of the first academic year (EG-I+E: t = 5.18, p < .001; EG-I: t = 4.47, p <.001; CG: t = 7.45, p < .001).

713

4.2 Student changes on Strategy Use in CC Reading
Repeated measures ANOVA indicated significant time effects in both word-level strategies(F = 52.70, p < .001, Partial K2 = .141) and text-level strategies (F = 18.41, p < .001, Partial
K2 = .056). A significant time × group interaction effect was found in word-level strategies (F= 12.76, p < .001, Partial K2 = .074). The results of paired samples t-tests indicated that whilestudents in the two EGs had similar and significant increases in their use of word-levelstrategies (EG-I+E: t = 5.34, p < .001; EG-I: t = 6.52, p < .001), CG students did not haveany significant change in the reading strategy post-test (t = .19, p = .852).

4.3 Student changes on Motivation in CC Reading
Repeated measures ANOVA indicated significant time effects in self-efficacy (F = 19.35, p <.001, Partial K2 = .055) and intrinsic motivation (F = 11.56, p < .01, Partial K2 = .034). A
significant time × group interaction effect was also found in both types of motivation (selfefficacy: F = 52.70, p < .001, Partial K2 = .141; intrinsic motivation: F = 52.70, p < .001,
Partial K2 = .141). The results of paired samples t-tests indicated that while students in thetwo EGs showed similar and significant improvement in their self-efficacy (EG-I+E: t = 3.54,
p < .01; EG-I: t = 3.80, p < .001) and intrinsic motivation (EG-I+E: t = 2.59, p < .05; EG-I: t =2.91, p < .01), CG students did not have any significant motivational change in the post-test(self-efficacy: t = .16, p = .113; intrinsic motivation: t = 29, p = .114).

4. Discussion and Conclusion
The significant improvement of EG students’ and CC reading performance and motivation
was consistent with previous SRL studies (e.g., Bai et al., 2021; Cousins et al., 2022;
Mohammadi et al., 2020; Otto & Kistner, 2017) and provided empirical evidence for
expanding the SRL research to the context of CC learning, indicating that SRL instruction is
also a feasible and effective method for enhancing students’ CC learning. Since students’major difficulties in CC reading are at the lexical level (Chen & Chen, 2020; Lau, 2019), thefirst two modules of the intervention program focused on the teaching of word interpretation
strategies to provide concrete ways for students to understand unfamiliar CC words. The
significant increase in EG students’ use of word interpretation strategies together with their
better reading performance support the importance of direct strategy instruction in SRL
instruction (Dignath & Büttner, 2018; Greene et al., 2015). Since most students regard CC
reading as very difficult, they usually have poor self-efficacy in CC reading (Lau, 2019; Chi &
Chiou, 2015). The increases of EG students’ self-efficacy, strategy use, and reading
performance are consistent with the view that students’ self-efficacy can be enhanced
through strategy learning and successful learning experiences (Schunk & Ertmer, 2000).
Moreover, the significant improvement in EG students’ intrinsic motivation supports thatinteresting and authentic instructional tasks and a high degree of autonomy can enhance
students’ intrinsic interest in learning (Dignath et al., 2008; Perry et al., 2002).
Contrasting to the view that teacher-centered instruction is not effective in enhancing
students’ CC reading performance (Lau, 2019; Liu, 2020), this study found that CG students
also demonstrated significant improvement in the CC reading post-test. As all treatmentgroups read a same number of CC texts during the study, this finding suggests that while EG
students improved their CC reading performance through consciously applying theknowledge and strategies they learned in the intervention program, CG students might
improve their CC reading ability through having more practices in doing CC reading
exercises. However, it is noteworthy that although CG students improved their performancein the reading comprehension post-test, their self-efficacy and intrinsic motivation was notenhanced because they only learned passively under the control of their teachers.
Both EG-I+E and EG-I had similar improvements in all post-test measures,
suggesting that adding the eLearning component in SRL instruction did not achieve
additional positive effect on student learning. The major advantage of eLearning inpromoting SRL is providing a flexible and autonomous environment for students to practice714

their SRL skills (Blau & Shamir-Inbal, 2017; So et al., 2019). As student autonomy is alsoemphasized in the Principle A of SRL instruction (Perry & Rahim, 2011; van Grinsven &Tillema, 2006), EG-I students might also perceive a high degree of autonomy whenparticipating in the student-centered in-class activities. Given that the in-class instruction forboth EGs was identical, it is possible that similar positive effects on students' learning wereachieved in both groups.
In conclusion, the findings of the first-year implementation of the intervention programprovide initial support for the effectiveness of using FC to integrate SRL instruction andeLearning to facilitate student learning in the context of CC reading. Despite CC reading
being considered a challenging and teacher-centered subject, this study's intervention
program design offers concrete ways for teachers to guide students in the gradualdevelopment of their SRL skills for CC reading. However, the similar improvements revealed
between the two EGs did not support the prediction that adding the eLearning component
should facilitate student SRL better than the traditional face-to-face instruction. The possible
benefits of using FC to increase the effectiveness of SRL instruction should be exploredfurther using the results of the second year when the degree of student autonomy and the
difficulty level of the leaning content is increased in the last two phrases. Finally, despite thesimilar backgrounds of the groups, controlling all confounding factors was challenging due topractical constraints in implementing a longitudinal intervention program within the students'
regular classes. To ensure better comparability between different treatment groups, futurestudies should consider measures such as assigning the same teachers to teach alltreatment groups and using the pre-tests results to screen participants into different groups.

Acknowledgements
This study was supported by by the Hong Kong Research Grants Council (RGC 14608820).

References
Al Mulhim, E. N. (2021). Flipped learning, self-regulated learning and learning retention of students withinternal/external locus of control. International Journal of Instruction, 14(1), 827-846.
Bai, B., Wang, J., & Zhou, H. (2021). An intervention study to improve primary school students’ selfregulated strategy use in English writing through e-learning in Hong Kong. Computer Assisted
Language Learning. Advance online publication.
Berrett, D. (2012). How “flipping” the classroom can improve the traditional lecture. The Chronicle ofHigher Education, 12, 1-14.
Blau, I., & Shamir-Inbal, T. (2017). Re-designed flipped learning model in an academic course: The roleof co-creation and co-regulation. Computers & Education, 115, 69-81.
Chen, M., & Chen, C. (2020). Do readers adjust their lower-and higher-level language skills according
to text structures? Evidence from eye movements in Chinese text reading. Journal of Research in
Reading, 43(2), 180-200.
Chi, L. C., & Chiou, G. F. (2015). The comprehension process of reading Classic Chinese texts (inChinese). Journal of Chinese Language Teaching, 12(2), 51–74.
Cousins, E., Bol, L., & Luo, T. (2022). Exploring long-term impacts of self-regulated learninginterventions in K-12 contexts: A systematic review. Current Issues in Education, 23(1).
Dignath, C., & Büttner, G. (2018). Teachers’ direct and indirect promotion of self-regulated learning inprimary and secondary school mathematics classes – Insights from video-based classroom
observations and teacher interviews. Metacognition and Learning, 13(2), 127-157.
Dignath, C., & Veenman, M. V. (2021). The role of direct strategy instruction and indirect activation ofself-regulated learning: Evidence from classroom observation studies. Educational Psychology
Review, 33(2), 489-533.
Dignath, C., Buettner, G., & Langfeldt, H. P. (2008). How can primary school students learn selfregulated learning strategies most effectively? A meta-analysis on self-regulation training
programmes. Educational Research Review, 3(2), 101-129.
Fulton, K. (2012). Upside down and inside out: Flip your classroom to improve student learning.
Learning & Leading with Technology, 39, 12–17.

715

Greene, J. A., Bolick, C. M., Jackson, W. P., Caprino, A. M., Oswald, C., & McVea, M. (2015). Domainspecificity of self-regulated learning processing in science and history. Contemporary Educational
Psychology, 42, 111–128.
Jayawardena, P. R., van Kraayenoord, C. E., & Carroll, A. (2019). Science teachers’ practices:Teaching for self-regulated learning in relation to Pintrich and Zusho’s (2007) model. International
Journal of Educational Research, 94, 100–112.
Lau, K. L. (2013). Chinese language teachers’ perception and implementation of self-regulated learningbased instruction. Teaching and Teacher Education, 31, 56-66.
Lau, K. L. (2019). Relationships between different instructional approaches and students’ classicalChinese reading comprehension and motivation. Journal of Research in Reading, 42, 523-540.
Lau, K. L. (2020). The effectiveness of self-regulated learning instruction on students’ classical Chinese
reading comprehension and motivation. Reading and Writing, 33(8), 2001-2027.
Lau, K. L. (2021). Integrating e-learning into self-regulated learning instruction: A holistic flippedclassroom design of a classical Chinese reading intervention program. In 29 th International
Conference on Computers in Education Conference Proceedings (Vol. 1), pp. 503-509.
Liu, R. (2020). Problems of classical Chinese reading instruction in junior language classes andstrategies for improvement (in Chinese). New Curriculum, 2020(9), 50-51.
Lombaerts, K., Engels, N., & van Braak, J. (2009). Determinants to teachers’ recognitions of selfregulated learning practices in elementary education. Journal of Educational Research, 102, 163–
173.
Mohammadi, R. R., Saeidi, M., & Ahangari, S. (2020). Self-regulated learning instruction and therelationships among self-regulation, reading comprehension and reading problem solving: PLSSEM approach. Cogent Education, 7(1), 1-22.
Neitzel, C., & Connor, L. (2017). Messages from the Milieu: Classroom instruction and context
influences on elementary school students’ self-regulated learning behaviors. Journal of Researchin Childhood Education, 31(4), 548–560.
Otto, B., & Kistner, S. (2017). Is there a Matthew effect in self-regulated learning and mathematicalstrategy application? Assessing the effects of a training program with standardized learning diaries.
Learning and Individual Differences, 55, 75–86.
Öztürk, M., & Çakıroğlu, Ü. (2021). Flipped learning design in EFL classrooms: implementing selfregulated learning strategies to develop language skills. Smart Learning Environments, 8(2), 1-20.
Perry, N. E., & Rahim, A. (2011). Studying self-regulated learning in classroom. In B. J. Zimmerman &D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 122–136). New
York: Routledge Press.
Perry, N. E., VandeKamp, K. O., Mercer, L. K., & Nordby, C. J. (2002). Investigating teacher-student
interactions that foster self-regulated learning. Educational Psychologist, 31, 5-15.
Rasheed, R. A., Kamsin, A., & Abdullah, N. A. (2020). Challenges in the online component of blended
learning: A systematic review. Computers & Education, 144, 103701.
Schunk, D. H. & Ertmer, P. A. (2000). Self-regulation and academic learning: Self-efficacy enhancing
intervention. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp.
631-650). Orlando, FL: Academic Press.
Schunk, D. H., & Zimmerman, B. J. (1997). Social origins of self-regulatory competence. Educational
psychologist, 32(4), 195-208.
Sletten, S. R. (2017). Investigating flipped learning: Student self-regulated learning, perceptions, andachievement in an introductory biology course. Journal of Science Education and Technology, 26,
347-358.
So, W. W. M., Chen, Y., & Wan, Z. H. (2019). Multimedia e-learning and self-regulated science learning:
A study of primary school learners’ experiences and perceptions. Journal of Science Education
and Technology, 28(5), 508-522.
Strelan, P., Osborn, A., & Palmer, E. (2020). The flipped classroom: A meta-analysis of effects on
student performance across disciplines and education levels. Educational Research Review,
100314
van Grinsven, L., & Tillema, H. (2006). Learning opportunities to support student self-regulation:
Comparing different instructional formats. Educational Research, 48, 77–91.
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich,
M. Zeidner (Eds.), Handbook of self-regulation (pp.531-566). San Diego: Academic Press.
Zimmerman, B. J. (2000). Attainment of self-regulation: A social cognitive perspective. In M. Boekaerts,
P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13-39). Orlando, FL: Academic
Press.

716

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Relations between Instructional Factors
and Student Acceptance of Flipped
Learning in Chinese Language Learning
Kit Ling LAUa & Quan QIANb
The Chinese University of Hong Kong, Hong Kong
*dinkylau@cuhk.edu.hk
bBeijing Normal University, ZhuhaiaAbstract: This study investigated the relations between instructional factors andstudents’ acceptance and use of flipped learning (FL) in the context of L1 Chinese
language learning. A total of 2160 students from ten secondary schools in Hong Kong
filled out two questionnaires measuring their perception of the instructional design andimplementation of FL in their Chinese language classes and their acceptance of andactual participation in FL activities. Findings of the descriptive analysis indicated thatfrom the perception of students, the design and implementation of FL in Chinese
language classes generally adhered to the instructional principles of FL. Students also
showed a moderately high level of acceptance of FL and a moderate level ofparticipation in FL activities. The results of structural equation modeling indicated thatboth the quality of in-class and out-of-class eLearning activities had significant andpositive effects on students’ perceived usefulness, ease of use, and enjoyment of FL
and, in turn, indirectly affected their actual participation of FL activities. The connection
between in-class and out-of-class eLearning activities also had a significant direct
effect on student participation. These findings highlight the important role of
instructional factors in promoting students’ willingness to accept FL as a new learning
approach in a traditional teacher-centered school subject.
Keywords: Chinese language learning, flipped learning, instructional factors, student
acceptance

1. Introduction
Flipped learning (FL) is a type of blended learning that emphasizes the active role oflearners by switching the traditional classroom instruction to pre-class home activities using
educational technologies (Blau & Shamir-Inbal, 2017; Bond, 2020). A typical flippedclassroom comprises two core components: out-of-class eLearning activities and face-toface classroom learning (Bergmann & Sams, 2012; Rasheed et al., 2020; Strelan et al.,2020). When compared to the traditional classroom learning, FL has many benefits tostudent learning. The out-of-class eLearning component of FL allows students to decide thetime, place, and frequency of learning based on their learning needs (Akçayır & Akçayır,
2018; Murillo-Zamorano et al., 2019; Rasheed et al., 2020). The in-class component of FL
focuses on in-depth discussions and interactive activities, which helps students deepen theirunderstanding of the learning content and enhance their higher-order thinking (Cheng et al.
al., 2018; Lo et al., 2018; Sletten, 2017). Many studies have supported the effectiveness of
FL in promoting students’ learning performance (Akçayır & Akçayır, 2018; Bond, 2020;
Strelan et al., 2020), motivation (Kirmizi & Kömeç, 2019, Zou et al., 2022) and self-regulated
learning (Blau & Shamir-Inbal, 2017; Çakiroglu & Öztürk, 2021).
As the learning approach of eLearning is largely different from the traditional teachercentered instruction, one major concern in technology enhanced learning research is
students' acceptance and intention to use the technologies (Estriegana et al., 2019; Teo,2019). As proposed by the technology acceptance model (TAM) (Davis, 1989; Venkatesh &717

Davis, 2000), users’ perception of the usefulness and ease of use of a new technology
significantly affects their intentional and actual use of the technology. In addition to perceived
usefulness and ease of use, some researchers have proposed to add perceived enjoyment
as another important indicator of student acceptance to represent students’ internalmotivation to use the technology (Padilla-MeléNdez et al., 2013; Zacharis, 2012). Although
TAM has been widely adopted to investigate user acceptance in many eLearning studies,
only a few studies applied this model in understanding students’ acceptance of FL (Chen
Hsieh et al., 2017; Haghighi et al., 2019; Padilla-MeléNdez et al., 2013). Most FL studies
only explored students’ general attitudes towards FL, and inconsistent findings were foundacross different studies. While some studies indicated that students held positive attitudes
toward FL (Akçayır & Akçayır, 2018; Kirmizi & Kömeç, 2019; Wang, 2016), some foundstudents preferred traditional instruction to FL because of the extra workload and lacking
teacher guidance in out-of-class eLearning (Akçayır & Akçayır, 2018; Bond, 2020; Lo & Hew,2017).
Instructional design and implementation pf FL have been posited as significant
determinants that affect students' perception and acceptance of technologies andconsequently affect their participation of FL activities (Dianati et al., 2022; Jiang et al., 2021).
Regarding the out-of-class eLearning component of FL, many studies revealed that thequality of online teaching videos and other eLearning materials and activities would affectstudents’ willingness to participant in out-of-class eLearning (Akçayır & Akçayır, 2018; Bond,
2020; Haghighi et al., 2019; Lau & Keung, 2021; Rasheed et al., 2020). In addition, explicit
instruction and constant feedbacks from teachers are important to support students’
independent learning in out-of-class eLearning activities, which can increase theiracceptance and use of FL (Dianati et al., 2022; Zou et al., 2020). Researchers have also
proposed some useful guidelines for the design and implementation of in-class learning
activities of FL. First, there should be a close connection between out-of-class eLearningand in-class learning activities (Bond, 2020; Ng, 2018; Rasheed et al., 2020). Second,
student-centered and cooperative learning activities should be designed to facilitate student
interactions and cultivate students' high-order cognitive abilities (Çakiroglu & Öztürk, 2021;
Haghighi et al., 2019; Lo & Hew, 2017; Strelan et al., 2020). Finally, teacher should also
provide support to facilitate student involvement during the higher-order and interactive
learning activities (Chen Hsieh et al., 2017; Sletten, 2017; Zou et al., 2020).
Compared with other subject areas, only a few FL studies were conducted in the areaof language learning and were predominated by studies in the field of learning English as asecond/foreign language (Turan & Akdag-Cimen, 2020; Zou, 2022). While findings ofprevious EFL/ESL studies supported the usefulness of FL in enhancing students' language
learning, certain difficulties and challenges were found when implementing FL in language
classes. After reviewing 43 studies on using FL to promote English learning, Turan andAkdag-Cimen (2020) concluded that although FL could facilitate students' interactions andoverall language performance, it increased the workload and technical problems for bothteachers and students. Studies on using FL to enhance language learning usually only
measured students’ general perceptions or attitudes of FL (e.g., Dianati et al, 2022; Kirmizi &Kömeç, 2019). Only a few studies adopted TAM as a comprehensive framework to examine
students’ acceptance of FL. For example, the results of TAM questionnaire in the study ofChen Hsieh et al (2017) indicated that students’ perceived usefulness, ease of use, andsystem characteristics were significant predictors of their intention of using FL to learn
English. Chen Hsieh et al’s (2017) study was also the only study that included instructional
factors in understanding students’ acceptance of FL in language learning. However, systemcharacteristics only represented one component of FL design. Up to the present, no exiting
studies have comprehensively examined the effects of different instructional factors on
students’ acceptance in the subject area of language learning.
2. The Present Study

718

The study aimed to investigate the relations between instructional factors and students’
acceptance and use of FL in the context of L1 Chinese language learning. As shown in theabove literature review, while FL has numerous positive effects on students’ learning, there
are significant challenges in students’ acceptance and use of this new learning approach. To
increase students’ acceptance and use of FL, the quality of the instructional design andimplementation of FL is critically important. Moreover, most FL studies on language learning
were conducted in the context of ESL/EFL. Influenced by Chinese Confucian culture,
Chinese language teachers are used to adopt the traditional teacher-centered approach(Lau, 2013) and not familiar with eLearning (Turan & Akdag-Cimen, 2019). Thus, whether
the design and implementation of FL in Chinese language classes adheres to the
instructional principles of FL and how these instructional factors may affect students’
acceptance and use of FL in this subject area need to be further explored.
Specifically, the study sought answers for the following three research questions:
RQ1: How did students perceive the instructional design and implementation of FL in theirChinese language classes?
RQ2: What were the level of students’ acceptance and use of FL in Chinese language
learning?
RQ3: Were students’ acceptance and use of FL significantly related to the instructional
design and implementation of FL in their Chinese language classes?

3. Methodology
3.1 Participants
Data of this study was drawn from a large-scale survey study entitled “Hong Kong students’
acceptance of using FL in Chinese language subject and influencing factors” led by the firstauthor. To ensure the representiveness of the sample, a similar number of students withdifferent achievement levels were invited to participate in the study on a voluntary basis. The
sample of this study consisted of 2160 students (1085 boys, 1017 girls, and 58 of unreported
gender) between the ages of 12 and 17 years (mean = 14.55 years, SD = 1.62) from ten
secondary schools in Hong Kong. Among these students, 724 were come from schools withhigh-achieving students, 742 from schools with moderate-achieving students, and 694 fromschools with low-achieving students.

3.2 Measures
The study adopted two questionnaires to measure students’ perception of the instructional
design and implementation of FL in their Chinese language classes and their acceptanceand use of FL. Reliability estimates of the two questionnaires are shown in Table 1.
3.2.1 Instructional Design and Implementation of Flipped Learning Questionnaire (IFLQ)This questionnaire was designed based on the instructional principles of FL and studies on
the effects of instructional factors on the effectiveness of FL (Akçayır & Akçayır, 2018;
Bergmann & Sams, 2012; Bond, 2020; Çakiroglu & Öztürk, 2021; Lo & Hew, 2017; Lo et al.,2018; Ng, 2018; Rasheed et al., 2020; Zou et al., 2020). It consisted of 36 items with fivesubscales. Two of the subscales focused on the out-of-class eLearning component of FL
measuring students’ perception of the quality of the eLearning materials and activities(OL_Q) and teacher support (OL_T) they received for out-of-class eLearning. Three of thesubscales focused on the in-class learning component of FL measuring students’ perceptionof the connection between out-of-class eLearning and in-class learning (CL_C), the quality of
the in-class learning materials and activities (CL_Q) and teacher support (CL_T). Students
were asked to rate how similar was the description of each item with the instructional design
and implementation of FL in their Chinese language classes on a 5-point Likert scale.

719

3.2.2 Technology Acceptance Model Questionnaire (TAMQ)
This questionnaire was adapted from the validated TAM questionnaires used in previous
studies (Davis, 1989; Estriegana et al., 2019; Padilla-MeléNdez et al., 2013; Scherer et al.,2019; Teo, 2019; Venkatesh & Davis, 2000). The wordings of some items were revised to fit
the context of Chinese language learning. The questionnaire consisted of 14 items with four
subscales. Besides the two core components of TAM, perceived usefulness (PE) andperceived ease of use (PEU), perceived enjoyment (PEN) was added because of its
importance in determining students’ acceptance of new technologies in the school setting(Lau & Keung, 2021; Padilla-MeléNdez et al., 2013). Students’ frequency of participating in
the FL activities in their Chinese language classes was used to replace the “intention of use”
variable in TAM to reflect their actual use (AU) (Estriegana et al., 2019; Zacharis, 2012).
Students were asked to rate the level of their agreement with each item on a 5-point Likert
scale.
Table 1. Descriptive statistics and reliability estimates for the two questionnaires
No. ofCronbach’s
Questionnaire/Subscale
Mean
Items
α
Instructional Design and Implementation of
Flipped Learning Questionnaire
Out-of-class eLearning: Quality of Learning
12
.95
3.63
Materials & Activities (OL_Q)
Out-of-class eLearning: Teacher Support (OL_T)
6
.92
3.65
In-class Learning: Quality of Learning Materials
8
.95
3.72
& Activities (CL_Q)
In-class Learning: Teacher Support (CL_T)
5
.93
3.82
Connection between Out-of-class eLearning &
5
.92
3.68
In-class Learning (CL_C)
Technology Acceptance Model Questionnaire
Perceived Effectiveness (PE)
5
.94
3.58
Perceived Ease of Use (PEU)
3
.87
3.65
Perceived Enjoyment (PEN)
3
.92
3.47
Actual Use (AU)
3
.86
3.25

SD

.75
.82
.79
.82
.81
.84
.85
.97
.84

3.3 Procedures and Data Analysis Plan
The initial version of the two questionnaires was reviewed by two researchers and four
experienced Chinese language teachers with good experiences on using FL. After making
revisions based on their comments, the questionnaires were piloted in a secondary schoolwith moderate-achieving students. Further refinements were made based on the results ofreliability estimates and confirmatory factor analysis (CFA) of the pilot study.
The questionnaires were distributed to the participating schools in June to August2022. Schools could choose to administer either the paper-based or online version ofquestionnaires to their students. All students were required to complete the questionnaires
anonymously under the supervision of their class teachers in about 10 minutes.
Standardized instructions for administration were prepared for teachers to ensure all schools
followed the same procedures to administer the questionnaires.
Reliability estimates and CFA were conducted again in the main study to check thepsychometric properties of the questionnaires. The findings indicated the reliability and factor
structure of both questionnaires were good (see Table 1 and 2). Descriptive analysis wasdone to examine students' perceptions of the instructional design and implementation of FL
in their Chinese language classes (RQ1) and their acceptance and use of FL (RQ2). SEM
was conducted to examine the relations between all latent variables of the study (RQ3).

720

Table 2. Goodness of fit for the measurement models and SEM model of the Study
Goodness-of-fit index
Measurement model
χ2
df
CFI
TLI
RMSEA

SRMR

IFLQ (5-factor model)
TAMQ (4-factor model)
SEM model

.029
.033
.042

3134.234***
323.771***5232.987***

584
71
1142

.933
.982
.929

.928
.977
.924

.045
.041
.041

4. Results
4.1 Instructional Design and Implementation of FL in Chinese Language Classes
As shown in Table 1, the mean scores of all subscales of IFLQ were higher than the midpoint of the 5-point Likert scale. Among the five subscales, the mean score of CL_T was thehighest, followed by CL_Q, CL_C, OL_T, and OL_Q. The findings indicated that from theperception of students, the design and implementation of FL in their Chinese language
classes generally adhered to the instructional principles of FL. Students showed a slightly
more positive perception on the in-class learning component of FL than the out-of-classeLearning component.

4.2 Student Acceptance and Use of FL in Chinese Language Learning
As shown in Table 1, the mean scores of all subscales of TAMQ were higher than the midpoint of the 5-point Likert scale. Among the three acceptance variables, the mean score of
PEU was the highest, followed by PE and PEN. The findings indicated that students
generally showed a positive attitude towards using FL in Chinese language learning. The
mean score of AU was slightly higher than the mid-point of the 5-point Likert scale, indicating
a moderate level of student participation in Chinese language FL activities.

4.3 Relations between Instructional Factors and Student Acceptance of FL in
Chinese Language Learning
SEM was adopted to explore the complex relationships between different instructional
factors and students' acceptance and use of FL in Chinese language learning. Based on theliterature reviewed above, the model hypothesized students' AU of FL as the dependent
latent variable, the five subscales of IFLQ as independent latent variables, and the threesubscales of TAMQ as mediators. Item scores were used as observed indicators for all
latent variables in the model. The SEM findings indicated a good fit for the hypothesizedmodel (see Table 2).
The results of SEM are shown in Figure 1. Regarding the relations between
instructional factors and students’ acceptance of FL, CL_Q had the largest significant effects
on PE, PEU, and PEN and OL_Q also had significant effects on these three acceptancevariables. No significant relation was found between the other three instructional variables
and student acceptance. Regarding the relation between students’ acceptance and use ofFL, PU had the largest significant effect on AU and PEN also had a significant effect on AU.
Among the five subscales of IFLQ, only CL_C had a significant direct effect on AU. CL_Q
and OL_Q mainly exerted effects on AU indirectly through the three acceptance variables.
The indirect effects of CL_Q and OL_Q on AU were .12 (p < .01) and .15 (p <. 01),
respectively.

721

Note.

To simplify the figure, only significant paths among the latent variables are shown in the model.
** p < .01; *** p < .001

Figure 1. Results of SEM on the relations between instructional factors, student acceptanceand use of FL in Chinese language learning.

5. Discussion and Conclusion
This study investigated Hong Kong secondary school students’ acceptance and use of FL in
the context of L1 Chinese language learning. Since FL is largely different from the traditional
teacher-centered instruction, previous studies found that not all students adapted to this newlearning approach (Akçayır & Akçayır, 2018; Blau & Shamir-Inbal, 2017; Bond, 2020). In
contrast to the prediction of the study that this problem may be more obvious when FL is
introduced to Chinese language classes which has long been dominated by the traditional
teacher-centered and knowledge-based instruction (Lau, 2013), the findings indicated amoderate level of student participation in Chinese language FL activities and students
generally showed a moderately high level of acceptance to this new learning approach.
Among the three acceptance variables, students had the most positive perception on ease ofuse. Since online learning has been adopted frequently in Hong Kong during the COVID-19
outbreak in the past three years, most Hong Kong secondary school teachers and students
are familiar with various types of eLearning tools and platforms, which may help reduce
students’ technical problems in FL. Students’ positive perceptions on the usefulness andenjoyment of FL are consistent with the positive effects of FL on students’ learning outcomes(Akçayır & Akçayır, 2018; Bond, 2020; Strelan et al., 2020) and motivation (Kirmizi & Kömeç,2019; Zou et al., 2022) revealed in many previous studies. As posited by some language
educators, FL move the teaching of a large amount of basic knowledge and lower-level
language skills to better organized and self-paced pre-class self-learning and free class time
for students to interact actively with their classmates to practice and apply their higher-orderlanguage skills, which consequently promote higher motivation and proficiency in language
development (Li et al., 2022; Jiang et al., 2022; Turan & Akdag-Cimen, 2019). The study’s
findings support that these proposed benefits of FL in language learning are also endorsedby Hong Kong students when FL is incorporated in their Chinese language classes.
Students’ positive perception on the design and implementation of FL in Chinese
language classes refute the traditional view that Chinese language teachers tend to adopt
teacher-centered instruction (Lau, 2013) and FL approach is seldom applied in Chinese722

language teaching (Zou et al., 2022). This may be due to a greater emphasis of studentcentered instruction in current Chinese language curriculum (HKCDC, 2017) and teachers’
more frequent use of eLearning tools and platform during the COVID-19 outbreak.
Nevertheless, the study revealed that students held more positive perception on the in-classlearning component of FL than the out-of-class eLearning component. It suggests that Hong
Kong Chinese language teachers are still more capable in designing high quality learningmaterials and activities and supporting students’ learning during face-to-face classroom
teaching than in the out-of-class eLearning context. Thus, more trainings should be provided
for Chinese language teachers to enhance their professional skills in incorporating eLearningin Chinese language teaching.
In line with the previous TAM studies (Chen Hsieh et al., 2017; Estriegana et al.,2019; Haghighi et al., 2019; Padilla-MeléNdez et al., 2013; Zacharis, 2012), PE and PEN
were found to have significant effects on AU. While the largest effect of PU on AU confirms
again that perceived usefulness is the most critical variable in TAM (Scherer et al., 2019),
the significant relation between PEN and AU and the non-significant relation between PEU
and AU suggest that young school students consider the feeling of enjoyment more than the
difficulty level of the tasks when they determine whether they would like to participate in FL
activities (Padilla-MeléNdez et al., 2013; Zacharis, 2012). The significant direct effects of the
instructional factors on student acceptance of FL support the important role of instructional
factors in enhancing students’ acceptance of using FL (Cheng, 2019; Dianati et al., 2022;
Hsieh et al., 2017; Jiang et al., 2021). Among various instructional factors, CL_Q and OL_Q
were found to have significant effects on all the three student acceptance variables andindirectly affected AU. These findings are consistent with the view that student-centered,higher-order, and interactive in-class activities, and interesting, diverse, and well-designed
eLearning materials and activities can enhance students’ positive attitude towards FL and, inturn, increase their willingness to participate actively in FL activities (Akçayır & Akçayır,
2018; Blau & Shamir-Inbal, 2017; Bond, 2020; Haghighi et al., 2019; Lau & Keung, 2021;
Rasheed et al., 2020). It is noteworthy that CL_C did not have any significant effect on
student acceptance but it had a direct effect on AU. Since many students view out-of-classeLearning activities as extra work for them and are not willing to make pre-class preparation(Akçayır & Akçayır, 2018; Rashead et al., 2020), teachers’ effective use of follow-up
activities is essential for encouraging students to complete pre-class learning tasks (Blau &Shamir-Inbal, 2017; Lau & Keung, 2021; Lo et al. 2018).
In conclusion, while previous FL studies have predominately focused on theeffectiveness of FL on student learning, the study made a unique contribution to the FL
research by verifying the relations between instructional factors, students’ acceptance, andtheir use of FL using more comprehensive measures of instructional factors and student
acceptance. The study’s findings highlight the importance of designing high quality in-classand out-of-class eLearning materials and activates to increase students’ willingness toparticipate in FL activities. The study’s findings also confirm the mediating role of student
acceptance between instructional factors and students’ actual participation in FC activities.
Students’ positive perceptions on the design and implementation of FC in their Chinese
language classes and their high level of acceptance of this new learning approach support
the potential benefits of adopting FL to enhance student learning in this traditional teacherdominated school subject. Lastly, it should be noted that the measure of instructional factors
in the study were based only on students’ perception. The significant relations between
instructional factors and student acceptance of FL should be replicated in future studies
using direct measures of instructional design and implementation.

Acknowledgements
This study was supported by the Standing Committee on Language Education and Research(EDB(LE)/P&R/EL/203/2).

723

References
Akçayır, G., & Akçayır, M. (2018). The flipped classroom: A review of its advantages and challenges.
Computers & Education, 126, 334-345. https://doi.org/10.1016/j.compedu.2018.07.021
Bergmann, J., & Sams, A. (2012). Flip your classroom: Reach every student in every class every day.
DC: International Society for Technology in Education.
Blau, I., & Shamir-Inbal, T. (2017). Re-designed flipped learning model in an academic course: The roleof co-creation and co-regulation. Computers & Education, 115, 69-81.
Bond, M. (2020). Facilitating student engagement through the flipped classroom approach in K-12: A
systematic review. Computers & Education, 151, 1-36.
Öztürk, M., & Çakıroğlu, Ü. (2021). Flipped learning design in EFL classrooms: implementing selfregulated learning strategies to develop language skills. Smart Learning Environments, 8(1), 2.
Çakiroglu, Ü., & Öztürk, M. (2021). Cultivating self-regulated learning in flipped EFL courses: A model
for course design. European Journal of Open, Distance and E-Learning, 23(2), 20-36.
Chen Hsieh, J. S. C., Huang, Y. M., & Wu, W. C. V. (2017). Technological acceptance of LINE in flipped
EFL oral training. Computers in Human Behavior, 70, 178-190.
Chen Hsieh, J. S., Wu, W. C. V., & Marek, M. W. (2017). Using the flipped classroom to enhance EFL
learning. Computer Assisted Language Learning, 30(1-2), 1-21.
Cheng, L., Ritzhaupt, A. D., & Antonenko, P. (2018). Effects of the flipped classroom instructional
strategy on students’ learning outcomes: A meta-analysis. Educational Technology Research &Development, 67, 793–824.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS quarterly, 13(3), 319-340.
Dianati, S., Nguyen, M., Dao, P., Iwashita, N., & Vasquez, C. (2020). Student perceptions oftechnological tools for flipped instruction: The case of Padlet, Kahoot! and Cirrus. Journal of
University Teaching & Learning Practice, 17(5). https://doi.org/10.53761/1.17.5.4
Estriegana, R., Medina-Merodio, J., Barchino, R. (2019). Student acceptance of virtual laboratory andpractical work: An extension of the technology acceptance model. Computers & Education, 135, 114.
Haghighi, H., Jafarigohar, M., Khoshsima, H., & Vahdany, F. (2019). Impact of flipped classroom on
EFL learners' appropriate use of refusal: achievement, participation, perception. Computer
Assisted Language Learning, 32(3), 261-293.
Hong Kong Curriculum Development Council (HKCDC). (2017). Chinese language education
curriculum guide (Primary One to Secondary Six). Hong Kong: Government Printer.
Jiang, M. Y. C., Jong, M. S. Y., Wu, N., Shen, B., Chai, C. S., Lau, W. W. F., & Huang, B. (2022).
Integrating automatic speech recognition technology into vocabulary learning in a flipped English
class for Chinese college students. Frontiers in Psychology, 13,1-16.
Kirmizi, Ö., & Kömeç, F. (2019). The impact of the flipped classroom on receptive and productive
vocabulary learning. Journal of Language and Linguistic Studies, 15(2), 437-449.
Lau, K. L. (2013). Chinese language teachers' perception and implementation of self-regulated learningbased instruction. Teaching and teacher education, 31, 56-66.
Lau, K.L., & Jong, M.S.Y. (2022). Acceptance of and self-regulatory practices in online learning andtheir effects on the participation of Hong Kong secondary school students in online learning.
Education and Information Technologies, Online First article. https://doi.org/10.1007/s10639-02211546-yLau, K. L., & Keung, H. M. (2021). Effectiveness and user acceptance of using out-of-class e-learning
activities to support classical Chinese learning. Journal of Chinese Language Teaching, 18(3),37-72.
Lo, C. K., Lie, C. W., & Hew, K. F. (2018). Applying “First Principles of Instruction” as a design theoryof the flipped classroom: Findings from a collective study of four secondary school
subjects. Computers & Education, 118, 150-165.
Lo, C., & Hew, K. (2017). A critical review of flipped classroom challenges in K-12 education: Possible
solutions and recommendations for future research. Research and Practice in Technology
Enhanced Learning, 12(4), 1-22.
Li, S., He, J., Tao, Y., & Liu, X. (2022). The effects of flipped classroom approach in EFL teaching: Can
we strategically use the flipped method to acquire communicative competence? Language
Teaching Research. Advance online publication.
Murillo-Zamorano, L. R., Sánchez, J. Á. L., & Godoy-Caballero, A. L. (2019). How the flipped classroom
affects knowledge, skills, and engagement in higher education: Effects on students' satisfaction.
Computers & Education, 141, 1-18.

724

Ng, E. M. (2018). Integrating self-regulation principles with flipped classroom pedagogy for first year
university students. Computers & Education, 126, 65-74.
Padilla-MeléNdez, A., Del Aguila-Obra, A. R., & Garrido-Moreno, A. (2013). Perceived playfulness,gender differences and technology acceptance model in a blended learning scenario. Computers& Education, 63, 306-317.
Rasheed, R. A., Kamsin, A., & Abdullah, N. A. (2020). Challenges in the online component of blended
learning: A systematic review. Computers & Education, 144, 1-17.
Scherer, R., Siddiq, F., & Tondeur, J. (2019). The technology acceptance model (TAM): A meta-analytic
structural equation modeling approach to explaining teachers’ adoption of digital technology in
education. Computers & Education, 128, 13-35.
Sletten, S. R. (2017). Investigating flipped learning: Student self-regulated learning, perceptions, andachievement in an introductory biology course. Journal of Science Education and Technology,
26(3), 347-358.
Strelan, P., Osborn, A., & Palmer, E. (2020). The flipped classroom: A meta-analysis of effects on
student performance across disciplines and education levels. Educational Research Review, 30,
1-22. https://doi.org/10.1016/j.edurev.2020.100314
Teo, T. (2019). Students and teachers' intention to use technology: Assessing their measurement
equivalence and structural invariance. Journal of Educational Computing Research, 57(1), 201225. https://doi.org/10.1177/0735633117749430
Turan, Z., & Akdag-Cimen, B. (2020). Flipped classroom in English language teaching: a systematic
review. Computer Assisted Language Learning, 33(5-6), 590-606.
Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Fourlongitudinal field studies. Management science, 46(2), 186-204.
Wang, Y. H. (2016). Could a mobile-assisted learning system support flipped classrooms for classicalChinese learning? Journal of Computer Assisted Learning, 32, 391–415.
Zacharis, N. Z. (2012). Predicting college students’ acceptance of podcasting as a learning tool.
Interactive Technology and Smart Education, 9(3), 171–183.
Zou, D., Luo, S., Xie, H., & Hwang, G. J. (2022). A systematic review of research on flipped language
classrooms: theoretical foundations, learning activities, tools, research topics and findings.
Computer Assisted Language Learning, 35(8), 1811-1837.

725

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The AI-Supported Instructional Design in
PBL Integrating Chinese Language
Learning and Multimedia Creation
Satoko SUGIE a*a Faculty of Humanities, Sapporo International University, Japan* satoko-sugie@ts.siu.ac.jp
Abstract: The rapid development of AI (Artificial Intelligence) is enhancing the potential
to accelerate a paradigm shift in foreign language teaching and learning. Instructional
design must shift beyond knowledge learning and skills training to include moresignificant opportunities for practical application in authentic contexts. To do so, it isnecessary to investigate the pedagogical possibilities of generative AI through specificeducational practices. Thus this study intends to explore the potential of using
generative AI to improve the efficiency and effectiveness of PBL (Problem/Project
Based Learning) which integrates foreign language, information media, andprofessional education. This study suggests a PBL course design integrating Chinese
language, tourism, and multimedia creation learning outcomes at a Japanese
university, as a preliminary phase to compare and examine the teaching and learning
processes between traditional methods and proactive AI-enabled methods. Throughqualitative exploration of integrating generative AI into traditional instructional design,
this paper considers how to organize the elements of teaching and learning for arenewed approach with greater efficiency and quality of learning outcomes.
Keywords: generative AI, Chinese as a foreign language, multimedia creation,
instructional design, Project-Based Learning

1. Introduction
Japanese universities commonly teach introductory foreign language courses throughlecture-style classes, where in-class activities are mainly individual knowledge learning.
Since the outbreak of the COVID-19 pandemic, the use of ICT (Information andCommunication Technology) in education has become somewhat more normal. From a
broader perspective, blended and hybrid ICT-based teaching/learning has been identified asa worthy challenge to assist teachers in proficiency of ICT skills (Beatty, 2019). However,even during the pandemic, MEXT (Japan’s Ministry of Education, Culture, Sports, Science,
and Technology) emphasized traditional, in-person instruction (MEXT, 2021a), expressing ahope that by 2023, there would be a return to traditional teaching (MEXT, 2021b).
In contrast to government hopes, the evolution of generative AI has been a societal
paradigm shift that educators noticed. Generative AI supports more efficient education of
vocabulary, grammar, and writing in foreign languages. Frequent updates have further
enhanced accuracy, rendering AI more cost-effective (Kondo et al., 2023; Niño, 2020; Tsai,
2020). However, most traditional teachers doubt the improvement through AI, and thinkcommunication practice cannot be achieved without repeating physical lessons on
vocabulary, grammar, and pronunciation (Onishi, 2008). Teachers from a behaviorist orcommunicative background trust far more in the methods they applied during their own
success as foreign language learners. Studies investigating how to use generative AI arevital because the problem of ‘the Chasm’ (Moore, 1991) is equally true in the education field,
and there may be a deep divide between early adopters and the majority of teachers on how
best to apply such new technology.

726

2. Research Question
This study seeks to use the Japanese government’s tentative guidelines for the useof generative AI (MEXT, 2023). The seven points of that guideline can be summarized asfollows.
Table 1. The seven points of the tentative guidelines for the use of generative AI

Since this study focuses on foreign language education, there are possibilities forlearners to use guidelines 3, 4, and 5, and for teachers to use 7.
This study is action research to explore situational questions with a qualitative
approach. Based on the instructional design of a class conducted by the author in 2023,
learning activities and tools have been organized which could potentially apply generative AI.
To examine the effectiveness of teaching and learning using generative AI compared to
traditional teaching and learning strategies, it is necessary to identify what teaching andlearning processes are used in traditional instructional design, starting with an exploration of
the potential for improvement. The research question of this study is as follows: How could
AI enhance a PBL course integrating foreign language learning and multimedia creation?

3. Methods
In this study, new viewpoints on how to integrate generative AI were examined incomparison to traditional instructional design, through subjective observations based on thisauthor’s years of teaching experience. Analysis was conducted on the design of a springsemester course (15 weeks), considering the strategies, outcomes and likely challenges inlesson design and learning tasks.

3.1 Class Setting
The learners were twenty-three first-year Japanese university students who have studied
basic Chinese as a foreign Language for less than one year. The class is composed of a
mixture of Humanities and Science majors. The learning environment is a face-to-face classin a general classroom, where students therefore bring their own devices such as laptops
and smartphones. The network environment is connected to the university's Wi-Fi. Learners
use the LMS (Learning Management System) for class communication, browsing learningmaterials, submission of assignments, and reviewing teacher feedback. Grading is based on
the submission of in-class and out-of-class assignments, in-class discussions, thesubmission of videos on learning outcomes, and a final presentation in Chinese.

727

3.2 Learning Goals and Activities
The class has two main themes. The first is cross-cultural understanding and vocabulary
learning necessary for welcoming Chinese FIT (Foreign Independent Tour) visitors to
Hokkaido, Japan’s Northernmost province. For this purpose, students learn words, phrases,
and conversational expressions used for tourism hospitality through a set textbook.
Moreover interactive learning happens as international students answer learners' questions
about Chinese culture. Second, the students create a tourism promotional video in Chinese.
For this purpose, students make a video in groups that discovers, co-creates, and promotes
local attractions from the learner's perspective. This requires extracurricular collaboration.
The three learning goals are as follows: (1) To be able to address and answer simple
Chinese questions in the tourism industry and hospitality situations, (2) Understand thenorms, beliefs, and culture of Chinese visitors to Japan from a cross-cultural perspective,and (3) To be able to utilize Chinese through tools including distance learning, automatic
voice translators, and video editing applications.

4. Data analysis
A qualitative approach was adopted involving analysis of lesson plans, teaching experienceand evaluation of possible student outcomes. The comparison of traditional and AIenhanced course designs, including learner and teacher outcomes are shown in Figure 1.
The conventional work of teachers and learners across the usual fifteen-week course wassummarized, and then potential AI-supported alternatives were devised.

728

Figure 1. Comparison of AI-supported instructional design, activities, and tasks729

5. Discussion
The improvements over the traditional teaching and learning process as outlined in Figure 1
can be categorized into the following seven aims. The acronyms shown in [ ] in AI-supported
elements in Fig.1 represent the same seven elements.
(1) [LI/TR] Language input and translation: Japanese to Chinese/Chinese to Japanesetranslation by text, Chinese input by voice(2) [PRON] Pronunciation Practice: Input (listen for accurate pronunciation) and output(correct pronunciation)(3) [DE/FAQ] Dialogue exercise, question and answer.
(4) [INF-CL] Information collection(5) [INF-CT] Information categorization(6) [INF-S] Information summary(7) [MG] Multimodal generation: create images, audios, and videos.
The following are examples of tools and applications that could be used for teaching
and learning activities. Most are free, but if there is a fee, it would be necessary to work withthe organization's instructional management department or for individual teachers tomanage the expense of using student licenses and setting up registration management.
(1) Translation: DeepL, Google Translate, ChatGPT, Bing and Bard
(2) Pronunciation practice: Ondoku-san (ondoku3.com/ja), Microsoft Azure(speech.microsoft.com/audiocontentcreation), Ondoku Chinese (ondoku.org)(3) Oral presentation practice: RESEMBLE.AI (English to Chinese conversion)(www.resemble.ai)(4) Dialogue practice: (1) Text-based dialogue: ChatGPT, (2) Voice-based dialogue:
TalkBerry (ChatGPT + GoogleChromeAddOn)(5) Image/video creation: Microsoft Designer (designer.microsoft.com), Google Imagen
Video (imagen.research.google/video), Meta Make a Video (makeavideo. studio), Gen-2(research.runwayml.com/gen2), Stable Diffusion (stablediffusionweb.com), Scribble
Diffusion (scribblediffusion.com)
We have recommended the use of such tools for (1) machine translation and (2)speech recognition for text input and pronunciation correction among traditional learning
activities. However, they were only used during extracurricular individual learning. To
increase the ratio of use in cooperative and autonomous learning within the classroom andrealize more efficient activities, it is necessary to consider the optimal teaching methods
which allow learners to know which tools to use in which situations, and how to practice theirskills.
The following are points to be considered in AI-supported instructional design.
Generative AI is still in the early trial-and-error stage of development so teachers must
determine which tools are optimal for which tasks. Teachers need to continue to improve thetask design and instruction of the most appropriate AI-enabled pedagogical methods throughimplementation and review of practical data. The cost of learners' low proficiency in tool
operation itself will squeeze time for cooperative learning activities and interactive feedbackin the classroom. It is thus necessary to improve AI literacy in the form of cross-curricular
activities using on-demand videos and flipped learning. Likewise, it is preferable to establish
a team-based instructional design and class management system.

6. Conclusion and Future Challenges
This study examined the potential of using generative AI in PBL, which combines foreign
language, ICT utilization, and professional education. Seven key areas were identified wheregenerative AI could improve teaching and learning efficiency. However, human teachers are730

still needed to provide integrated instruction based on their experience in dealing with eachdiverse learner. In conversations and collaborative work, teachers must provide pedagogicalinstruction that fosters autonomous, interactive, and in-depth learning. As well as blended or
hybrid ICT-based teaching/learning, the AI-supported instructional design also requires
establishing a cross-subject educational system by networking with information course
teachers and other advanced educational practitioners. It is essential to reconfirm thatteachers are core actors for instructional design, and then recognize how best to allocate
teaching time to AI-enabled resources. In addition to that, I believe that it is necessary to
maximize the allocation of learning support that only a human teacher can provide.
To test the implications of this study, future studies must add quantitative practical
data on implementation of the suggested instructional design. As this field grows, it is vital tosee more sustained comparison of the learning process and outcomes between traditional
learning methods and activities fully utilizing a generative AI.

Acknowledgments
This work was supported by JSPS KAKENHI Grant Number 19H01276.

References
Beatty, B. J. Costs and Benefits for Hybrid-Flexible Courses and Programs. In Beatty, B. J. (2019)
Hybrid-Flexible Course Design: Implementing Student-Directed Hybrid Classes, EdTech Books.
https://edtechbooks.org/hyflex/
Katelay, A., Ide, K., & Kishimoto, M. (2023, April 13). Seisei AI (Generative AI) no ronri-teki hōteki
shakai-teki kadai (ELSI) ronten no gaikan [Overview of ethical, legal, and social issues (ELSI) in
generative AI] Version March 2023. https://elsi.osaka-u.ac.jp/research/2120
Kondo, Y., Kimura, S., Sakaba, H., Toyoshima, C., Nakanan, M., Yamashita, M., Yamanaka, T.
(2023). AI kikai honyaku no eigo seika jugyo he no daikibo donyu to sono kadai [Feasibility Studyon Large-scale Introduction of Neural Machine Translation into PBL English Classes],
Proceedings of the CIEC Spring Conference2023, (14), 41-44.
Lave, J. and Wenger, E. (1991). Situated Learning: Legitimate Peripheral Participation, Cambridge
University Press.
Ministry of Education Culture, Sports, Science and Technology [MEXT]. (2021a). Reiwa 3 nendo koki
no daigaku tou ni okeru jugyo no jisshi hoshin tou ni tuite [Notes on the Conduct of Classes atUniversities and Other Institutions in Academic Year 2021 and Countermeasures against New
Coronavirus Infections, etc. (Notification)]. https://www.mext.go.jp/content/20210305mxt_kouhou01-000004520-02.pdf
Ministry of Education Culture, Sports, Science and Technology [MEXT]. (2021b). Policy on the
Conduct of Classes at Universities and Other Institutions in the First Half of the 2021 Academic
Year. https://www.mext.go.jp/content/20210702-mxt_kouhou01-000004520_2.pdf
Ministry of Education Culture, Sports, Science and Technology [MEXT]. (2023). Tentative Guidelines
for the Utilization of Generative AI in the Primary and Secondary Education Levels.
https://www.mext.go.jp/a_menu/other/mext_02412.html
Moore, G. A. (1991). Crossing the Chasm: Marketing and Selling Technology Products to Mainstream
Customers, HarperCollins.
Niño, A. (2020). Exploring the use of online machine translation for independent language learning.
Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2402
Onishi, H. (2008). Korekara no daini gaikokugo kyoiku no houkousei – Chugokugo touitsu tekisuto
Kaihatsu no kokoromi [Direction of Second Language Education from Now Onward: Efforts in
Developing a Unified Textbook for Chinese]. Kinki University Department of Language Education
Journal, 4, 13-24.
Sugie, S. (2023, March 18). Kanko Chugokugo Maruchimedia wo tougou shita purojekuto gatagakushu no seika to kadai [Achievements and Challenges of Project-Based Learning Integrating
Tourism, Chinese, and Multimedia Education], 20th Association for e-Learning Language
Education Annual Conference 2023, Ritsumeikan University, Umeda campus.
https://researchmap.jp/ssugie/presentations/41770097
Tsai, S. C. (2020). Chinese students’ perceptions of using Google Translate as a translingual CALL
tool in EFL writing, Computer Assisted Language Learning, 35(5-6)1250-1272.
https://doi.org/10.1080/09588221.2020.1799412

731

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Experimental Comparison of Promotion
Effect for EFL Reading Comprehension
between Conventional Summarization andToulmin Argument Reconstruction
Banni Satria ANDOKOa*, Rosa Andrie ASMARAa, Vivin Ayu LESTARIa, Deasy Sandhya
Elya IKAWATIa, Arief PRASETYOa, Tsukasa HIRASHIMAb & Yusuke HAYASHIba
Information Technology Dept, State Polytechnic of Malang, IndonesiabGraduate School of Advanced Science and Engineering, Hiroshima University, Japan*ando@polinema.ac.id
Abstract: Summarization is a conventional task to promote reading comprehension.
However, because it is impossible to diagnose the summary immediately andaccurately, it is not easy to realize formative feedback for ongoing reading
comprehension. To solve this issue, we have proposed Viat-map reconstruction as atask for promotion of reading comprehension. Viat-map is a three-component (Data,
Warrant and Claim) representation of the Toulmin Argument. Reconstruction meansthat a learner is required to reconstruct a Viat-map prepared by an expert by using
provided components. By comparing the reconstructed map with the original map, it ispossible to realize immediate and accurate diagnosis and ongoing formative feedback
for the reading comprehension. We have conducted a comparative experiment ofpromotion effect for EFL reading comprehension between conventional summarization
and Viat-map reconstruction. Fifty-eight second year students of Information
Technology Department from State Polytechnic of Malang, Indonesia were involved in
the experiments. All of them are using English as Foreign Language. The experiment
was conducted in an English course Subject for three weeks. Experiment and control
group were created equally based on the initial test of reading comprehension test. The
experiment group was using Viat-map, while control group is using conventional
summarization method. The result of the experiment was analyzed by using ANOVA
statistical analysis and Holm’s Sequentially Rejective Bonferroni Procedure for POST
Hoc analysis. The result shows that Viat-map reconstruction overachieved
summarization and both groups can maintain the memory one week after the lesson.
However, the delay-post test score for experiment group is higher than control group.
Keywords: English Reading Comprehension, Toulmin Argument, Viat-map, EFL

1. Introduction
English has become one of the most popular languages in the world and is recognized as themost widespread language of communication in the fields of education, technology, business,
diplomacy, science, and sports, as well as the service-oriented high-tech industries and manyother fields (Zhang, 2011). For every student, regardless of their English proficiency, it isimportant that they can implement their writing, reading, speaking, and listening skills in theiracademic or scientific environment as well as their social life. (Dehham et al., 2022). In formal
education, studying English as English as Foreign Language (EFL) students, took moreaccount on reading abilities, normally because it is the main point of contact with the target
language. English knowledge as international language of communication, is becoming morenecessary for many adult university learners (Allard & Mizoguchi, 2021).
Reading is one of the important activities to gain knowledge and information. In otherwords, reading is one of the gateways to gaining an understanding. Because reading is anactive and continuous process that involves people with reading material to build meaning or
information (Carrell, 1989). Reading is also interpreted as an active task in which the reader

732

makes choices from specific words derived from the text and relates them to the situational
context to build a model of meaning that describes the meaning that is the same as themeaning intended by the writer (Dole et al., 1991).
Sentence structures in reading materials are generally presented to readers in the formof linear sentences without any good directions to assist the reader in recognizing the logicalstructure of the text (Eftekhari et al., 2016; Eftekhari & Sotoudehnama, 2018). The condition
creates a huge confusion, especially for EFL students. Logical structure of text could assist
students to promote an autonomous learning and enhance the depth of learning (Andoko etal., 2020). Toulmin Argument could also be applied to practical teaching(Yang, 2022b, 2022a).
The motivation of this study is assisting students to understand the reading material
by using Toulmin Arguments strategy and Viat-map application. Research questions of thisstudy are :1. Do students who use Viat-map application have a better understanding compared
to ones who use conventional summarization method?
2. Do students who use Viat-map application maintain their understanding compared
to ones who use conventional summarization method?
Several procedures will be conducted to answer the research question, measuring thelevel of students understanding and memory retention when using the Viat-map applicationcompared to the ones who are using the conventional summarization method. First, we willcompare the pre-test score with the post-test score from each group to find any improvement
and then comparing the post-test score between groups to find any differences as therepresentative of students’ understanding. Secondly, we will compare the delayed test scorewith the post-test score from each group to find any changes and then compare the delayed
test between groups as the representative of students’ memory retention.

2. Literature Review and Related Reseach
2.1 Toulmin Argument
Toulmin Arguments is a form of implementation of a graphical strategy based on the
Argumentation model. The argumentation model can give confidence and influence on thereader because it presents logical and strong reasons to prove the truth of an opinion basedon data and facts (Hitchcock, 2005; Junaidah Januin, 2021). Toulmin's argument is a complete
argumentation pattern and has a good understanding concept, Toulmin's argument has adescription of the learning process which consists of claims, qualifiers, ground, warrants,
rebuttals, and backing (Magalhães, 2020; Toulmin, 2003). Toulmin's argument is formed by 3
basic elements, namely Claim, Ground and Warrant (Figure 1). Claim is a sentence,
information, conclusion to be conveyed. Ground are facts, evidence or data that become areference for a Claim, while Warrant is a sentence that connects Ground with a Claim. These
three must be formed into a logical framework to provide a strong understanding of literacy(Andoko et al., 2022).

GroundClaim
Warrant

Figure 1. Toulmin Argument concept.

733

2.2 Viat-map
Viat-map is an application developed to provide a deep understanding of literacy. Using it for
Reading Comprehension in English for EFL students shows a promising result (Andoko et al.,2022; Rismanto et al., 2021). The main concept is to form a logical framework of text byadopting Toulmin's argument. It intends to emphasize reasoning so students can construct astrong understanding of information within the text. There are three main phases of Viat-map(Figure 2) :1. Teacher’s logical map: in this phase, teacher is responsible to create several exercises byselecting important sentences as claim and provide one correct evidence or fact as groundalong with two incorrect evidences. Teacher also provides one correct relation between
sentences and evidence (warrant) followed by two incorrect relations.
2. Students’ working space: in this phase, students are constructing provided exercise by theteacher by selecting the correct ground and warrant, they cannot move to the next claim
until they find the correct ground and warrant.
3. Teacher’s overlapping analysis feedback: in this phase, teacher can provide feedback byusing the overlapping analysis. Teacher can emphasize more feedback by following thenumber present in each line of answer. The number consists of two different pieces of
information, the first number stands for the number of mistakes made by the students, thesecond number is for the total number of students who made the mistakes. For further
information, the teacher can select the line and it will show the information of the students
and the number of mistakes made by each student.

Phase 1

Phase 2Phase 3Figure 2. The phases of Viat-map application.

734

4. Methods
4.1 Experimental Setting
The study took place in October 2022 in an English subject Class of second year Information
Engineering Department student of State Polytechnic of Malang. All the students were using
English as Foreign Language. There were fifty-eight students involved coming from two
different classes. The student never used and familiar with Toulmin argument before. The
study was conducted in three weeks where the first week is used to introduce them with
Toulmin Argument concept, Viat-map application and to conduct the initial test used tocategorize students into a control or experiment group. Each class was divided into two groups
to ensure the consistency of the experiment. The next following week was used to conduct
the experiment. The third week was used for delayed test. During the experiment, the teacher
never give explanation or feedback. Her task was providing the reading material, creating the
Toulmin Argument and evaluating the score.
Control groups would use the conventional summarization method where they werecreating a summary from the reading material to simplify or to emphasize the most importantsentence and words. While control group would be using the Viat-map application to constructtheir understanding of the reading material. The initial test was using three different reading
comprehension material without any treatment to ensure that we get their basic score of
reading comprehension test. The experimental design used in the second week was carefully
conducted to ensure that we had a controlled environment to avoid unbalanced treatment.
The second reading material used of the experiment was using the underlined importantsentences so that the control group receive the same hint as the experimental group. The
design for the experiment is shown in Figure 3

Figure 3. Experimental design.

4.2 Measurement and Analysis
The scoring measurement used in this experiment was using a normal calculation score, eachcorrect answer will receive 1 point. The test material for each group for pre, post and delaytest are using the same material. Students never know their mistakes since the test result isnever exposed to them until the experiment is over. The test material consists of 10 multiple
choices. Each correct answer will be given 1 point and 0 point for incorrect. Total score is thesummary of correct answer. The Analysis was using ANOVA (Analysis of Variance).
Considering the students are having the same background, age, and the percentage of gender735

is almost the same where Male is 54% while female is 46%. Post hoc analysis using Holm'sSequentially Rejective Bonferroni Procedure was also conducted to explain the ANOVA result
in detail.

5. ResultThis section will analyze the reading comprehension score from each group.The first step ofanalysis is the mean analysis for each group for every test (Table 1). The result shows thatthe standard deviation is smaller than mean. It indicates that each data is clustered aroundeach mean. The second step is checking the Sphericity Test and Epsilons before conducting
the ANOVA calculation (Table 2). Based on the result, the sphericity is violated (p-value =0.0001; p < .001), it is occurred because the number of data sample is small. GreenhouseGeisser (GG), Huynh-Feldt-Lecoutre (HF) and Chi-Muller (CM) correction are used to correct
the violation of the assumption of sphericity, it shows that the variance of differences between
groups are equal (GG = 0.7816; HF = 0.8179; CM = 0.8148) and the ANOVA calculation canbe conducted.
Table 1. Mean Analysis for each group and test
GROUP
Control
Control
Control
Control
Experiment
Experiment
Experiment
Experiment

test
El.InitTest
El.PreTest
El.PostTest
El.DelayedTest
El.InitTest
El.PreTest
El.PostTest
El.DelayedTest

n
29
29
29
29
29
29
29
29

Mean
5.6897
6.10345.86215.79315.55175.96556.96556.8966

S.D.
1.62801.87741.76722.45501.8242
1.4264
1.59201.7596

Table 2. Mendoza's Multisample Sphericity Test and Epsilons
EffectLambdatest

0.0000Approx.
Chi
37.2606df

p-value

11

0.0001 ***

LB

GG

HF

0.3333 0.7816 0.8179

CM
0.8148

LB = lower.bound, GG = Greenhouse-Geisser HF = Huynh-Feldt-Lecoutre, CM = Chi-Muller
After checking the sphericity, ANOVA calculation was conducted, and the result canbe seen in Table 3. The result indicates that there is no significant difference found between
groups for all combined test scores (p-value = 0.2271). However, there are significant different
found for test within the groups (p-value = 0.0012; p < .01) and each test between groups (pvalue = 0.0013; p < .01). Based on the ANOVA calculation results, there are significant
differences found for the test and each group test. However, the detailed information cannot
be determined and needs to conduct a post hoc analysis.
Tabel 3. ANOVA calculation result for each group ant tests.
SourceSS

df

MS

F-ratiop-valueGROUP
S X GROUP

13.5172507.50001
56

13.5172
9.0625

1.49160.2271 ns

test
GROUP X testS X GROUP X test

22.758622.3448229.3966

3
3
168

7.58627.44831.36555.55585.45480.0012 **0.0013 **736

Total795.5172231

3.4438+p < .10, *p < .05, **p < .01, ***p < .001

Holm's Sequentially Rejective Bonferroni Procedure analysis for “tests” and “GROUP
X test” need to be conducted to show a better result. The alpha level used is 0.05. Firstly,mean analysis needs to be conducted to make sure that the data is clustered around the mean
(Table 4). The result shows that all the standard deviations are below the mean of each test.
Multiple Comparison for "test" then conducted (Table 5). It shows that the combination of posttest score is higher than the combination of initial test while other remain equal.
Table 4. Mean Analysis for test in groups.
test
El.InitTest
El.PreTest
El.PostTest
El.DelayedTest

N
58
58
58
58

Mean
5.6207
6.0345
6.41386.3448

S.D.
1.71501.65401.75752.1889

Table 5. MULTIPLE COMPARISON for "test"
Pair

Diff

t-value

Df

p

Adj.p

E1.InitTest-E1.PostTest-0.79313.560356

0.00080.0046

E1.InitTest < E1.PostTest *E1.InitTest-E1.DelayedTest-0.72412.603756

0.01180.0589

E1.InitTest = E1.DelayedTest

E1.PreTest-E1.PostTest-0.3793

2.340556

0.02280.0914

E1.PreTest = E1.PostTest

E1.InitTest-E1.PreTest-0.4138

2.021756

0.04800.1440E1.InitTest = E1.PreTest

E1.PreTest-E1.DelayedTest-0.3103

1.340656

0.18550.3709

E1.PreTest = E1.DelayedTest

E1.PostTest-E1.DelayedTest

0.06900.375256

0.70890.7089

E1.PostTest = E1.DelayedTest

Simple Effects for "GROUP x test" interaction is conducted to explain the interaction
between group and test comparisons. Mendoza's Multisample Sphericity Test and Epsilons
for GROUP X test was examined to check for any Sphericity violation (Table 6). The result
showed that there is a violation found due to the small number of samples for test at Control(p-value = 0.0025) and test at Experiment (p-value = 0.0069), however, correction have been
made by using GG, HF and CM. After the correction, simple effects for "GROUP x test"
interaction was conducted (Table 7). The result showed that: (1) No significant difference
found for the initial test between control and experiment group (p-value = 0.7624). (2) No
significant difference found for the pre-test between control and experiment group (p-value =0.7539). (3) Significance difference found for the post-test between control and experimentgroup (p-value = 0.0154; p < .05). (4) Significance difference found for the delayed test
between control and experiment group (p-value = 0.0541; p < .10). (5) No significant difference
found for all test in control group (p-value = 0.6465). (6) Significance difference found for all
test in experiment group (p-value = 0.0000; p < .001).
Table 6. Mendoza's Multisample Sphericity Test and Epsilons for GROUP X test
EffectLambdadf

p-value

0.0001

Approx.
Chi
18.3672

test at Control
test at Experiment

LB

GG

HF

CM

5

0.0025 **0.3333 0.7321 0.7970

0.78520.000215.989950.0069 **0.3333 0.7278 0.7917

0.7800

LB = lower.bound, GG = Greenhouse-Geisser HF = Huynh-Feldt-Lecoutre, CM = Chi-Muller
Table 7. Simple Effects for "GROUP x test" interaction

737

Source

SS

df

MS

F-ratiop-value

GROUP at E1.InitTest
0.2759Er at E1.InitTest 167.3793

156

0.27592.98890.09230.7624 ns

GROUP at E1.PreTest
0.2759Er at E1.PreTest 155.6552

156

0.27592.7796

0.09920.7539 ns

GROUP at E1.PostTest
17.6552
Er at E1.PostTest 158.4138

1
56

17.6552
2.82886.24120.0154 *GROUP at E1.DelayedTest
17.6552
Er at E1.DelayedTest 255.4483

1
56

17.6552
4.56163.87040.0541 +test at Control
2.6897s x test at Control 135.8103

3840.8966
1.61680.55450.6465 ns

384

14.1379
1.1141

12.6898

0.0000 ***test at Experiments x test at Experiment42.4138
93.5862+p < .10, *p < .05, **p < .01, ***p < .001
Multiple Comparison for "test at Experiment" using Holm's Sequentially Rejective
Bonferroni Procedure was conducted to break down the result found in the simple effect for"GROUP x test" (Table 8). The factor is analyzed as dependent means and the alpha level
used is 0.05. The result showed that: (1) initial test score is less than post-test score. (2) initialtest score is less than delayed test score. (3) pre-test score is less than post-test score. (4)pre-test score is less than delayed test score. (5) initial test score is equal to pre-test score.
(6) post-test score is equal to delayed test score.
Table 8. MULTIPLE COMPARISON for "test at Experiment" using Holm's Sequentially
Rejective Bonferroni Procedure
Pair

Diff

t-value

Df

p

Adj.p

E1.InitTest-E1.PostTest-1.4138

4.590128

0.00010.0046

E1.InitTest < E1.PostTest *E1.InitTest-E1.DelayedTest-1.34484.027128

0.00040.0589

E1.InitTest < E1.DelayedTest *E1.PreTest-E1.PostTest-1.0000

3.8079280.0007

0.0914

E1.PreTest < E1.PostTest *E1.InitTest-E1.PreTest-0.93103.087028

0.00450.1440E1.PreTest < E1.DelayedTest *E1.PreTest-E1.DelayedTest-0.4138

1.650628

0.11000.3709

E1.InitTest = E1.PreTest

E1.PostTest-E1.DelayedTest

0.06900.386428

0.70890.7021E1.PostTest = E1.DelayedTest

6. Discussion and Conclusion
Viat-map application allows students to build a kind of logical structure following the Toulmin
Arguments concept. This kind of action will also form a kind of cognitive structure in theirperception. By representing students with the visible logical structure in the application, they
will also be able to view their own cognitive structure. Viat-map application also directs thestudents by following the teacher’s understanding of the text. Closed-ended approached isbeneficial for directing the student to have the same level of understanding as the teacher andwill be easier for the teacher to compare the mistakes made by the students. The map
reconstruction approach has been implemented for the concept map(Hirashima, 2019) andseveral investigations have reported that it is useful to realize immediate and accurate
diagnosis and feedback (Andoko et al., 2020; Pailai et al., 2017)
Based on the result above, the main objective of this study is to find out the comparisonof students’ understanding and memory retention between students who use Viat-map andsummarization conventional method. Firstly, to measure student understanding, the test score738

was used as the representation of students understanding. Secondly, to measure memory
retention, the delay-test score was used to measure the students’ memorization.
The first aim is to measure students’ understanding by using pre-test and post-testscores. Based on the calculation in, the experiment group having an improvement from pretest to post-test (Table 8), while the control group remain the same (Table7) where test atcontrol found no significant difference (p-value = 0.6465). The comparison of post-test
between experiment and control group also shown that experiment group overachieved thecontrol group (Table 7; P-value = 0.0154; p < .05). The improvement from pre-test to post-test
in experiment group also indicates that there is an improvement of score before using theapplication (pre-test) compared to the score after using the application (post-test). The
experimental result indicated that the use of application helped the students to get a better
test score than one that didn’t use. Assisting students in learning process by providing therequire components to be used by the students could improve their understanding (Hirashima,
2019; Hirashima et al., 2015).
The second aim for memory retention was using the post-test and delay-test score.
Based on the result, both groups can get the same score between post-test and delay-testscore. However, considering that the delay-test score between control and experiment groups
are different, where experiment group overachieved the control group (Table 7; p-value =0.0541 +; p < .10) we can assume that the experiment group is better than the control groupfor memory retention. To show a better view of the overall test, a line chart of mean for eachtest in both groups is created (Figure 4). The differences of mean between groups for all tests
are clearly showed. Other perspective of the score distribution for each test in both groups,
box plot chart was also conducted (Figure 5). The initial test from both group shows that thescore distribution is a little unbalance where the control group is denser than the experimentgroup. However, the score distribution was changed for pre-test where they are almost having
the same distribution and means. A huge change occurred for post-test where means value
for experiment group is higher than control group, however the score distribution is similar.
Other changed is also occurred for delay-test where the distribution score for control group
become wider compared to their post-test even the means remain the same.
ŽŶƚƌŽů

ǆƉĞƌŝŵĞŶƚ

ϴ͕ϬϬϬϬ
ϳ͕ϬϬϬϬ
ϲ͕ϬϬϬϬ
ϱ͕ϬϬϬϬ
ϰ͕ϬϬϬϬ
ϯ͕ϬϬϬϬ
Ϯ͕ϬϬϬϬ

ϭ͕ϬϬϬϬ
Ϭ͕ϬϬϬϬ
>͘/E/dd^d>͘WZd^d>͘WK^dd^d>͘>zd^dFigure 4. Line chart for control and experiment group.

Figure 5. Box plot chart for control and experiment group.

739

In summary, constructing Toulmin arguments as logical structure offering students tounderstand the reading material in a different way. Using applications for study could also
assist them to enrich their learning process. This study found that students who used Viatmap application have a better understanding compared to one who is not using the application.
They also could retain their understanding 1 week after the lesson. However, there are several
limitations for this study. First, this study only involves students from one University and one
department that cannot represent the university students. Second, the number of sample size
is only fifty-eight students which is very small. This study requires more evidence and many
students to represent an ideal used of application in a classroom situation. For future research,
it is important to conduct a comprehensive log analysis to give a better explanation and new
points of view to improve the result of this study.

Acknowledgements
We would like to thank State Polytechnic of Malang for funding this research under Research
grant number: SP DIPA-023.18.2.677606/2023.

References
Allard, D., & Mizoguchi, R. (2021). Dr. Mosaik: a holistic framework for understanding theEnglish tense–aspect system based on ontology engineering. Research and Practice inTechnology Enhanced Learning, 16(1), 23. https://doi.org/10.1186/s41039-021-00163-x
Andoko, B. S., Hayashi, Y., Hirashima, T., & Asri, A. N. (2020). Improving English readingfor EFL readers with reviewing kit-build concept map. Research and Practice inTechnology Enhanced Learning, 15(1). https://doi.org/10.1186/s41039-020-00126-8
Andoko, B. S., Mubarok, F. U., Hirashima, T., Arhandi, P. P., Astiningrum, M., & Najib, M.
F. (2022). Constructing Toulmin’s Logical Structure Through Viat-map Application for
Reading Comprehension of EFL Students. 2022 Fifth International Conference on
Vocational
Education
andElectricalEngineering(ICVEE),
196–200.
https://doi.org/10.1109/ICVEE57061.2022.9930471
Carrell, P. L. (1989). Metacognitive Awareness and Second Language Reading. The Modern
Language Journal. https://doi.org/10.2307/326568
Dehham, S. H., Kadhim Bairmani, H., & Shreeb, A. (2022). JOURNAL OF LANGUAGE
AND LINGUISTIC STUDIES Developing Iraqi EFL preparatory students’ performancein reading comprehension by flipped learning strategy. In Journal of Language andLinguistic Studies (Vol. 18). www.jlls.org
Dole, J. A., Duffy, G. G., & Roehler, L. R. (1991). Moving from the Old to the New : Research
on Reading Comprehension Instruction Author ( s ): Janice A . Dole , Gerald G . Duffy ,Laura R . Roehler and P . David Pearson Published by : American Educational Research
Association Stable URL : http://www.jsto. 61(2), 239–264.
Eftekhari, M., & Sotoudehnama, E. (2018). Effectiveness of computer-assisted argument
mappingforcomprehension,
recall,andretention.
ReCALL.
https://doi.org/10.1017/S0958344017000337
Eftekhari, M., Sotoudehnama, E., & Marandi, S. S. (2016). Computer-aided argument mappingin an EFL setting: does technology precede traditional paper and pencil approach indeveloping critical thinking? Educational Technology Research and Development.
https://doi.org/10.1007/s11423-016-9431-z
Hirashima, T. (2019). Reconstructional concept map: automatic Assessment and reciprocalreconstruction. In International Journal of Innovation, Creativity and Change.
www.ijicc.net (Vol. 5, Issue 5). www.ijicc.net

740

Hirashima, T., Yamasaki, K., Fukuda, H., & Funaoi, H. (2015). Framework of kit-build concept
map for automatic diagnosis and its preliminary use. Research and Practice in Technology
Enhanced Learning, 10(1). https://doi.org/10.1186/s41039-015-0018-9
Hitchcock, D. (2005). Good Reasoning on the Toulmin Model. Argumentation, 19(3), 373–
391. https://doi.org/10.1007/s10503-005-4422-yJunaidah Januin, W. H. O. (2021). Analysing ESL Persuasive Essay Writing Using Toulmin’s
Model of Argument. Psychology and Education Journal, 58(1), 1810–1821.
https://doi.org/10.17762/pae.v58i1.1034
Magalhães, A. L. (2020). Teaching How to Develop an Argument Using the Toulmin Model.
International Journal of Multidisciplinary and Current Educational Research, 2(3), 1–7.
www.ijmcer.com
Pailai, J., Wunnasri, W., Yoshida, K., Hayashi, Y., & Hirashima, T. (2017). The practical useof Kit-Build concept map on formative assessment. Research and Practice in Technology
Enhanced Learning, 12(1), 20. https://doi.org/10.1186/s41039-017-0060-x
Rismanto, R., Nurul Asri, A., Satria Andoko, B., Hirashima, T., & Arifiandi Leonanta, A.
(2021). A Preliminary Study: Toulmin Arguments in English Reading Comprehension forEnglish as Foreign Language Students. Proceedings - IEIT 2021: 1st International
ConferenceonElectricalandInformation
Technology,
6–10.
https://doi.org/10.1109/IEIT53149.2021.9587398
Toulmin, S. (2003). The Uses of Argument (2nd ed.). Cambridge University Press.
https://doi.org/https://doi.org/10.1017/CBO9780511840005
Yang, R. (2022a). An empirical study of claims and qualifiers in ESL students’ argumentative
writing based on Toulmin model. Asian-Pacific Journal of Second and Foreign Language
Education, 7(1), 6. https://doi.org/10.1186/s40862-022-00133-w
Yang, R. (2022b). An empirical study on the scaffolding Chinese university students’ English
argumentative writing based on toulmin model. Heliyon, 8(12), e12199.
https://doi.org/10.1016/j.heliyon.2022.e12199
Zhang, S. (2011). English as a global language in Chinese context. Theory and Practice inLanguage Studies. https://doi.org/10.4304/tpls.1.2.167-176

741

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Effects of a Machine Learning-empowered
Chinese Character Handwriting Learning
Tool on Rectifying Legible Writing inYoung Children: A Pilot Study
Lung-Hsiang WONGa*, Guat Poh AWa, He SUNa, Ching-Chiuan YENb, Chor Guan TEOb,& Yun WENaa
National Institute of Education, Nanyang Technological University, Singaporeb
NUS-KEIO CUTE Center, National University of Singapore, Singapore*lhwong.acad@gmail.com
Abstract: The logographic nature of Chinese script is a major dissuading factor forlearning handwriting. The challenge is the complex psycholinguistic process behind
handwriting. Thus, we developed AI-Strokes, a Chinese handwriting learning tool thatassists teachers in facilitating students’ handwriting practice in various modalities, andprovides personalized feedback for the students. By leveraging a trainable Machine
Learning back-end framework, the tool diagnoses and scores students’ handwriting
errors. This paper reports a pilot study in a Singapore primary school with an early
prototype of AI-Strokes. Two classes of students went through AI-Strokes-based
Chinese handwriting lessons (the experimental group) and conventional lessons (thecontrol group) respectively. Pre- and post-tests were administered, and theirhandwriting processes were analyzed regarding errors in stroke orders, extra/missing
strokes, and errors in stroke directions. The results show that the experimental group
has yielded significantly better learning gains than the control group. It is posited thatthe personalized feedback of AI-Strokes has formed a feedback loop to support
students’ trial-and-error process in improving their handwriting skills. The multimodal
handwriting task design may have also fostered their orthographic awareness throughthe activation of alternative psycholinguistic pathways during their handwriting lessons.
Keywords: AI in education, Computer-assisted language learning, Learning ofChinese handwriting, machine learning

1. Introduction
The logographic nature of Chinese script is a major dissuading factor for novice learners inlearning handwriting. Chinese handwriting involves complex visual-perceptual-motoringprocessing (Haas & Rees, 2010). Acquiring such an integrated processing skill is cognitively
demanding for, in particular, young Chinese as a second language learners including mostyoung students in Singapore (Wong et al., 2011). Furthermore, in the context of Singapore
primary schools, hampered by big class sizes and limited lesson time, timely formative
feedback by the teachers during individual students’ learning process is hardly materialized.
This study seeks to address the challenges by developing AI-Strokes ᷉➏㾕㎷᷊, a
web-based Chinese handwriting learning tool that assists teachers in facilitating handwriting
lessons with various types of prompts, and providing personalized feedback to the students.
By leveraging a trainable Machine Learning back-end framework, AI-Strokes could
automatically diagnose and score handwriting errors of individual students.
This paper reports on a pilot study in a Singapore school with an early prototype of
AI-Strokes. The quasi-experimental study involved two Primary 2 classes who went through
AI-Strokes-based Chinese handwriting lessons (the experimental group) and conventional
lessons (the control group) respectively. Pre- and post-tests were administered to both742

groups, and their handwriting processes were analyzed in terms of errors in stroke orders,
extra/missing strokes, and errors in stroke directions. The research question is as follows,“Would the incorporation of AI-Strokes into the Chinese as a second language handwriting
lessons in lower primary school levels help reduce students’ Chinese handwriting errors, ascompared to conventional lessons without personalized ICT support?”

2. Literature Review
2.1 Chinese character handwriting
The Chinese scripts are a principled and rule-based system. Each Chinese charactercomprises one or more components spatially arranged with certain principles (Liang, 2004).
Each character is composed of strokes, the basic and smallest unit of a character without
any semantic connotation (Lam & McBride, 2018), with fixed, codified stroke orders for
identical components in different characters. Handwriting processes involve multifactorial
pathways that connect phonology, orthography, and semantics (Yin et al., 2005). The
process of writing a Chinese character can be summarized into three steps as follows,
Step 1: Retrieving the orthographic representation (or, the mental image) of apreviously learnt character from long-term memory and store in working memory –specifically, the speech sound of the character activates the phonological representation of
the character, which activates the lexical representation related by meaning via the lexical
semantic pathway (Lam & McBride, 2018).
Step 2: Processing of the mental image in working memory with the stroke orders.
Step 3: Actual handwriting actions which involve the following sensorimotor
performance components, all activated almost simultaneously (Klein, et al., 2011):- visual
perception, visual skills, fine motor, and visual motor.
There are at least two levels of objectives for learning to handwrite Chinesecharacters: “legible writing” and “orthographic retrieval”. “Legible writing” (Tsai et al., 2012)
focus on Steps 2 and 3 - to handwrite “legible” characters with correct strokes, strokeorders, spacing, etc. Typical learning designs are “copying tasks”, e.g., to display a fullcharacter for students to copy. Conversely, “orthographic retrieval” (Qu & Damian, 2015)
tackles Step 1 where students handwrite characters without visual references of “modelcharacters”. The activities could be, (a) written picture naming task (e.g., to show an imageof a tree, and the student writes 㭘); (b) dictation task (e.g., a sound clip utters, ᱑㪛᷍㪛㡙⭥㪛᱒, and the student writes 㪛); (c) translation task (for second language students; e.g., toshow the English word “flower” and the student translates it mentally and write ⿉).
This study focuses on assisting teachers in improving young students’ competencies
in performing Steps 1 and 2 of the Chinese character handwriting process with AI-Strokes.

2.2 Automation of diagnosis of students’ handwriting
Traditional technology has been used to facilitate the recognition and scoring of Chinese
handwriting (Hsiao et al., 2015) according to established systems and rules. Using neuralnetworks for recognition, compared to classical methods, a system may present anadvanced alternative to human-based scoring. The ability to learn and establish parameters
during the training process means no manual hardcoding is needed. This makes the system
scalable and sustainable, and resilient to changes in handwriting styles.
The use of machine learning for recognizing handwritten numbers and English
characters has been explored to a great extent, with early works on neural networks
recognizing numbers with an error rate of 0.7% (LeCun et al., 1998). The use of
Convolutional Neural Networks for recognizing handwritten English characters peaked in2011, when Ciresan and team achieved an error rate of just 0.27% (Ciresan et al., 2011),
making it comparable to human-like performance.

743

Diagnosis of Chinese handwriting, however, has not been explored in depth, withsome having moderate success at handwriting recognition (Bai et al., 2014; Zhang et al.,2019; Zou et al., 2019). These attempts are empowered by large datasets or corpuses ofwritten characters, whilst the system being developed performs machine learning on theprocess of the writing itself, thereby giving it more data to work with while doing diagnosis.
AI-Strokes can also point out the errors made during the writing process (e.g., wrong
directions of strokes), which the systems reported in the aforementioned publications cannot.

3. System Design of AI-Strokes
AI-Strokes is a web-based teaching aid that helps teachers manage and record the processof each student’s Chinese handwriting and provide personalized diagnosis on their strengths
and weaknesses, empowered by the advanced Machine Learning backend. With more
student handwriting data being collected, the system can self-learn to analyze and score
handwritten characters that it has not been trained with. The web-based tool should beaccessed by touchscreen devices for a more natural handwriting experience. The systemdesign is intended for use in classroom lessons (e.g., learning new characters in the
textbook). The system user interface (UI) comprises teacher’s console and students’ client.
Before each lesson, the teacher creates a new lesson plan with the teacher’s console
UI via a laptop or a tablet computer by specifying the characters to practice. At the lesson,
the teacher selects the pre-stored lesson plan and launches it, with the teacher’s console UI
being projected to the screen. Each student who is assigned a tablet may then log in andjoin the lesson with the student UI. Every time when the teacher administers a character forthe entire class to practice, (s)he may provide the prompt through teacher’s console in oneof the following modalities: (1) the character itself (copying task); (2) an image or an
animation that depicts the meaning of the character (picture naming task); (3) the equivalentEnglish word (translation task); (4) a sound clip that pronounces the character withdisambiguation (dictation task). The copying task corresponds to “legible writing” while thethree other tasks correspond to “orthographic awareness” (see section 2.1).
In both modes, students are given immediate feedback (Figure 1) upon submission of
their handwriting via the automated scoring system powered by the Machine Learning
backend. The Machine Learning backend scores each student’s writing with how well it
matches the correct answer in terms of number of strokes, order of strokes, directions ofstrokes, and speed. The aspects of handwriting to be diagnosed is a subset of the validated
Tseng’s Handwriting Problem Checklist (THPC) (Tseng, 1993). According to THPC, there are
24 common Chinese character handwriting problems among young students. Only 16 of them
may be automatically diagnosed by typical AI-empowered web-based software. For this early
prototype of AI-Strokes, automated diagnoses on 8 of the problems are implemented, lumped
into 4 categories as listed in the score card in Figure 1. In this early prototype, a submitted
handwritten character may be awarded a star for each of the achievements of correct numberof strokes, order of strokes and directions of strokes. Thus, the full score is 3 stars.
In teacher’s console, there is an additional classroom view. After administering acharacter for the students to practice, the teacher may switch to the classroom view with all
students’ writing are displayed in real-time. The teacher may get a glimpse of how each
student has done, or project it on the screen for class wide comparison and discussion.

4. Research Design of the School-based Pilot Study
Fifty (50) Primary 2 students from a public school in Singapore participated in the study,
including 28 from the experimental group/class and 22 from the control group/class. The
experiment comprised four one-hour lessons that spanned across 1.5 months. During theexperiment, the teachers of both classes tapped on the regular Chinese lessons to conduct
handwriting activities with the same sets of characters. However, AI-Strokes was employedin the experimental class while the prevailing handwriting instructions were carried out in the744control class. In the control class, the teacher first demonstrated writing a character on thewhiteboard. The students then practiced writing individual characters in the air with fingers oron their mini whiteboards and displayed them to the teacher. Eventually, all students were
drilled in handwriting in their exercise books by copying the given characters or by dictation.

Student ID

x

x
x
x

Practice

Number of
strokes
Stroke orders
Strokedirections
Writing speed

Try again
Figure 1. The student’s UI of AI-Strokes with the handwriting field on the left and the scorecard on the right
The pre-test and the post-test took place before the first lesson and after the final
lesson respectively and were administered as copying tasks. At each test, students used AIStrokes to handwrite 10 characters – 䒄炻Ṛ炻⒕炻㰌炻傥炻䇙炻崟炻䪼炻䛨炻㜩. The
system captured the students’ writing processes (stroke orders) and products (the final
appearance of each character) for analysis. The characters were selected from 30
characters covered in the four lessons with a maximum variety in the character components.
Stroke-by-stroke images of the writing process of each character were captured by
the system for analysis. The process was assessed in: (a) Errors in stroke orders (“s-orders”
hereafter): The number of strokes written in wrong orders; (b) Number of extra/omitted
strokes (“eo-strokes” hereafter): The sum of the absolute number of extra strokes and theabsolute number of omitted strokes ȸ including cases where the student breaks one stroke
into two or more strokes (e.g., writing the third stroke of 䒄 in two strokes ȸ counted as 1
extra stroke), or write two or more strokes in one stroke (e.g., writing the three-stroke
component ⎋ in one stroke or two strokes – counted as 2 or 1 omitted stroke(s)); (c) Errors
in stroke directions (“s-dir” hereafter) (e.g., writing 㧒 (a horizontal stroke from left to right炸
as 㐯 (a titled stroke from right to lower left) is considered an error). Thus, for all three
indicators, the lower the numbers are, the better the student’s performance is.

5. Findings
We employed SPSS 28 to analyze the pre- and post-test data (Table 1). To determine
whether parametric or non-parametric Analysis of Covariance (ANCOVA) should be
performed on the three indicators respectively (Corder & Foreman, 2009), the normal
distribution tests of Shapiro-Wilk were executed. The results showed that there were normal
distributions in both the pre- and post-test scores of s-orders (p=.324>.05 and p=.258>.05
respectively), while there was normal distribution in the pre-test score of eo-strokes(p=.153>.05) but no normal distribution in the post-test score of the same indicator (p<.05),
and there were no normal distribution in both the pre- and post-test scores of s-dir (p<.001
and p<.001 respectively). Therefore, parametric ANCOVA was performed to compare thepre- and post-test scores of s-orders between the two groups, while Quade’s non-parametric
ANCOVA was executed to compare the eo-strokes and s-dir scores respectively.
The results showed that the handwriting performance of the experimental group was
improved significantly as compared to the control group. By comparing the pre- and post-test

745

means of the control group, there were even dips in their performance in s-orders and eostrokes. Notwithstanding, the dips are insignificant, according to our supplementary paired
samples t-tests (s-orders: p=.079>.05; eo-strokes: p=.353>.05; s-dir: p=.164>.05). On thecontrary, the experimental group exhibited significant improvements in all indicators, as seenin the results of paired-sample t-tests (s-orders: p<.001; eo-strokes: p<.001; s-dir: p=<.05).
Table 1. Summary of descriptive statistics and results of parametric or non-parametric
ANCOVA on the three indicators of students’ handwriting performance⸶IndicatorGroups-orders(parametric ANCOVA) ⸶
eo-strokes(Quade’s ANCOVA)
s-dir(Quade’s ANCOVA)

Experimental
Control
Experimental
Control
Experimental
Control

Pre-test
Mean (SD)77.43 (4.23)
73.86 (3.91)
3.96 (2.43)
5.55 (3.14)
1.75 (1.84)
2.14 (2.05)

Post-test Mean(SD)
70.93 (4.94)
75.77 (3.89)
1.96 (2.73)
5.86 (1.71)
0.89 (1.69)
1.55 (1.54)

Fp

11.447.001

28.587

.000

8.812

.005

Levene’s Test of Equality of Error Variances: F=.252, p=.618>.05.

High error rates of stroke orders in both tests and across both groups are observed in
the statistics. The correct total number of strokes of the 10 tested characters is 88. Thus, theerror rates according to the four mean scores of s-orders is ranging from 80.5% to 88.0%.
The roots of the phenomenon can probably be found in the student demography and theteachers’ instructional approaches, which will be discussed in the next section.

6. Discussion and Conclusion
This study found that the experimental group which experienced the AI-Strokes-empowered
Chinese handwriting lessons has yielded significantly better learning gains in the threemeasured indicators, as compared to the control group. It is posited that the personalized
feedback feature of AI-Strokes has formed a feedback loop to trigger and support students’
trial-and-error process to improve their handwriting skills. The multimodal handwriting tasks
have also played the role of fostering students’ orthographic awareness through activation ofalternative cognitive and psycholinguistic pathways during their handwriting lessons.
Yet why were there high error rates in stroke orders even at the post-test? Scholars(Lam & McBride, 2018; Law et al., 1998) reiterated the need to teach general stroke-orderrules. When students have internalized these rules, they could handwrite any novel
character with correct order. However, during the limited lesson time, teachers tend to focus
on demonstrating stroke orders of individual characters, and only introduce scattered strokeorder rules by chance. Students might ignore these rules but just memorize the stroke orderof each character. In paper-based assignments, the correctness of stroke orders cannot be
assessed as only the completed handwriting is submitted for grading. Thus, most students
are not motivated to learn correct stroke orders as long as their “final products” “look right”.
In our study, however, the experimental group students have constantly received
feedback from AI-Strokes after handwriting each character. If the feedback indicated that
their stroke order was not correct, they might try to figure out what went wrong by recalling
what the teacher had taught before or asking around their classmates, with the hope ofscoring higher in their next writing attempt. Thus, we postulate that AI-Strokes had subtly
influenced the experimental group students to start paying attention to the stroke orders.
Despite the significant improvement in stroke orders among the experimental groupstudents, their error rates in the post-test are still relatively high. By comparing their strokeorders between the pre- and post-tests, it is observed that at the post-test they were more
likely to handwrite the characters with partially correct stroke orders. Yet most of the timethey did not manage to handwrite whole characters with perfectly correct stroke orders. The
reasons are two-folded. First, the prevailing issue of the lack of systematic explicit teachingof stroke order rules would have limited the students’ development of stroke order746

“competency”. Second, the feedback given by the current AI-Strokes prototype is relatively
coarse-grained and is unable to pinpoint the sources of errors; but just alerts the students
that “something was wrong” and they need to find out the errors through other means.
To address these limitations, we advocate a holistic revamp in AI-Strokes-based
lessons. In the instructional aspect, the teachers should set aside the time to systematicallycover the general stroke order rules. When a new rule is introduced, the teachers should let
students practice writing multiple characters where this rule is applicable (rather thaninsisting to practice the characters appearing on the current textbook passage) on AIStrokes, to help them internalize the rule and later apply it to writing novel characters withthe same “stroke configuration”. In the technical aspect, AI-Strokes should be upgraded tooffer finer-grained feedback. AI-Strokes may play a crucial role in providing personalized
formative assessment of handwriting, which is unable to be performed by the teachers.
Given its current machine learning backbone engine, AI-Strokes has the potential tobe further upgraded to incorporate the capability of diagnosing additional aspects of THPC
including spacing, spatial relationship, size and formation. More young children’s handwriting
samples need to be collected as training data for this purpose. With an upgraded AI-Strokes
system, a longer-term study can be carried out with a full-fledged Chinese handwriting
lessons to develop a stronger psycholinguistic and handwriting foundations in the students.

References
Bai, J., Chen, Z., Feng, B., & Xu, B. (2014). Chinese image character recognition using DNN andmachine simulated training samples. International Conference on Artificial Neural Networks,
Hamburg, Germany.
Ciresan, D. C., Meier, U., Gambardella, L. M., & Schmidhuber, J. (2011). Convolutional neuralnetwork committees for handwritten character classification. 2011 International Conference
on Document Analysis and Recognition, Beijing, China.
Haas, B., & Rees, G. (2010). Multiple stages of cross-modal integration in visual processing:
Comment on crossmodal influences on visual perception: By L. Sham & R. Kim. Physics ofLife Reviews, 7, 287-288.
Hsiao, H.-S., Chang, C.-S., Chen, C.-J., Wu, C.-H., & Lin, C.-Y. (2015). The influence of Chinesecharacter handwriting diagnosis and remedial instruction system on learners of Chinese as aforeign language. Computer Assisted Language Learning, 28(4), 306-324.
Lam, S. S.-Y., & McBride, C. (2018). Learning to write: The role of handwriting for Chinese spelling inkindergarten children. Journal of Educational Psychology, 110(7), 917-930.
Law, N., Ki, W. W., Chung, A. L. S., Ko, P. Y., & Lam, H. C. (1998). Children’s stroke sequence errors
in writing Chinese characters. In C. K. Leong & K. Tamaoka (Eds.), Cognitive processing of
the Chinese and the Japanese languages. Neuropsychology and cognition (pp. 113-138).
Springer.
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 86(11), 2278 - 2324.
Liang, Y. (2004). The characteristics of Chinese character components and Chinese as a Foreign
Language teaching. Language Education and Research, 6, 76-80.
Qu, Q., & Damian, M. F. (2015). Cascadedness in Chinese written word production. Frontiers in
Psychology, 6(1271), 1-11.
Tseng, M.-H. (1993). Factorial validity of the Tseng Handwriting Problem Checklist. Journal of
Occupational Therapy Association Republic of China, 11, 13-28.
Wong, L.-H., Boticki, I., Sun, J., & Looi, C.-K. (2011). Improving the scaffolds of a mobile-assisted
Chinese character forming game via a design-based research cycle. Computers in Human
Behavior, 27(5), 1783-1793.
Yin, W., He, S., & Weekes, B. S. (2005). Acquired dyslexia and dysgraphia in Chinese. Behavioural
Neurology, 16, 159-167.
Zhang, X. Y., Wu, Y. C., Yin, F., & L., L. C. (2019). Deep learning based handwritten Chinesecharacter and text recognition. In H. K., H. A., W. Q. F., & Z. R. (Eds.), Deep Learning:
Fundamentals, Theory and Applications (pp. 57-88).
Zou, J. Y., Zhang, J. L., & Wang, L. (2019). Handwritten Chinese Character Recognition by
Convolutional Neural Network and Similarity Ranking. arXiv preprint.
https://arxiv.org/abs/1908.11550

747

Shih, J. L. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in Education.
Asia-Pacific Society for Computers in Education

Using TAASSC to Investigate Fine-Grained
Grammatical Complexity in Reading Texts
of Two High-Stakes English Tests in China
Shengshu LIN
School of Foreign Languages, Nanjing University of Posts and Telecommunications, China
shengshu2016@hotmail.com
Abstract: Fine-grained grammatical complexity measures are better predictors thanlarge-grained indices of register variation and writing of different levels in that they
provide in-depth explanation of what accounts for syntactic complexity. Computer
technologies have made it possible to study these fine-grained measures based onlarge corpora. Using the Tool for Automatic Analysis of Syntactic Sophistication andComplexity (TAASSC) (Kyle, 2016), the present study investigates phrasal and clausal
complexity in reading texts of two high-stakes English tests in China (CET-4 and
NETEM) different in difficulty level. The results showed that clausal and phrasal
complexity indices could predict the two tests. More importantly, in combined analysisof the two types of measures, the results demonstrated that noun phrases, particularly
those with phrasal modifiers, were better predictors of the more difficult test NETEM.
These findings support the importance of noun phrases and the diverse modifiers theytake in informational texts. Findings of this research will help college students advancetheir English learning with a specific purpose and in a right direction.
Keywords: clausal complexity, phrasal complexity, reading texts, tests

1. Introduction
Reading comprehension has always been an important task in time-limited English tests. Yet
grammatical complexity is given scant attention in reading accuracy despite its wide attention
in writing. Grammatical complexity in spoken and written English and its historical changes
were fully documented in Biber and Gray (2016). Recent research showed that fine-grained
grammatical complexity measures were more effective in accounting for sentence length andwriting quality (Kyle & Crossley, 2018; Zhang & Lu, 2022; Zhang & Liu, 2022; Gray, Geluso, &Nguyen, 2019). According to Biber, Gray and Poonpon (2011), advanced phrasal features are
not acquired naturally and not always successfully acquired. The empirical research by Priven(2020) showed that fine-grained phrasal complexity pervasive in university textbooks caused
difficulty to ESL readers. In Bychkovska (2021), the improved quality of writing by ESL writers
after explicit teaching of phrasal features also lent support to the importance of fine-grainedcomplexity features in reading. Due to the washback effects of tests, reading texts of a test area critical source of language input for learners. For example, Chinese learners in preparation
for NETEM (the National Entrance Test of English for MA/MS) use past years’ real tests as
major learning materials. The present study is conducted with the purpose of shedding light on
the fine-grained linguistic features in reading texts of two important tests in China and helping
test takers advance their learning in a right direction.

2. Research Background
2.1 Fine-Grained Measures

748

Measures of grammatical complexity have undergone dramatic changes with mounting
attention to fine-grained measures because they can account for what contributes to thelength of length-/ratio-based large-grained measures.
In this line of research, Biber and colleagues have always employed form-function
framework to define linguistic features. They believe only by taking structural types and theirgrammatical functions into consideration can register variations be fully described, because
the same structural type may have different functions in different registers. By now, the eightform-function categories have included over 130 specific linguistic features and proved fruitful.
Register studies showed that academic discourse was complex in the dense use of phrasal
features as nominal modifiers, while spoken conversation was clausally complex. Language
learners progress along a sequence of linguistic features acquiring grammar in conversation
first and that of writing later (Biber et al., 2011). Their theory was supported by many
researchers (Staples et al., 2016; Parkinson & Musgrave, 2014; Gray et al., 2019).
Although fine-grained measures based on form-function framework are easy tounderstand, the tools used to tag and count linguistic features are not publicly available. From
the usage-based theory of language acquisition, Kyle (2016) developed the Tool for theAutomatic Analysis of Syntactic Sophistication and Complexity (TAASSC) by utilizing Stanford
Neural Network Dependency Parser (version 3.5.1), making it possible to analyze a large
number of fine-grained linguistic units. Research conducted with TAASSC showed that
fine-grained measures were better predictors than large-grained measures of learner writing(Kyle & Crossley, 2018; Zhang & Liu, 2022; Zhang & Lu, 2022). Kyle and Crossley (2018) was
the first research that directly compared the predictive power of large-grained indices(calculated by L2SCA), clausal and phrasal complexity in TOEFL independent writing by L2
writers from different language backgrounds. Large-grained indices explained 5.8% ofvariance in holistic essay scores while fine-grained clausal complexity explained 3.0% andfine-grained phrasal complexity explained 18.9%. The large predictive power was supported
by recent studies (Zhang & Liu, 2022; Zhang & Lu, 2022). Zhang and Liu (2022) examined
Chinese university students’ English expository writing. Their results showed that fine-grained
indices accounted for 22.2% of the variance in students’ writing quality, significantly higher
than 7.2% by large-grained indices. Zhang and Lu (2022) investigated grammatical complexity
in application letters and argumentative essays by Chinese students. Traditional indicesexplained 20.2% of the variance in letter scores, and 15.7% of the variance in essay scores
while fined-grained measures explained 31.9% and 30.6% respectively.
The research aforementioned focused on the predictive power of fine-grained
measures in writing while investigation of reading texts in tests has been ignored by now.
Some fined-grained measures such as (multi-)noun as premodifiers and postmodifying
prepositional phrases are compressed with dense information, and may cause confusion inreading (Priven, 2020). Therefore, utilizing TAASSC, the present study makes contributions tothis area of research by investigating the reading texts of two important English tests in China.

2.2 CET-4 and NETEM in China
College English Test Band 4 (CET-4) as a proficiency test for university students in China is
influential among college students in China. In the third year or even earlier, millions of college
students will start to prepare for NETEM held in December annually to pursue postgraduate
studies. Reading tasks common in CET-4 and NETEM require test takers to answer fivemultiple-choice questions after reading a passage. Although reading texts in these two tests
are extracts from some publications in English-speaking countries including The Economist,
The Guardian or other academic reading materials, CET-4 is easier for its simpler vocabulary,shorter sentence length and text length. Since NETEM is regarded as the most challenging
test for non-English majors, some publications encourage test takers to use complex clauses
to obtain higher scores without mentioning phrasal features (Zhou, 2022). The status quo
reflects the stereotype of academic writing as clausally complex and clausal elaboration as afocus in most EAP teaching (Parkinson, 2020). To reveal grammatical complexity in thereading texts of the two tests, the present study addresses three research questions:749

RQ1. What clausal complexity measures can predict CET-4 texts and NETEM texts?
RQ2. What phrasal complexity measures can predict CET-4 texts and NETEM texts?
RQ3. What clausal and phrasal complexity measures when considered together can
predict CET-4 texts and NETEM texts?

3. Methodology
3.1 Corpus
Two corpora (NETEM and CET-4) used in this study were built by the author. NETEM is held
annually with four reading passages in each real test paper澳of the previous 13 years from 2011
to 2023. 52 texts were collected with a total of 23,023 words. CET-4 reading texts were moreeasily available because it is administered twice a year with 6 different reading passages eachtime. In order to control the effect of time on CET-4 language use, we randomly selected four
texts in each year to ensure that CET-4 reading passages are evenly distributed from 2010 to2022. 52 CET-4 texts totaled 18,411 words. All texts were saved in a .txt format with thefilename “year_test_text_number” that can be processed by TAASSC.

3.2 Tools and Procedures
The current study uses TAASSC (Kyle, 2016; Kyle & Crossley, 2018) to investigate clausal
and phrasal complexity. From usage-based theory perspective, the clausal complexity in
TAASSC refers to a main verb and its associated structures. TAASSC creates a dependency
representation of each sentence, that is, the functional categories and their grammatical
relations. It includes 31 clausal indices, of which 2 are general indices of clausal complexity
and 29 calculate the average number of particular units per clause. The phrasal complexity
involves 7 noun phrase types and 10 dependent types. Phrasal measures produced by
TAASSC have pronoun version and no-pronoun version, each including 66 indices. This study
used no-pronoun version because pronoun-headed phrases are not common (Lan et al.,2019). Of the 66 phrasal indices, 16 calculate the average number of dependents per eachphrase type and for all phrase types, and their standard deviations (SDs). 10 indices calculate
the occurrence of particular dependent types regardless of the type of noun phrases theyoccur in. 40 indices calculates the average occurrence of particular dependent types inparticular types of noun phrases. A higher value indicates a higher frequency of a linguistic
unit. SDs are used to measure syntactic variation.
These 97 complexity indices collected by TAASSC were entered in SPSS as
independent variables. Since rareness of some linguistic structures is inherent in fine-grained
analysis (Kyle & Crossly, 2018), we removed those that displayed “0” value. Due to nonnormal
distributions of the variables, non-parametric independent samples Mann-Whitney U test was
performed to identify the indices on which CET-4 and NETEM differed significantly (p < 0.5).
This step identified the independent variables that were used for binary logistic regression.
Binary logistic regression was performed on the significantly different clausal complexity
indices and phrasal complexity indices respectively to determine which were predictors ofreading texts of the two tests. The results from these two steps answered RQ1 and RQ2. To
answer RQ3, a combined analysis of the significantly different clausal and phrasal indices was
conducted to determine which indices entered the final regression model.

4. Results and Discussion
Table1, Table 2 and Table 3 include the variables that entered into binary logistic regression.
The variables with asterisks were predictors of texts in the two tests at the statistical level of750

0.05. Exp(B) values more than 1 indicated the complexity indices had predictive effects on
texts in NETEM, while Exp(B) values less than one predicted CET-4 texts.
Table 1. Logistic Regression of Clausal Complexity
澳
1
2
3
4
5
澳

澳
Adjective complements per clause
Conjunctions per clauseDirect objects per clause
Bare noun phrase temporal modifiers per clause
Auxiliaries per clause
Constant

B-3.6526.568
4.534
48.906
7.367-4.215

S.E.
5.262
4.955
2.592
19.923
3.322
1.549

Wald
0.482
1.757
3.060
6.026
4.919
7.401

Sig.
0.488
0.185
0.080
0.014*
0.027*
0.007

Exp(B)
0.026
712.14393.152
1.736
1583.482
0.015

Table 2. Logistic Regression of Phrasal Complexity
1澳
2
3
4
5
6
7
8
澳

澳
Dependents per nominal
Dependents per object of the preposition
Dependents per nominal subjects (SD)
Dependents per passive nominal subject (SD)
Dependents per agent(SD)Verbal modifiers per nominal
Nouns as nominal dependents per nominal
Possessives per direct object
Constant

B-1.645
3.254
3.965
0.678
1.191
-31.467
4.992
8.531-7.178

S.E.
2.685
2.047
1.572
0.547
0.649
13.879
4.275
3.531
2.487

Wald
0.375
2.527
6.363
1.538
3.369
5.141
1.364
5.839
8.329

Sig.
0.540
0.112
0.012*
0.215
0.066
0.023*
0.243
0.016*
0.004

Exp(B)
0.193
25.88252.720
1.970
3.290
0.000
147.2875071.844
0.01

Table 3. Logistic Regression of Clausal Complexity and Phrasal Complexity Combined
澳
1
2
3
4
5
6
7
8
9
10
11
12
13
澳

Independent variables 澳
Dependents per nominal
Dependents per object of the preposition
Dependents per nominal subject (SD)
Dependents per passive nominal subject (SD)
Dependents per agent (SD)
Verbal modifiers per nominal
Nouns as nominal dependents per nominal
Possessives per direct object
Adjective complements per clause
Conjunctions per clauseDirect objects per clause
Bare noun phrase temporal modifiers per clause
Auxiliary per clause
Constant

B-3.475
4.801
4.775
0.544
1.724
-37.935
1.801
10.437-2.666
3.663
5.389
78.275
3.541-10.874

S.E.
3.233
2.432
1.795
0.672
0.811
16.829
4.518
4.230
6.630
6.374
3.179
28.267
4.187
3.435

Wald
1.155
3.896
7.074
0.655
4.522
5.081
0.159
6.089
0.162
0.330
2.873
7.668
0.715

Sig.
0.282
0.048*
0.008*
0.418
0.033*
0.024*
0.690
0.014*
0.688
0.565
0.090
0.006*
0.398

Exp(B)
0.031
121.638118.500
1.724
5.607
0.000
6.054
34083.305
0.070
38.992218.883
9.873
34.494

To answer RQ1, Table 1 shows that bare noun phrase temporal modifiers andauxiliaries are more common in NETEM texts, and the more use of these features indicatesthe higher probability that reading texts are from NETME. Many of NETME texts are extracted
from top newspapers, where temporal words are important linguistic resources in narrating
events. The ability to capture the temporal dimension is essential for understanding news
events. Auxiliaries can convey much grammatical information such as aspect, tense, voice
and modality. More importantly, modal auxiliaries express a variety of meanings related toconcepts such as ability, permission, necessity, and obligation (Biber et al., 1999). It is very
common in NETEM reading comprehension that questions are asked about attitudes of thenews writer or people involved in news events. NETME is more complicated than CET-4 in theuse of temporal words to connect events and auxiliaries to express grammatical information
and attitudes, though both tests use news extracts as a major source of reading texts.

751

With regard to RQ2, Table 2 shows that NETEM employs diverse modifying structuresin noun phrases as a subject, as is illustrated by Excerpt 1. Excerpt 1 includes adjective“personal”, (multi-)noun “grievance” and “Employment Relations”, of-preposition “of New
Zealand’s...2000”, and possessive “New Zealand’s”.
Excerpt 1: The personal grievance provisions of New Zealand’s Employment Relations
Act 2000 (ERA) prevent an employer from firing an employee without good cause.
(2022_NETEM_text_4)
A subject with various types of modifiers are quite common in NETEM reading texts
because the extended subject packages dense information into phrasal premodifiers. Anotherindex that makes reading texts more likely to be NETEM texts is the use of more possessives
modifying a noun in direct object positions. Possessives, nouns and adjectives are the mostcompressed modifiers in noun phrases (Biber & Gray, 2016). However, using possessives,
especially genitive ’s is not encouraged among Chinese students because it may strengthen
the stereotype of “Chinese English” since its counterpart structure in Chinese is prevalent. In
this sense, our findings counter this stereotype and highlight the importance of these featuresin reading. According to Biber and Gray (2016), the use of verbal modifiers is viewed as morestructurally elaborate rather than compressed, and is more common in spoken conversation.
This is supported by our finding that more use of verbal modifiers of a noun will increase thelikelihood of reading being CET-4 texts. Instead, phrasal modifiers are a common feature inpassages with information focus (Biber & Gray, 2016).
Results in Table 3 provide answers to RQ3. As can be seen in Table 3, when clausal
and phrasal complexity were combined, two new phrasal indices entered the regression
model: dependents per object of prepositional phrases and dependents per agent (SD), while
only bare noun phrase temporal modifiers per clause was retained as clausal index. Modifying
devices of agents are also more diverse in NETEM reading. The index of dependents of
prepositional objects further affirmed that NETEM were more likely to use extended nounphrases with embedded modification as illustrated by Excerpt 2. The prepositional phrase “ofhuman activity” also takes an embedded modifier “in recent climate change”, which again
takes an adjective “recent” and a noun “change” as its modifiers.
Excerpt 2: These would emphasise the primacy of human activity in recent climatechange and encourage discussion of mitigation measures. (2023_NETEM_text_1)
Based on Biber et al.’s (2011) language development theory, extended prepositional
phrases as post-modification are one of the most advanced features of academic language,
and also the most difficult for both native and EFL language learners. NETEM is more likely touse diverse dependents to add information in compressed structures. According to Kyle andCrossley (2018), Zhang and Lu (2022), phrasal complexity is more predictive of higher level
learner writing. This study further affirmed that, in English news, fine-grained indices, phrasal
level features in particular, correlated significantly with the degree of difficulty and revealed
what specific patterns contributed to difficulty of textual sentences in NETME. It is noteworthy
that all the indices included in the regression model of combined analysis were realized
linguistically by noun phrases that may take a variety of modifiers. This lends support to Biber
and Gray’s (2016) claim about noun phrases that they are prevalent in information writing,especially those with phrasal modifiers. In view of empirical evidence that extended nounphrases may cause difficulty in reading university textbooks to L2 learners (Priven, 2020),
noticing and understanding complex nominal modifiers should be emphasized in teachingadvanced English learners. For those who prepare for NETEM or plan their learning after
passing CET-4, using these compressed phrasal features to convey dense amounts of
information should be a central area of learning.

5. Conclusion
Utilizing TAASSC, the present study provides insights into what fine-grained measures are
predictive of reading texts of two important tests for university students in China. Findings will
help millions of students focus on the linguistic features that are more typical of NETEM rather

752

than just spending time on clausal subordination. Future research can include vocabulary as apredictor variable, comprehensively looking at what vocabulary characteristics and linguisticfeatures can predict reading texts in tests for different purposes and at different stages ofEnglish learning.

Acknowledgements
This study was supported by the 11th China Foreign Language Education Fund(ZGWYJYJJ11Z025) and Research on Philosophy and Social Sciences of Jiangsu
Universities in 2022 (2022SJYB0101).

References
Biber, D., & Gray, B. (2016). Grammatical Complexity in Academic English: Linguistic Change inWriting. Cambridge University Press.
Biber, D., Gray, B., & Poonpon, K. (2011). Should we use characteristics of conversation to measuregrammatical complexity in L2 writing development? TESOL Quarterly, 45(1), 5-35.
Biber, D., Johansson, S., Leech, G., & Finegan, E. (1999). Longman Grammar of Spoken and Written
English. Pearson Education Limited.
Bychkovska, T. (2021). Effects of explicit instruction on noun phrase production in L2 undergraduate
writing. Journal of English for Academic Purpose, 54,1-16.
Gray, B., Geluso, J., & Nguyen, P. (2019). The longitudinal development of grammatical complexity at
the phrasal and clausal levels in spoken and written responses to the TOEFL iBT® test. ETS
Research Report Series, (1), 1-51.
Kyle, K. (2016). Measuring Syntactic Development in L2 Writing: Fine-grained Indices of Syntactic
Complexity and Usage-based Indices of Syntactic Sophistication. Unpublished doctoral
dissertation. Georgia State University, America.
Kyle, K., & Crossley, S. A. (2018). Measuring syntactic complexity in L2 writing using fine-grained
clausal and phrasal indices. The Modern Language Journal, 102(2), 333-349.
Lan, G., Lucas, K., & Sun., Y. (2019). Does L2 writing proficiency influence noun phrase complexity? A
case analysis of argumentative essays written by Chinese students in a first-year composition
course. System, 85, 1-13.
Parkinson, J. (2020). Book review. Journal of English for Academic Purposes, 45,1-2.
Parkinson, J., & Musgrave, J. (2014). Development of noun phrase complexity in the writing of English
for Academic Purposes students. Journal of English for Academic Purposes, 14, 48-59.
Priven, D. (2020). “All These Nouns Together Just Don’t Make Sense!”: An Investigation of EAP
students’ challenges with complex Noun Phrases in First-Year college-level textbooks. Canadian
Journal of Applied Linguistics, 23(1), 93-116.
Staples, S., Egbert, J., Biber, D., & Gray, B. (2016). Academic writing development at the university
level: Phrasal and clausal complexity across level of study, discipline, and genre. Written
Communication, 33(2), 140-183.
Zhang, X., & Liu, W. (2022). Predictive effects of syntactic complexity on the quality of Chinese
university students’ English expository writing. Modern Foreign Languages, 45(3), 331-343.
Zhang, X., & Lu, X. (2022). Revisiting the predictive power of traditional vs. fine-grained syntactic
complexity indices for L2 writing quality: Then case of two genres. Assessing Writing, 51, 1-14.
Zhou, S. (2022). An Interpretation of the Syllabus for National Entrance Test of English (1) for MA/MS.
Beijing: Higher Education Press.

753

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Using Multidimensional Analysis toInvestigate the Extrapolation Inference of aHigh-Stakes Test
Shengshu Lin
School of Foreign Languages, Nanjing University of Posts and Telecommunications, China
shengshu2016@hotmail.com
Abstract: The degree of correspondence of situational characteristics between test
tasks and target domains is believed to affect how similar language use on tests is to
the target domains (i.e., extrapolation). Multidimensional analysis (MDA) has gained in
popularity in extrapolation research because it not only interprets language usefunctionally associated with situational characteristics but also provides a quantitative
method to investigate co-occurring linguistic features. Using the automatic toolMultidimensional Analysis Tagger (MAT), the present paper applies MDA to explore
the linguistic features between writing of the National Entrance Test of English for
MA/MS (NETEM) and disciplinary writing on four dimensions in Biber’s (1988) seminal
work. Overall, NETEM writing differed significantly from all four disciplinary domains on
Dimension 1 and Dimension 4, and showed similarities with Humanities and Arts
papers but differed from the other three domains on Dimension 2 and Dimension 3.
NETEM writing was weak in its extrapolation to real academic contexts.
Keywords: Multidimensional analysis, writing, linguistic features

1. Introduction
Whether test tasks elicit linguistic features that students will use in real academic contexts is
an important extrapolation concern in argument-based validity (Kane, 2012). Take TOEFL iBTfor example. The inclusion of source-based integrated writing task in 2005 makes TOEFL
Writing Section more representative of language use in real learning environments (Llosa,
Grapin, & Friginal, 2019). The underlying assumption of extrapolation inferences is that similar
situational characteristics of the target domain elicit the linguistic features associated with it(LaFlair & Staple, 2017; Kyle et al., 2021). The emphasis on communicative functions
associated with situational characteristics in tests justifies the application of Multidimensional
analysis (MDA) (Biber, 1988) to the investigation of extrapolation of test performances to thosein the target domain. MDA takes the register perspective, believing that some linguisticfeatures are more frequently used in a register because “they are functionally adapted to thecommunicative purposes and situational contexts of texts from that register” (Biber & Conrad,2019, p. 2). More importantly, this method provides a corpus-based quantitative method tocompare a large number of linguistic features with shared communicative functions (LaFlair &Staple, 2017; Yan & Staples, 2019). The present study is the first empirical research that
employs this method to examine the linguistic features elicited in the writing task of a highstakes test in China. The purpose of this study is twofold: 1) to demonstrate how MDA can beused to investigate the extrapolation inference of this test; 2) to discuss the implications fortest developers and learners.

2. Research Background
Language users make systematic lexical and grammatical choices appropriate to thesituational characteristics of the register (Biber, 2019). Utilizing computational techniques,
MDA takes a bottom-up approach to study a large number of fine-grained lexico-grammatical754

features distributed in a register and identify co-occurrence patterns with the same underlying
functional underpinnings (Biber, 2019). The systematic co-occurring sets of linguistic features
are factors or dimensions, and should be interpreted functionally with respect to situational
characteristics. It is based on co-occurrence linguistic features that important register
differences can be revealed and explained.
In recent years, MDA has attracted mounting attention in validation research,
particularly in the investigation of extrapolation inferences because MDA not only extends TLU
domain but serves as a quantitative framework for examining the linguistic relevance between
test performances and the domains to which they are extrapolated (LaFlair & Staples, 2017;
Staples et al., 2018). Weigle and Friginal (2015) was the first empirical research using additive
MDA to examine the comparability of TOEFL Independent writing with disciplinary papers
along four dimensions established by Hardy and Römer (2013). Results indicated that TOEFL
Independent writing was characterized by more narrative expressions of opinion and stance,and expressions of possibility, whereas the latter relied more on linguistic features associatedwith procedural information. They concluded that Independent writing task was limited in its
generalization to real academic contexts. These findings were supported by Llosa et al. (2019)
who used the same four dimensions and extended their research by comparing TOEFL
Integrated writing with disciplinary papers. Llosa et al. found that this writing taskcomplemented Independent task by eliciting linguistic features associated with information
and argumentation. Based on these findings, researchers claimed that TOEFL Writing Section
approximated successful academic writing than Independent task alone. Utilizing the four
dimensions by Gardner et al. (2019), Staples et al. (2018) compared TOEFL writing with thenarrower registers such as case studies and argumentative essays in disciplinary writing.
Again, the previous findings about Independent writing were supported. However, Staples etal. pointed out that although TOEFL Integrated writing was better representative of disciplinary
writing, its overall patterns were more similar to Independent writing than they were to mostdisciplinary registers. As a result, the extrapolation of TOEFL writing was still a problem.
The current study contributes to this area of study by investigating Writing Task 2 of
the National Entrance Test of English for MA/MS (referred to as NETEM writing throughout
the paper) in China, a national high-stakes test that aims to select candidates with required
English proficiency for future intellectual pursuits. NETEM with millions of test takers isregarded as the most challenging English test for non-English majors. NETEM writing is anessay of 160-200 words based on one cartoon or a set of cartoons. Test takers are required
to describe the cartoon(s) briefly, interpret the meaning, and give comments. The differencesof NETEM writing from disciplinary papers in situational characteristics including the topic, thepurpose, and the production circumstances may result in different choices of linguistic
patterns. Our research into the linguistic features in these two situations will inform language
testing and teaching. The following research question will be addressed:
How similar are the linguistic features of NETEM writing to those of graduate leveldisciplinary papers?

3. Methodology
3.1 Corpus
The corpora for the current study consist of NETEM writing corpus and a reference corpus.
The former contains samples of NETEM writing from the past ten years’ (2013-2022) real
papers. These papers were compiled by ten different educational training and services
providers and published by some well-known publishing companies in China. Although
NETEM directions suggest 160-200 words, test takers are encouraged to produce over 180
words in order to score high. In the present research, the type-token ratio in MAT was set at
180 and four essays less than 180 words were excluded. 96 essays were used with a total of
20,891 words. The reference corpus is the Michigan Corpus of Upper-Level Student Papers(MICUSP). It is a corpus of A-graded papers by native and non-native English speakers atfour levels of studies, and can be divided into four academic categories (Weigle & Friginal,755

2015). See Table 1. The present study uses papers by graduate students in that they are inline with what NETEM candidates are expected to produce in future studies.
Table 1. NETEM Writing and Disciplinary Writing
Five domains
Number of papers
NETEM writing
96
Biological and Health Sciences
92
Humanities and Arts
78
Physical Sciences
79
Social Sciences
150

Tokens (words)
20,891
254,335
374,569
248,383
494,020

3.2 Additive MDA and Dimensions of Linguistic Features
Biber’s (1988) dimensions have become stable references for additive MDAs across various
domains (Nini, 2019; Berber Sardinha et al., 2019) in that this study described the overall
patterns of register variation based on a wide range of language use situations. Additive MDAs
allow researchers to focus on one dimension or several dimensions of interest in their studies.
Another strength is that it makes research possible based on relatively small restricted corpus.
This is the case in our study. The current study focuses on the first four dimensions because
they have more explanatory power (Biber, 2019). The four dimensions are: Involved vs.
Informational Production; Narrative vs. Non-Narrative Concerns; Explicit vs. SituationDependent Reference; Overt Expressions of Persuasion. In this study, 48 linguistic features
were used in calculating mean scores of each dimension. Refer to Biber (1988, 2014) for
details of the individual linguistic features. Dimension 1 and Dimension 3 include positive andnegative features, which suggest they are distributed in a text in a complementary manner.

3.3 MAT and Data Analysis
All the fine-grained features on the four dimensions should be tagged and computed.
Multidimensional Analysis Tagger (MAT) is an automatic tool developed by Nini (2019) that
replicates the original Biber Tagger (not publicly available). It uses the Stanford Parser toparse lexical, syntactic and semantic information of the linguistic features in Biber (1988) andeffectively replicates Biber’s algorithms and results. The reliability of MAT is an important
driving factor for the application of Biber’s dimensions in additive MDAs (Berber Sardinha etal., 2019). This study uses MAT version 1.3.2 to map the five domains onto Biber’s 4
dimensions. We used MAT to complete 3 of the steps in Berber Sardinha et al. (2019): 1)
tagging linguistic features in each text, 2) computing and standardizing their normed frequency
counts, 3) computing the dimension scores for the five domains. SPSS was used to compare
the differences in frequency counts for the linguistic features, and means on each dimensionof the five domains. Despite the normal distribution of data in this study, Levene’s test ofequality of variance was significant on all four dimensions. Therefore, Tamhane’s T2(M) posthoc tests, which don’t assume equality of variance, were run to determine where significant
differences were found.

4. Results
MAT classified NETEM writing as general narrative expositions while graduates’ papers as
scientific expositions. This indicated NETEM writing used different language features fromdisciplinary papers. ANOVA results showed significant differences on all four dimensions:
Dimension 1, F(4, 220.24) = 43.223, p = .000; Dimension 2, F(4, 223.356) = 26.616, p = .000;
Dimension 3, F(4, 219.012) = 18.350, p = .000; Dimension 4, F(4, 218.466) = 25.215, p = .000.
The means on four dimensions are displayed in Table 2 and visually shown in Figure 1, where
the shared letters on each dimension represent mean scores that are not significantly different
from each other. Post-hoc results with illustrative excerpts will be reported as follows.

756

Table 2. Descriptive Statistics (Means and SDs) across Dimensions and Domains
Dimension 1
Dimension 2
Dimension 3
Dimension 4(S.D.)(S.D.)(S.D.)(S.D.)
Biological and Health Sciences -14.44 (5.54)-3.23(1.54)
7.58 (2.86)
0.56 (3.81)
Humanities and Arts
-11.16 (6.64)-1.86 (1.61)
5.42(2.52)-1.06 (2.83)
Physical Sciences-15.19 (4.57)-3.9 (1.36)
6.56 (2.83)-0.46 (2.90)
Social Sciences-13.06 (4.93)-2.51(1.57)
7.64 (2.39)-0.42 (2.66)
NETEM Writing-4.04 (7.23)-1.39 (3.36)
4.69 (4.18)
4.71 (5.07)

10

ab

b
cd

5

a
a

d

Dimension 4
Dimension 2

Dimension 1

b

0

a

a
a

-5

c

bc bc

Dimension 3cBiological and Health Sciences

Humanities and Arts

Physical Sciences

Social Sciences

b
d

-10

bc
b
-15

c

d

NETEM writing-20

Figure 1. Mean Scores on Four Dimensions across Domains
On Dimension 1, Table 2 shows the mean score of NETEM writing is -4.04,
significantly higher than four disciplinary papers, indicating the involved and non-informational
focus. This textual style can be illustrated through the reliance on present verbs, 1st person
pronoun and possibility modals. Expert 1 illustrates the use of some of these features.
Excerpt 1: To be exact, mobile phones have occupied so much of our time that weeven don’t have time to talk with people around us! In this sense, we can say that mobile
phones have separated us from each other…(NETEM writing, Dimension 1 score = -9.29)
Contrary to these features, Excerpt 2 from Physical Sciences is characterized bynegative features such as nouns, attributive adjectives, and prepositions. Meanwhile, word
length as a negative feature suggests the use of advanced vocabulary in disciplinary writing.
Excerpt 2: If a simple transport model is sought, it is recommended that a spatiallyvarying reaction rate be employed with a constant diffusion coefficient. Future work could
incorporate hydrologic models to assess the impact of stormwater surges on contaminanttransport. (CEE.G3.03.1 Dimension 1 score = -21.89)
On Dimension 2, the mean score of NETEM writing is -1.39. Figure 1 indicates nodifference between NETEM writing and Humanities and Arts papers. They both used morenarrative features than the other three disciplinary groups. Narrative features like 3rd person
pronoun, present participial clauses are very common in NETEM writing (see Excerpt 3).
Excerpt 3: … he is convinced that it is his interest to explore the deep space thatsignificantly contributes to his present achievement, making it easier for him to survive in thiscompetitive commercial world. (NETEM writing, Dimension 2 score = 3.1)
On Dimension 3, NETEM scored 4.69, close to Humanities and Arts papers but
significantly lower than the other three groups. This low score indicated more use of contextdependent features. Explicit references such as “nominalization and pied piping construction”

757

characteristic of informational writing are significantly less used in NETEM writing. Instead, it
used more place adverbials and other adverbials as illustrated in Excerpt 4, whilenominalization was found to be a salient feature in all disciplinary papers.
Excerpt 4: Many students are facing various choices ahead …studying abroad…
(NETEM writing, Dimension 3 score = -0.26)
On Dimension 4, the mean of NETEM writing is 4.71. Figure 1 indicates NETEM writing
is clearly distinct from all disciplinary writing. The higher the score is, the more persuasive thetextual style is. NETEM writing used many linguistic features associated with persuasion suchas infinitives and obligation modals as a way of suggesting what should be done to solve theproblem mirrored in the prompt. Excerpt 6 is an example. Also, split auxiliaries are common in
NETEM writing, usually in the form of formulaic expressions such as “as is vividly
shown/illustrated in the picture” and “It is universally held that”, to name a few.
Excerpt 6: College students should not miss any opportunity to acquire knowledge toachieve their all-round development. (NETEM writing, Dimension 4 score = 5.83).

5. Discussion and Conclusion
Using corpus linguistic method MDA, the present study found that NETEM writing differedsignificantly from all four disciplinary domains on dimensions 1 and 4, and showed similarities
with Humanities and Arts but differed from the other three domains on dimensions 2 and 3
Regarding Dimension 1 and Dimension 4, NETEM writing was characterized byinterpersonal interactions and persuasion, while disciplinary papers used more linguisticfeatures associated with information density. These differences can be primarily explained by
the differing purposes in writing tasks. Disciplinary tasks as a type of source-based writing
focus on the presentation of propositional information while NETEM writing requires test takers
to describe the cartoons vividly and argue convincingly for the viewpoints generated. Given
that Dimension 1 is the most basic variation among registers in nearly all MDA studies (Biber,2014) and therefore, an important marker for varying degrees of information density, a weaklink exists between NETEM writing and real academic contexts. That is, it is impossible topredict how learners will perform linguistically on these two dimensions based on test scores(LaFlair & Staples, 2017). It is worth noting that the major syntactic functions of the linguisticfeatures with negative weights in Dimension 1 are phrasal modifiers in noun phrases. The
dense use of phrasal modifiers is the unique style of academic prose (Biber et al., 2011). Yet
these features are disproportionately underrepresented in NETEM writing. Just as Kyle et al.
(2021) pointed out “…if key linguistic features of a particular domain are not required for the
successful completion of a task, support for the extrapolation inference is also weakened”,
NETEM writing failed to elicit the language patterns required in graduate level studies.
The alignment of NETEM writing with Arts and Humanities papers on Dimension 2 andDimension 3 is not surprising since this domain consists of many papers from philosophy,
history and classical studies where narration and description are common. This finding is
consistent with Weigle and Friginal (2015), Llosa et al. (2019), and Staples et al. (2018).
However, similarities on these two dimensions do not support the extrapolation of test task toother disciplines because the sharp differences from disciplinary writing on other dimension(s)still make the authenticity of the task problematic (Staples et al., 2018).
In addition to the nature and requirements of NETEM writing, the stereotype ofacademic language as clausally complex is also at work, influencing the choices in NETEM
writing of linguistic features different from disciplinary papers. For example, the bestseller
among NETEM candidates published by a renowned press in China encourages the use ofparticipial clauses and adverbial clauses as a way to score higher. Some education andtraining providers encourage test candidates to use more personal examples as an effective
way to argue for viewpoints, thus making narrative features salient. All this is no different fromplacing an obstacle to learners’ academic language development.
The present study also affirms that independent writing alone is not representative of
linguistic features that learners are expected to learn in the university setting. Although
NETEM did undergo changes in test tasks, cartoons as writing prompts has been used since758

the 1980s. It is time to re-examine and reform the test task. Admittedly, it is hard to solve theextrapolation problem once and for all. Yet test developers should give more attention tosource-based writing skills, an important academic literacy in today’s higher education.

Acknowledgements
This study was supported by the 11th China Foreign Language Education Fund(ZGWYJYJJ11Z025) and Research on Philosophy and Social Sciences of Jiangsu
Universities in 2022 (2022SJYB0101).

References
Berber Sardinha, T., Veirano Pinto, M., Mayer, C., Carolina Zuppardi, M., & Herique Kauffmann, C.
(2019). Adding register to a previous Multi-Dimensional analysis. In T. Berber Sardinha, & M.
Veirano Pinto (Eds.), Multi-dimensional Analysis: Research Methods and Current Issues (pp.165186). Bloomsbury Academic.
Biber, D. (1988). Variation across Speech and Writing. Cambridge University Press.
Biber, D. (2014). Using multi-dimensional analysis to explore cross-linguistic universals of registervariation. Languages in Contrast, 14(1), 7-34.
Biber, D. (2019). Multi-dimensional analysis: A historical synopsis. In T. Berber Sardinha, & M. Veirano
Pinto (Eds.), Multi-dimensional Analysis: Research Methods and Current Issues (pp.11–26).
Bloomsbury Academic.
Biber, D., & Conrad, S. (2019). Register, Genre, and Style. Cambridge University Press.
Biber, D., Gray, B., & Poonpon., K. (2011). Should we use characteristics of conversation to measuregrammatical complexity in L2 writing development? TESOL Quarterly, 45(1), 5-35.
Gardner, S., Nesi, H., & Biber, D. (2019). Discipline, level, genre: Integrating situational perspectives ina new MD analysis of university student writing Applied Linguistics, 40(4), 646-676.
Hardy, J. A., & Römer, U. (2013). Revealing disciplinary variation in student writing: A multi-dimensional
analysis of the Michigan Corpus of Upper-Level Student Papers (MISCUP). Corpora, 8(2),183-207.
Kane, M. T. (2012). Articulating a validity argument. In G. Fulcher, & F. Davidson (Eds.), The Routledge
Handbook of Language Testing (pp. 34-47). Routledge.
Kyle, K. A. T., Choe, A. T., Eguch, M., LaFlair, G., & Ziegler, N. (2021). A comparison of spoken andwritten language use in traditional and technology-mediated learning environments. ETS
Research Report Series, (1),1-29.
LaFlair, G. T., & Staples, S. (2017). Using corpus linguistics to examine the extrapolation inference inthe validity argument for a high-stakes speaking assessment. Language Testing, 34(4), 451-474.
Llosa, L., Grapin, S. E., Friginal, E., Cushing, S. T., & Malone, M. E. (2019). Linguistic dimensions of
TOEFL iBT essays compared with successful student disciplinary writing in the university. TESOL
Quarterly, 54(1), 251-265.
Nini, A. (2019). The multi-dimensional analysis tagger. In T. Berber Sardinha, & M. Veirano Pinto (Eds.),
Multi-dimensional Analysis: Research Methods and Current Issues (pp. 67-94). Bloomsbury
Academic.
Staples, S., Biber, D., & Reppen, R. (2018). Using corpus-based register analysis to explore theauthenticity of high-stakes language exams: A register comparison of TOEFL iBT and disciplinary
writing tasks. The Modern Language Journal, 102(2), 310-332.
Weigle, S. C., & Friginal, E. (2015). Linguistic dimensions of impromptu test essays compared withsuccessful student disciplinary writing: Effects of language background, topic, and L2 proficiency.
Journal of English for Academic Purposes, 18, 25-39.
Yan, X., & Staples, S. (2019). Fitting MD analysis in an argument-based validity framework for writing
assessment: Explanation and generalization inferences for the ECPE. Language Testing, 37(2),189-214.

759

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Enabling Visually Impaired People’s
Chinese Literacy learning throughInformation Technology
Shelley Shwu-Ching YOUNGa* & Jen-Li WANGb
Institute of Learning Science and Technology, National Tsing Hua University, TaiwanbNational Center for High-Performance Computing*scy@mx.nthu.edu.twaAbstract: How to help the visually impaired people in Taiwan to be able to learn
Chinese literacy online has been a big challenge. However, in order to overcome the
obstacle, we intentionally designed and implemented a cloud-based Visually Impaired
People (VIP) E-learning Platform that affords to empower their Chinese literacy
learning via specific typing skills and relevant course design so that they would havethe opportunity to become independent online navigators and learners as well. To
facilitate and examine their performance in terms of typing accuracy and speed on the
Chinese typing and information searching online, etc., a competition activity wasorganized. Both quantitative and qualitative data were collected and analyzed via aquestionnaire administered to the 18 participants, divided into three groups based on
their ages. The results indicate that overall the participants are highly satisfied with the
VIP platform for its empowerment in Chinese literacy learning that affords their online
independent learning opportunity and barrier-free communication skills with sighted
people after using the VIP platform. Additionally, some suggestions for the VIP platform
improvement were given by the participants to achieve a more user-friendly and better
online learning environment in the future.
Keywords: Visually impaired people, e-learning, typing competition, Chinese literacy

1. Introduction
A computer-based multimedia learning environment is one of the most commonlyused ways of learning nowadays. Multimedia learning emphasizes not only the use of visual
aids but also auditory aids to improve learners’ learning and understanding (Mayer et al.,2002). Moreover, multimedia tools can be effective aids to the research consent process
under some circumstances (Palmer et al., 2012).
However, for those who are visually impaired, the abundant visual aids provided byMultimedia are almost ineffective. The visual system is the main sensory source of humanbeings. Eight-seven percent of the total capacity of the brain process visual information(Hsieh, 2014). The limited channels for the visually impaired to understand and learn about
the world is listening and tactility.
Providing equal learning opportunities for those who are visually impaired is getting
more and more attention nowadays. Educational service organizations around the world
have begun to provide online learning courses and digital learning resources for thosevisually impaired.
A university and an organization in Taiwan have jointly committed to the developmentof a cloud-based Visually Impaired People (VIP) E-learning Platform based on thescaffolding theory and Web Content Accessibility Guidelines (WCAG) 2.0 to help the visually
impaired to learn online by using computer typing since 2016. It is hoped that through longterm research and promotion, the visually impaired could learn to benefit from this platform.

760

Therefore, given the length of the manuscript, this paper does not report the VIP
system. Rather, this paper aimed to exam the potential of the visually impaired in Chinese
typing and information searching through the competition results and questionnaire analysis.
The following would be the research questions.
1. What was the participants’ engagement in terms of frequency of practicing, time spend,
and frequency of typing the same materials?
2. How did visually impaired learners feel about using the VIP E-learning platform, in
terms of advantages and the needed improvement?
3. What would be the performance of those participants in term of typing accuracy andspeed on the Chinese typing and information searching competitions?

2. Literature Review
2.1 Computer-based Multimedia Learning
Learners who receive information mainly via both auditory and visual channels
simultaneously can absorb more sensory information, retain more memory, and increase
their problem- solving ability (Mayer, 2005). Besides, learners can take advantage of five
aids of multimedia to improve the effectiveness of learning, such as multimedia aids,
contiguity aids, coherence aids, modality aids, redundancy aids in the computer-basedlearning environment (Mayer, 2002). However, apart from Modality aids, none of the otherfour seem to apply to the visually impaired since the other four rely on the visual aids.
Moreover, the web pages nowadays consist of a large number of animations, special effects,
or pictures that so- called multimedia, but no matter how attractive the visual aids are, thevisually impaired still cannot benefit from them (Lin, 2003).

2.2 The Challenges for the Visually Impaired people in Typing ChineseContacting people and searching for information on the Internet using computer
typing is undoubtedly a common activity and necessary skills for people living in theinformation era. For the visually impaired, computer typing is the main way for them to do
text communication (Wang et al., 2014). However, the visually impaired encounter difficulties
in typing in Chinese, such as the homophones with different meanings and the unavailability
to check the correctness of words. Compared with English alphabetic systems, vocabulary iscomposed of letters and the pronunciation is basically as shown in spelling. For example,
both “Male” and “Mail” are pronounced as “[meil],” and both “Whole” and “Hole” arepronounced as “[houl].” The spelling of the English words differentiates based on thecontext, but can be determined if the word is correctly used or not. On the other hand,
Mandarin Chinese belongs to the logographic system, in which symbols represent the words
themselves, and many words with homophones but different meanings are common. For
example, both expectation (㛇⼭)and Umbilical cord (兵ⷞ) are pronounced as “qídài” in
Mandarin Chinese; both stray and elk are pronounced as “mílù” “οϕȃηόȄ”; both stray(徟嶗) and elk ( 渳渧) are pronounced as “mílù” “εϕȃ κϖȄ” in Mandarin Chinese but
their meanings are different. This phenomenon will cause difficulties in computer text
communication with a group of people and searching information on the Internet. In addition,
the visually impaired cannot immediately distinguish the correctness of words by “seeing”
when typing, so typing with typos is a common phenomenon (Wang, 2014; Weng, 2015).

3. Methodology
3.1 Introduction to the Competition
The competition for the visually impaired on Chinese typing and information searching
was jointly organized by Taiwan Digital Talking Book Association (TDTB) and Graduate

761

Institute of Learning Sciences and Technologies, National Tsing Hua University, andsponsored by the foundation of Elite-well. The competition included two sections, Mandarin
Chinese typing and information searching. In order to assure the competition quality, typing
speed of at least 15 Chinese characters per minute is minimally required for the participants.

3.2 Rules for the competition
In the Chinese typing section, participants were given 10 minutes to type an article
selected by the organizers with at least 1000 words presented via the NVDA report in the
Mandarin Chinese twice, the higher record would be the final score. As long as the time isup, the VIP system will automatically stop and display the typing speed and accuracy. The
difficulty of the selected articles encountered by the three groups of students in thecompetition increases according to their age.
In the information searching section, participants proceeded 20 minutes to search thecorrect answers to the designated questions designed by the organizers on the Internet andsubmitted the resource link. The difficulty of the questions encountered by the three groupsof students in the competition increases according to their age. The more correct answers
would win the competition.
The two sections of competition aimed to cultivate participants’ skills of computer
operation and typing ability for further facilitate self-learning and satisfying career
requirement.

3.3 Participants
The participants of this research are 18 people who participated in the typing
competition for the blind on November 21, 2021. Their age ranges from 10 to 41 years old;
education level ranges from elementary school to college or above.

3.4 Groups
1.
2.
3.

Elementary school (under 12 years old): 2 participants with one male and one female
Junior high and senior school (13-18 years old): participants with 5 males and 3
females
Adult (above 18 years old): 8 participants with 4 males and 4 females

3.5 Four Phrases of this study
1.
2.
3.
4.

Preparation of competition materials, including selected articles for typing, and web
searching questions
Questionnaire design
Competition: Chinese typing and information searching
Data collection and analysis

3.6 Research Methods
This research adopted mixed research methods, including both quantitative andqualitative approaches. The quantitative data were collected through the competition resultsof both Chinese typing and information searching and the questionnaire based on the 5Point-Likert-Scale questions. The qualitative data were collected through the open-ended
questions in the questionnaire, which was used to investigate the advantage anddisadvantage of the VIP platform.
The questionnaire used in this research is designed with the Likert five-point scale toinvestigate the participants’ perception to the VIP platform, which was distributed after theend of the VIP typing competition on November 21, 2021.

762

3.7 Questionnaire analysis
Among the questionnaires returned, the valid questionnaire is 18. The following
section would address research question 1: What is the participants’ engagement in terms offrequency of practicing, time spend, and frequency of typing the same materials?
1)Frequency of Practicing the Typing materials Per Week on the VIP platform
As shown in the Figure 1, among the 18 participants, 9 people use the VIP E-learning
platform once or twice a week, accounting for 50% of the overall; 5 people watch the course
3-4 times a week, accounting for 28% of the overall; one person didn’t watch the course every
week, accounting for 5% of the total; 3 people who have never watched the course,
accounting for 17% of the total.
2)Time on Each Chinese Typing Practice on the VIP Platform
As shown in the figure 2, among the 18 participants, 4 had never practiced, accountingfor 22% of the total; 5 had practiced for 10-20 minutes each time, accounting for 28% of thetotal; 4 had practiced for 30-40 minutes each time, accounting for 22% of the total; 3 people
practice 50 minutes to 1 hour each time, accounting for 17% of the total; 2 people practicemore than 1 hour each time, accounting for 11% of the total.
3)
Frequency of Course Review on VIP Platform
As shown in Figure 3, among the 18 participants, 13 have never reviewed, accounting
for 72% of the total; 2 have reviewed once a month, accounting for 11% of the total; 2 havereviewed twice a month, accounting for 11% of the total; 1 only watch the course two weeks
before the game, accounting for 6% of the total. Two reasons for the high number of neverreviewers are:
1. They only use the platform as a place to practice typing.
2. They don’t have time to review due to busy schoolwork or extracurricular activities.
The following sections would address the research question 2: How do visually impaired
learners feel about using the VIP E- learning platform, in terms of advantages and the needed
improvement.
4)
Satisfaction of Using VIP Platform
Eight satisfaction questions are provided in the form of the 5- point Likert scale. As
shown in Table 1, apart from the discussion forum and online consultation, participants arehighly satisfied with the VIP platform.
Table 1. Satisfactions of Using the VIP Platform
Options
Convenience of accessibility
Learning method
Exercises after each course
Videos, text files, slides, and extended information
Discussion forum
Online consultation
Listening and typing exercises
Exercise of information searching

Mean
4.33
4.11
4.07
4
3.5
3.5
4.6
4.22

SD
0.68
1.07
0.86
0.9
0.9
0.9
0.48
0.73

5)
Advantages of VIP Platform
The advantages of the E-Learning platform can be classified into two categories basedon the results of the questionnaire, including the platform operability and the practicality oflearning content.
First, platform operability. Except from the elementary school group, 3 participants injunior & senior high school group and four participants in the adult group think the VIP platform
provide a simple interface and easy to use, accounting for 39% of the total. For example, J2A:
“This website is accessible, easy to operate, and easy to use.” C8C: “This website provides asimple interface and is easy to use.”
Second, practicability of learning content. Forty-four percent of the participant aresatisfied with the practicability of the learning content to empower them to type accurately,763

accounting for 44% of the total. For example, E1A: “I become more familiar with the keyboard
when operating a computer.” S1A: “I can practice typing speed and accuracy on the website.”
J6A: “It can let the learners themselves know which words they type are incorrect, and allow
those who are visually impaired know what the word means.” Three did not provide advice,accounting for 17% of the total.
6)
Needed Improvements of the VIP Platform
The needed improvements can be categorized into three categories: information
searching materials, interface of the VIP platform, and functions of the VIP platform.
First, information searching materials. Participants recommended to make thesearching easier for Junior and Senior high school group.
Second, interface of the VIP platform. Participants recommended to add theautomatic login function, improve the quality of online consultation, shorten the website URLof the VIP platform, make the whole webpage into the situation that the screen reader canread the words out.
Third, functions of the VIP platform. Participants recommended to classify thelearning content according to difficulty, proofread the typos in the listening and typing
articles, and update the article in a timely manner.
7)
Competition Results
The competition was composed of two types of skills, including Chinese typing andinformation searching. Because typing with a high error rate will cause difficulty incommunication or misunderstandings, the requirement for the error rate of the best typing
record must be less than 6%.
First, Chinese typing competition. As shown in the Table 2, the best typing record in
the Elementary school group is 430 words in 10 minutes, with an average of 38 words per
minute and an error rate of 4%. The best typing record in the Junior & Senior high schoolgroup is 411 in 10 minutes, with an average of 36 words per minute and an error rate of 4%.
The best typing record in the Adult group is 813 words in 10 minutes, with an average of 79
words per minute and error rate of 1%. Although JS1 in Junior & Senior high school group
can type 698 words in 10 minutes with an average of 60 words per minute; and A1 in Adultgroup can type 1039 words in 10 minutes with an average of 86 words per minute, due totheir 6% error rate, neither of them won.
Based on the Techficiency Quotient Certification (TQC) for Chinese typing, theChinese typing speed can be divided into three levels according to the average typing speed
per minute: Practical level: 15-29 characters per minute with the error rate less than 10%;
Advanced level: 30- 79 characters per minute with the error rate less than 10%; Professional
level: over 80 characters per minute with the error rate less than 10%. As we can see from
Table 2, the best record for the Elementary group and the Junior & Senior high school group
can reach to the advanced level. Besides, the best record in the adult can achieve to theprofessional level. Therefore, by diligent practicing on the VIP platform, the visually impaired
can even perform better than those who are not visually impaired.
Second, information searching competition. The difficulty of the information searchingdifferentiates depends on the age of each group. As shown in the Table 3, the best record in
the Elementary school group can search 9 correct answers in 20 minutes. The best record in
the Junior & Senior high school group can search 10 correct answers in 20 minutes. The
best record in the adult group can search 6 correct answers in 20 minutes.
Searching information through the Internet is a necessary skill nowadays, bypracticing on the VIP platform, the visually impaired are empowered to have the ability tosearch information.
Table 2. The best record for each group in Chinese typing
Group/Numbers of participant
Winner
Total
Error rate
Elementary school/2
E1
430
4%
Junior & Senior high school/8
JS2
411
4%
Adult/8
A2
813
1%
Table 3. The best record for each group in information searching

764

Average typing speed
38 words/min.
36 words/min.
79 words/min.

Group/Numbers of participant
Elementary school/2
Junior & Senior high school/8
Adult/8

Total
9/15
10/18
6/20

4. Conclusion
The results of this study indicate the effects of the modality aids proposed by Mayer. The
online platform could benefit those VIP and they are highly satisfied with the VIP platform.
Their performance on the typing speed and accuracy are outstanding and impressive. Whenthe visually impaired is continuously practicing on the VIP platform, they can gradually
reduce the typing error rate, which not only allows them to strength their information
searching and communication skills, apply for jobs with this skill, but also reduces theinconvenience in communication with people in written language. Moreover, it is expectedthat the users on the VIP platform can take the advantage of this the practicing materials toachieve the self-directed learning on the Internet.

Acknowledgements
Thanks for the partial support from the Ministry of Science and Technology (MOST), R.O.C.,
under the grant codes: MOST 108-2511-H-007 -008 -MY3 and MOST 112-2410-H-0070023, moreover, all of the participants involved in this study.

References
Hsieh, C.Y. (2014). See the Music and Tell a Story through Space: Interactive storytelling with Art andTechnology. Unpublished Doctoral dissertation.
Lin, P. J., (2003). A Study on G-Mouse System and Web Content Accessibility. Master's thesis,
Tamkang University.
Mayer, R. E. (2005). Cognitive theory of multimedia learning. The Cambridge handbook of multimedia
learning, 41, 31-48. Cambridge university press.
Mayer, R. E., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning andinstruction, 12(1), 107-119.
Palmer, B. W., Lanouette, N. M., & Jeste, D. V. (2012). Effectiveness of multimedia aids to enhancecomprehension during research consent: A systematic review. Irb, 34(6), 1.
Wang, J.-L., Huang, C.-W., & Young, S. S.-C. (2014). The study on choosing computer Chinese input
method for visually impaired people. Paper presented at the 10th Taiwan E-learning Forum.
Taipei, Taiwan.
Weng, T.-H. (2015). Establishing a Learning Model of Boshiamy Input Method for Visual Impaired.
Master's thesis, National Tsing Hua University.

765" !   !" " "   #" !
#" !"$  #" ! #"           

&$&"&'' *(+  ')% ! # %& %  "!"   
 
 !"  !" 
C>:8<>C>CD5B 02 9?  %>BC43D20C8>=0;C427=>;>684B5>AC40278=6;0=6D064B:8;;B68E4BCD34=CBC74
B0<4;40A=8=64G?4A84=2410B43>=C740BBD<?C8>=C70C4E4AHBCD34=C;40A=B0CC74
B0<4A0C4 !=>A34AC>?A>E834<>A48=38E83D0;8I438=BCAD2C8>=F4=443C>CA02:BCD34=C
:=>F;43640C058=46A08=43;4E4; +78BA4B40A274G?;>A4B7>FC>033C74018;8CHC>CA02:
:=>F;4364 8= 0= 4G8BC8=6 43D20C8>=0; C427=>;>6H BHBC4< 5>A "0?0=4B4 B42>=3
;0=6D064 ;40A=4AB  .4 4G?;>A4 B4E4A0; ?>C4=C80; B:8;; <>34;B 10B43 >= C74 540CDA4B
0E08;01;48=C74BHBC4<0=3C74=0??;H00H4B80=:=>F;4364CA028=60;6>A8C7< .40;B><0:40;0A6430C0B4C5>A"0?0=4B4;0=6D064;40A=8=60E08;01;4F8C7=>?A4E8>DB
0??;843:=>F;4364CA028=6  #=>F;4364+A028=6*:8;;<>34;8=6 =3$0=6D064$40A=8=6"0?0=4B4

+78BA4B40A278B5>2DB43>=8<?A>E8=6C74CA02:8=6>5BCD34=CB42>=3;0=6D064;40A=8=6
:=>F;43645A><0=4G8BC8=6@D4BC8>=10B43CA08=8=6BHBC4<5>A"0?0=4B40B0B42>=3
;0=6D064 ,B8=64BC01;8B743:=>F;4364+A028=6#+C427=8@D4BF4F4A401;4C>20;2D;0C4
C7414BC58CC8=6:=>F;4364<>34;0=3F4A401;4C>B7>FC70C1H0338=6030?C8E48=BCAD2C8>=
F42>D;3B0E4;40A=4AB5A><2><?;4C8=6C7>DB0=3B>5?A>1;4<B8=C74BHBC4<C70CC74H70E4
0;A403H<0BC4A430;;>F8=6C74<C>5>2DB>=B:8;;BC70C6>D=<0BC4A43 +74DB4>543D20C8>=0;C427=>;>6HBHBC4<BBD270B>=;8=42>DAB4F0A40=33868C0;
;40A=8=6?;0C5>A<B70E4144=B7>F=C>1445542C8E48=74;?8=6;40A=4AB02@D8A40B42>=3
;0=6D064.411><0= )4B40A2770BB7>F=C70CA4?40C43?A02C8248B0:4H
502C>A8=BD224BB5D;B42>=3;0=6D064;40A=8=60=3<0=H43D20C8>=0;C427=>;>6HBHBC4<B
?A>E834<D;C8?;4>??>ACD=8C84B5>A;40A=4ABC>?A02C8240=3DB4C74=4FF>A3B0=3B:8;;BC74H
0A4;40A=8=6&0:0C0 +8<4;H0=32>AA42C8E45443102:8B0;B>0=8<?>AC0=C540CDA4>5
43D20C8>=0;C427=>;>6HBHBC4<B5>AB42>=3;0=6D064;40A=8=60B8C74;?B;40A=4ABC>834=C85H
0=32>AA42C4AA>AB8=A40;C8<4 0= H;4E4A068=6030?C8E8CH8=C74B4B42>=3
;0=6D064;40A=8=6BHBC4<BC742>AA42C8E45443102:0=3<D;C8?;4?A02C824>??>ACD=8C84B
?A>E8343C>BCD34=CB20=14<034<>A445542C8E45>AC748A;40A=8=6.411><0='=4BD27F0HC>4=01;4C78B030?C8E8CH8BC7A>D67#%>34;8=6F7827=>C>=;H4=01;4B
030?C8E8CH1DC8C0;B>?A>E834B0F0HC>022DA0C4;H<40BDA4BCD34=C:=>F;4364 #B0A4
58=46A08=43A4?A4B4=C0C8>=B>5:=>F;4364C70C8=2;D342>=BCA08=CBB274<0B0=3?A>3D2C8>=
AD;4B*C0<?4A#>438=64A +74#B0A4CH?820;;H<0??43C>4E4AHBC4?>50
?A>1;4<F78278B0BB>280C43F8C70=02C8>=C74BCD34=C=443BC>C0:48=>A34AC>B>;E4C74
?A>1;4< 027?A>1;4<BC4??A>E834B0=>??>ACD=8CH5>ABCD34=CBC>34<>=BCA0C4C748A
<0BC4AH>5C740BB>280C43#B +78B<0??8=6>5#BC>C74?A>1;4<BC4?B8B:=>F=0B0:=>F;43642><?>=4=C<>34;#%0=3=443BC>14022DA0C4;H34B86=43C><>34;BCD34=C
;40A=8=6*C0<?4A0=3#>438=64A +74DB4>5:=>F;4364CA028=6#+C><>34;
BCD34=C;40A=8=6C7A>D67C74?A>6A4BB8>=>5#B70B144=?A>E4=45542C8E48=C743><08=>5

766

;0=6D064;40A=8=6)8IE84C0;  0H4B80=#=>F;4364+A028=6#+8B0F834;HDB43
#+0;6>A8C7<F7827DB4B?0A0<4C4AB5>A;40A=8=6A0C40;>=6F8C76D4BB0=3B;8??0A0<4C4AB
>A14CC=34AB>=  +78B<>34;8BDB438=<0=H8=C4;;864=CCDC>A8=6BHBC4<B0=3
F>A:B5>A0=H3><08=  
+7443C427BHBC4<DB438=C78BBCD3H8B34B86=43C>@D8I"0?0=4B4B42>=3;0=6D064
;40A=4AB0=38B5>2DB43>=?08A438=CA0=B8C8E40=3CA0=B8C8E4E4A1B !=C74"0?0=4B4
;0=6D064C74B4E4A1BB70A40BC4<0=370E4?08ABF8C74027>C74A4 6 0CBD
<4AD60C74AB><4C78=60=30CBD<0AD?4>?;460C74A +7438554A4=24B>5C7>B4
?08A438=CA0=B8C8E40=3CA0=B8C8E4E4A1B0A410B820;;HB4=C4=24BCAD2CDA40=3C74?0AC82;4B
DB43F8C7C74< !=CA0=B8C8E4E4A1B70E4C74BD1942CF78;4C74CA0=B8C8E4E4A1B3>=>CA4@D8A4
C74BD1942C E4=85"0?0=4B4;0=6D064;40A=4AB<0BC4AC7>B4E4A1B0CC741468==4A;4E4;
C74H0A4>5C4=2><<>=<8BC0:4B5A><8=C4A<4380C4>A03E0=243;4E4;BCD34=CB0BC74H
?A>6A4BB':8<>C>
 *8=248C8B>5C4=385582D;C5>A"0?0=4B4;0=6D064;40A=4ABC><0BC4A
C7>B4?08A43E4A1B"0?0=4B4;0=6D064C40274AB>5C4=70E4385582D;CH1D8;38=6>=C74B4
2>=24?CBF8C7>DC58ABC033A4BB8=6C74385582D;C84BBCD34=CB4=2>D=C4A&0:08B78
 
>AC74?A4B4=CBCD3H0C>C0;>58C4<BF4A4DB438=0F44:2>DAB4>5>=;8=4BCD3H
F78278=C4A<4380C4;4E4;"0?0=4B4;0=6D064;40A=4AB9>8=43 (0AC828?0=CB0=BF4A430C>C0;>5 @D4BC8>=B?4A0224BB>=40224BB?4A30H58E430HB0F44: >AC7458E48C4<CH?4BC4=
8C4<BF4A4A0=3><;HB4;42C43 +740E4A0642><?;4C8>=C8<45>A 8C4<BF0B0??A>G8<0C4;H
14CF44= C> <8=DC4B !C4<BF4A468E4=A0=3><;H0=340278C4<0??40A43C>  
C8<4B8=0F44:?4A8>3 >AA42C8E45443102:F>D;30??40A05C4AC74BCD34=C0=BF4A43
40278C4< +74A40A4CF>CH?4B>55443102:>=45>A2>AA42C0=BF4ABC74>C74A5>A8=2>AA42C
0=BF4AB?08A43F8C74G?;0=0C8>=B0=32>AA42C0=BF4AB  
+74?0AC828?0=CB8=C78BBCD3HF4A4"0?0=4B4;0=6D064;40A=4ABF7>F4A44=A>;;438=
2>DAB4B4@D8E0;4=CC>C74;0C48=C4A<4380C4C>40A;H03E0=243;4E4;B0C0=8=BC8CDC8>=>578674A
43D20C8>=8=+>:H> +74A4F4A4 ?0AC828?0=CB >5F7><F4A48=C742>=CA>;6A>D?F7>
C>>:>=;HC74?A4C4BC0=3?>BCC4BC0=3 8=C744G?4A8<4=C0;6A>D?F7>C>>:C74?A4C4BC
F44:>=;8=4BCD3H0=3?>BCC4BC >C7C742>=CA>;6A>D?0=3C744G?4A8<4=C0;6A>D?9>8=43
C74"0?0=4B4;0=6D0642>DAB40CC748=BC8CDC8>= +741A40:3>F=>5=0C8E4;0=6D064BF0B 
78=4B4=0C8E4B?40:4AB#>A40==0C8E4B?40:4AB=6;8B7=0C8E4B?40:4AB0=3
B?40:4AB>5>C74AB80=;0=6D064B+060;>6-84C=0<4B40=3!=3>=4B80= +748C4<
30C0B4C2>=B8BCB>58C4<BC4BC8=61468==4A0=38=C4A<4380C4;4E4;?08A438=CA0=B8C8E40=3
CA0=B8C8E4E4A1DB064>5"0?0=4B4;0=6D064;40A=4AB +74A40A40338C8>=0;20C46>A8I0C8>=B>5
C748C4<BE>201D;0AH385582D;C"$(+;4E4;!=CA0=B8C8E4>A+A0=B8C8E4E4A1+4A0<DA0E4A1
+4A0<DA0  0=3<>C8>=E4A1 +74E>201D;0AH385582D;CH0=3"$(+;4E4;0A434C4A<8=43
1HC74%0CBDB78C030C010B4%0CBDB78C0 +4A0<DA08B48C74A >A734?4=38=6
F74C74AC74E4A18B8=C74+4A0<DA0E4A1B4C !=C7420B4C70CC74C0A64CE4A18B8=C74
+4A0<DA030C0B4C8C8B;014;43J7K858C8B=>C?A4B4=C8=C7430C0B4C8C8B;014;43J K $0BC;H<>C8>=E4A1B >A<34B2A814F74C74AC74E4A18=C748C4<8B0<>C8>=10B43E4A10E4A1
CH?4"0?0=4B4B42>=3;0=6D064"*$;40A=4AB70E4385582D;CHF8C7     

!=8C80;#%BF4A42A40C431HDB8=6C74540CDA4B>5C748C4<B8=C74BHBC4< .4A4?>ACC7458C
>558E4834=C85843#%B1H0H4B80=!=5>A<0C8>=A8C4A80!0=3!C4<;>2:43)>>C%40=
*@D0A43AA>A!)*%8=+01;4 !8B0<>34;2><?0A8B>=<4CA82C70C8BF834;HDB43
C>2><?0A4B:8;;<>34;B8=8=C4;;864=CCDC>A8=6BHBC4<B$8D4C0;
0=3033B0?4=0;CH5>AC742><?;4G8CH>50<>34;*C0<?4A4C0;   .4DB4#+C>834=C85HC74=D<14A>5
>??>ACD=8C84BC70CBCD34=CB=443C>A4027<0BC4AH8=4027:=>F;43642><?>=4=CF8C7C74
#<>34;F7827F0BC7414BC1H! BCD34=C8B2>=B834A430B<0BC4A8=60#85C748A
8=54AA43<0BC4AH>5C78B#0CC7458=0;0CC4<?C8B0C;40BC >A14CC=34AB>=  
0B43>=C78B=>C8>=F45>D=3C70C0;;1DCBCD34=CB<0BC4A430;; #BF74A4C74HF4A4

767

68E4=0C;40BC >A<>A4>??>ACD=8C84B>=068E4=# %>BC>5C74#BF4A4<0BC4A43A0C74A
@D82:;HBD664BC8=6C70CBCD34=CB0A468E4=<>A4>??>ACD=8C84BC70=C74H02CD0;;H=4430=3
C70C0338=6030?C8E8CH2>D;38<?A>E4;40A=8=6C8<42>=B834A01;H .8C7C740??;820C8>=>5#<>34;8=60=30338C8>=>5#=>F;4364+A028=6F4B7>FC70CF420=<>A4?A428B4;HCA02:
;40A=8=68=>DA;0=6D064@D4BC8>=8=6BHBC4<     ! " 
%>34;&0<4
1468==4A"$(+;4E4;8C4<CH?4
8C4</CH?4
1468==4A"$(+;4E4;8C4<CH?4+4A0<DA0<>C8>=
1468==4A"$(+;4E4;
@D4BC8>=/8C4</30C0

#B

!
!)*%
         
    
 'DA=4GCBC4?8BC>DB4C74#+C>?A>E8345443102:C>C74BCD34=CB ;>=64AC4A<
6>0;8BC>033CAD4030?C8E8CH0=38=C4;;864=CCDC>A8=620?018;8C84BC>>DA;0=6D064BHBC4<8=
>A34AC>8<?A>E4C74;40A=8=64558284=2H>5BCD34=CBDB8=6C74BHBC4<B0E8=6C7>DB0=3B>5
?A>1;4<B .78;4C78B8=8C80;4G?;>A0C8>=F0B>E4AB4E4A0;#<>34;BF8C7#+5>A:=>F;4364
CA028=6F4?;0=C>3><>A41H4G?;>A8=60338C8>=0;#+0;6>A8C7<B0=3AD=B><4?8;>CC4BCB
DB8=6C74<>34;B0=30;6>A8C7<B5>A030?C8E8CH DAC74AF470E4<034C7430C0B4C0E08;01;4
74A48=0C0*7>?#>438=64A4C0; 7CC?B ?B;230C0B7>? F41 2<D 43D (A>942C83
0=37>?4>C74ABF8;;0;B>C0:403E0=C064>5C78B30C0B4CC70C2DAA4=C;H70B<0=HBCD34=C
0CC4<?CB0=3=><>34;B>A0;6>A8C7<BD=34A;H8=6C74>A34A>5@D4BC8>=BC>C74;40A=4AB 

>A14CC + =34AB>=" )   #=>F;4364CA028=6%>34;8=6C7402@D8B8C8>=>5?A>243DA0;
:=>F;4364 '&!# "")'&$("(&(#"   
0=/  .A8CC4=2>AA42C8E45443102:5A><0=42>;>6820;?4AB?42C8E4+748=C4A02C8>=14CF44=
C742>=C4GC0=38=38E83D0;;40A=4AB +'(!    #>438=64A# ) *C0<?4A"  $414A *:>6B7>;<  $40A=$01B0C0*7>?0C0)4?>B8C>AH0=3=0;HC82B+>>;*4C5>A>6=8C8E4*284=24 #$'" #"(*"
 $8D) 0E4=?>AC" *C0<?4A"  4H>=3$>68;4B,B8=6%D;C8%>30;0C0*CA40<B
C>F0A3B0C0A8E4=#%>34;!<?A>E4<4=C  "(&"(#"  )(#"  (""#(+ &0:08B78/  *42>=3$0=6D0642@D8B8C8>=)4B40A27>="0?0=4B4(08A43!<?4A0C8E40=3
+A0=B8C8E4-4A1B #+#$" "")" ) ()&) '" 
&0:0C0+  >4BA4?40C43?A02C824<0:4?4A542C+7445542CB>5F8C78=B4BB8>=A4?40C43
A4CA84E0;>=B42>=3;0=6D064E>201D;0AH;40A=8=6 ()'"#"")%)'(#" %0CBDB78C0+  ->201D;0AH0C010B45>A)4038=6"0?0=4B4-)"-4A  
7CC? FFF  D8 B0:DA0 =4 9? C0CBD< 30C010B4 7C<;E84F43D6DBC ':8<>C>+  %8BDB4+A4=3B8=(08A43!=CA0=B8C8E40=3+A0=B8C8E4-4A1B><?0A8B>=>5CF>
2>A?>A05A><38554A4=CC8<4?4A8>3B  (#(')'#)&" #&$"' "))(#"   )8IE8* 0DC784A D:DA>E0% %0EA8:8B%  G0<8=8=64=34A8554A4=24B8=0<4
0B43$40A=8=6+7A>D67#+(0A0<4C4ABC8<0C8>= != "(&"(#"  #"&"#"&( "( "" )(#"??    *C0<?4A" #>438=64A# %2;0D67;8= "D;H 2><?0A8B>=>5<>34;B4;42C8>=<4CA82B8=
30C0B7>? != )(#"  ("" *C0<?4A"  #>438=64A# ) "D=4  D<0=<0278=4BCD34=C<>34;38B2>E4AH0=3
8<?A>E4<4=CDB8=60C0*7>? !=&(  "( "" )(#"??   
+4A0<DA0    >;;42C8>=>5G0<?;4B>5"0?0=4B4%8BDB41H>A486=$40A=4AB'B0:0
,=8E4AB8CH(E4AB8>=&0C8>=0;!=BC8CDC45>A"0?0=4B4$0=6D0640=3$8=6D8BC82B
.411% ><0=  !<?02CB>55;8??432;0BBA>><B>=;40A=4A0CC8CD34BC>F0A3B
C427=>;>6H4=70=243;0=6D064;40A=8=6  #!$)(&'''(")&""  
 


768

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Effect of Timing Differences in Online
Corrective Feedback on Adult Verbal
English Learners' Learning Engagement: A
Micro-genetic Study
Wanying LIANGa, Guang CHENb* & Wei CHENGc
a
Faculty of Psychology, Beijing Normal University, Chinab
School of Educational Technology, Faculty of Education, Beijing Normal University, Chinac
School of Educational Science and Technology, Nanjing University of Posts andTelecommunications, China*guang@bnu.edu.cn
Abstract: This study examines the impact of different timing direct corrective feedback
strategies utilized by instructors in online verbal English class on the learning
engagement of EFL adult learners. The study randomly divided 8 participants with thesame English proficiency into two groups, one receiving immediate direct corrective
feedback and the other receiving delayed direct corrective feedback. Applying a microgenetic method, the study observed the online learning of the 8 participants andimplemented two types of direct corrective feedback through online learning. The study
found that both feedback methods could improve learners' learning engagement, but
immediate direct corrective feedback had a more positive effect on learners' learning
engagement, and could improve it in a shorter amount of time. Additionally, learners'
learning engagement showed dynamic changes during the learning process due toother influencing factors.
Keywords: Micro-genetic method, immediate corrective feedback, delayed corrective
feedback, learning engagement, online learning

1. Introduction
With the rise of globalization and international communication, more and more adults arelearning English as a foreign language (EFL). EFL adult learners worldwide, particularly inthe area of speaking, face various challenges. These challenges include difficulties inpronunciation, grammar, vocabulary, and cultural differences from their native language.
Among this group of learners, feedback teaching strategies, including immediate anddelayed corrective feedback, are widely used by teachers, and their effectiveness has been
extensively studied. However, different scholars have reached varying conclusions regarding
the impact of these two feedback strategies on students' learning. Some scholars believethat immediate corrective feedback is more helpful for learners' English language learning(Sakiroglu, 2020), while others argue that delayed corrective feedback is more effective(Nassaji & Kartchava, 2017). Similarly, some scholars have conflicting conclusions about the
impact of different timing of corrective feedback strategies on learners' emotional states(Estaji & Farahanynia, 2019; Sarifah, 2019; Shabani & Safari, 2016a). These divergent
research findings may stem from individual differences and the diversity of influencing
factors, which further illustrates the complexity of the factors influencing learners'
acceptance of feedback methods. In previous relevant research, the predominant traditional
approach often centered on pretest-posttest methodologies, typically involving
measurements before and after interventions. However, this approach didn't capture thechanges in students' involvement during the learning process濁 Learning engagement is acomplex process, and a new research method is needed to understand it better. The micro-

769

genetic method could be the solution to this gap in research. It's considered a good
approach because it allows researchers to closely observe and analyze individual behaviors
over time. It can help researchers gain a deeper understanding of the behavior and thoughtprocesses of EFL adult learners and provide more targeted learning support for them
(Siegler & Crowley, 1991). On the other hand, learner engagement includes cognitive,
emotional, and behavioral engagement (Fredricks et al., 2004). It can help observe changes
in learner engagement under different feedback strategies, which can comprehensively
understand the effectiveness of learners' learning, providing more guidance for teachers' useof teaching strategies. The fact that there hasn't been much research using the microgenetic method to study how different feedback approaches affect learners' engagement
highlights a gap in our knowledge. Clearly, studying the impact of various corrective
feedback methods on how engaged learners are using the micro-genetic approach is very
important and valuable for research.

2. Literature Review
2.1 The Impact of Types of Direct Corrective Feedback on EFL Learners
Direct corrective feedback refers to the teacher directly pointing out learners' errors andproviding correct expressions. It can help learners correct pronunciation and grammaticalerrors and improve their verbal expression ability (Harmer, 2007). In addition, there are twomain types of direct corrective feedback. The first is focused corrective feedback, whichusually only corrects learners' errors in specific language targets or language skills to avoid
overloading learners with too much information and unnecessary interference (Mackey &Goo, 2007; Nassaji & Swain, 2000; Sheen, 2007). The second is comprehensive corrective
feedback, which refers to correcting students' language errors so that they can discover andcorrect errors on their own, thereby promoting their autonomous learning and language
proficiency (Bitchener & Knoch, 2009; Ellis, 2009; Lyster & Ranta, 1997). Studies haveshown that comprehensive corrective feedback has a positive effect on the learning
performance of L2 learners (Cahyono, 2016; Van Beuningen et al., 2012) and can alsosignificantly improve their learning interest and motivation, while focused corrective feedback
has no significant effect (Falhasiri, 2021; Hartshorn & Evans, 2015). Therefore, this study willuse direct comprehensive corrective feedback as the main feedback teaching strategy toobserve the changes in learning engagement of EFL verbal English adult learners.

2.2 The Effects of Direct Corrective Feedback at Different Time Intervals on EFL
Learners
The direct corrective feedback teaching strategy can be divided into two temporaldimensions: immediate direct corrective feedback and delayed direct corrective feedback.
Immediate direct corrective feedback refers to correction given immediately after the learnerproduces an error, while the learner is still in the process of expressing themselves. Delayed
direct corrective feedback, on the other hand, refers to correction given after the learner has
completed the sentence or paragraph (Lyster & Ranta, 1997). Some studies have found that
immediate direct corrective feedback can improve L2 learners' learning performance (Li etal., 2016; Tesnim, 2019). However, other studies have found that delayed corrective
feedback can be more effective in improving L2 learners' fluency and accuracy of oral
production (Hunter, 2012; Rahimi & Dastjerdi, 2012).
In addition to learning outcomes, the effects of direct corrective feedback at different
time intervals also have different impacts on learners' emotional and affective states. Somescholars have found that learners receiving delayed feedback experience more negative
emotions during the learning process, such as anxiety, frustration, and embarrassment(Quinn, 2014). However, other researchers have found that delayed feedback can help
learners regulate emotions, improve learning outcomes, and have a positive impact on their770self-worth and self-esteem (Estaji & Farahanynia, 2019). On the other hand, some studies
have shown that immediate corrective feedback can help learners correct errors faster, andimprove their affective state during the learning process, such as confidence and optimism(Yakisik, 2021). Interestingly, some scholars have drawn opposite conclusions. Immediate
feedback may have negative effects on L2 learners’ confidence and produce negative
emotion although it can help L2 learners improve their language proficiency (Shabani &Safari, 2016).
In summary, the effects of immediate and delayed corrective feedback on learners'
emotions and affective states are complex and depend on various factors, such as individual
learners, different subject areas, and learning environments.

2.3 The Application of Micro-genetic Method in Studying Learning Engagement
Micro-genetic method is a real-time analysis method for studying learning, which
decomposes the learning process into small time units and analyzes the interaction between
learners and the learning environment during these units. This method allows researchers to
identify patterns and micro-changes in learner behavior, understand the basic mechanismsof learning, and study real-time dynamic processes in the learning process (Siegler &Crowley, 1991). Learning engagement refers to the degree to which learners actively
participate in the learning process, including behavioral, emotional, and cognitive
engagement (Fredricks et al., 2004). In recent years, micro-genetic methodology has been
increasingly used to study the learning engagement of linguistic learners. Kaivanpanah andMiri (2018) used micro-genetic methodology to study changes in task-induced involvementof 24 Iranian EFL learners over an 18-month observation and recording period. The study
found that cognitive engagement was influenced by multiple factors, such as task type,evaluation and teacher instructional style. In addition, emotional engagement predicted
learners' learning engagement. Jiang(2019) explored the changes in learning engagement
and influencing factors of 12 EFL learners over a 10-week task interaction period using
micro-genetic methodology. The results showed that learners' learning engagement
fluctuated multiple times during the task process and was influenced by task type, language
input and output, and other factors. Furthermore, learners generated many questions during
the learning process, which promoted their learning engagement.
Therefore, learners are influenced by various factors during the learning process,leading to dynamic changes in learning engagement. Observing the trajectory of changes inlearners' engagement during the learning process can provide a comprehensive
understanding of their learning situation, thereby facilitating the search for more effective
teaching strategies.

3. Research Question
Based on the current research, most studies examining the effects of different timing forcorrective feedback have primarily focused on the writing skills of EFL learners, whileneglecting the exploration of oral proficiency development. Moreover, the application of
micro-genetic methods to investigate changes in learner engagement among EFL learners
remains largely unexplored. Therefore, this paper aims to use the micro-genetic method toobserve changes in EFL adult learners' engagement in online learning environments andanalyze differences in their engagement trajectories after receiving direct corrective
feedback at different timing. The goal is to gain a deeper understanding of the dynamic
impact of these two types of feedback on learning engagement among adult EFL learners,
and to compare the differences between them, so that more effective feedback approaches
can be identified for instructional practices. To achieve this research objective, the following
research questions are proposed:
1. Are there differences in the impact on EFL adult learners' learning engagement between
immediate direct corrective feedback and delayed direct corrective feedback teachingstrategies?

771

2. In the teaching strategies of immediate direct corrective feedback and delayed direct
corrective feedback, what are the specific trajectories of the impact on EFL adult learners'
individual learning engagement?

4. Research Design
4.1 ParticipantsThis study recruited eight participants whose English proficiency level was at theinternational A2 elementary level, as defined by the Council of Europe (Council of Europe,2001). All participants had graduated from university several years ago and were engaged indifferent professions but had a personal need to improve their verbal English skills. Amongthe participants, there were two male and six female participants, ranging in age from 25 to
45 years old. The participants were randomly assigned to two experimental groups: one
group received immediate direct corrective feedback instruction, and the other group
received delayed direct corrective feedback instruction.

4.2 Instruments
4.2.1 Learning Engagement Scale
The Online Student Engagement Scale (OSE) (Cronbach’s alpha = .95, p<.01) (Dixson,
2015) will be used in this study to measure students' learning engagement. This scaleconsists of four dimensions: skill engagement, emotion engagement, participation
engagement, and performance engagement. Skill engagement refers to students' cognitive
engagement in the learning process. Emotion engagement relates to students' emotional
experiences and attitudes towards learning. Participation engagement pertains to the extent
of students' involvement in online courses and tasks. Performance engagement reflects
students' performance and grades in learning tasks. The scale comprises 19 questions anduses a 5-point Likert scale, ranging from "strongly disagree" to "strongly agree." As theexperimental instruction in this study is one-on-one, the questions related to peer interactionin the participation dimension were modified to focus on interaction with the teacher.
4.2.2 Verbal English Task
The International English Language Testing System (IELTS) speaking exam is acomprehensive test that assesses test-takers' verbal proficiency. Studies have shown that
IELTS speaking scores demonstrate high validity in predicting English language ability andactual performance (Li & Zhang, 2018). This study employed the first section of theCambridge IELTS Speaking test for one-on-one online oral instruction. Each session lasted
for 30 minutes, encompassing three distinct topics. Each topic consisted of four questions,with the primary instructional emphasis on grammar structures and oral expressions
pertinent to the IELTS Speaking test.
4.2.3 Procedure
In this study, 8 participants were randomly divided into two groups: the immediate direct
corrective feedback group and the delayed direct corrective feedback group. Fourparticipants were assigned to each group. The study consisted of four one-on-one online
oral English training sessions, which were conducted via Tencent Meeting. Prior to theexperimental training, all participants received training in English sentence structure andrelated experimental instructions for the first part of the IELTS speaking test. During theexperimental training, the examiner asked the participants English questions and they
responded verbally. For the immediate direct corrective feedback group, the examiner
immediately interrupted and corrected the participants when they made expression errors,772and then required them to repeat the corrected sentence before continuing with their
response until the end of the question. For the delayed direct corrective feedback group, theexaminer noted down the errors made by the participants during the oral response process
and provided comprehensive corrective feedback after the completion of the entire topic. In
addition, the examiner asked the participants to re-answer the questions. Written homeworkassigned by the teacher was completed before the start of the next class. At the end of eachsession, the participants completed a survey on their learning engagement and recorded avideo of their responses to specific questions. The experimental training consisted of one
session per week for four weeks.

4.3 Data Analysis
The online verbal English task in this study was conducted through Tencent Meeting
software, allowing video recording. Participants' learning engagement scores were collected
through questionnaire responses, and further data collection on the reasons for changes inlearning engagement was carried out using observational methods and self-reporting by theparticipants. Subsequently, the micro-genetic method was applied to encode and categorize
the overall changes in learning engagement of the participants over time, facilitating a morecomprehensive understanding of trends in engagement growth and decline. This method
enabled an in-depth investigation into the impact of direct corrective feedback teachingstrategies on participants' learning engagement performance at different time intervals. The
data analysis in this study was mainly conducted using the Excel 2021 version and the
SPSS 27.0 version software. The learning engagement scores of the participants in the two
experimental groups are presented and analyzed as below.
Table 1. 澸濙濧濗濦濝濤濨濝濪濙澔濇濨濕濨濝濧濨濝濗濧澔濣濚澔激濙濕濦濢濝濢濛澔澹濢濛濕濛濙濡濙濢濨澔濇濗濣濦濙濧澔濖濭澔澺濙濙濘濖濕濗濟澔濈濭濤濙Feedback Type
Immediate Direct Correct
Delayed Direct Correct

M
73.5
70.25

MD
8.12
6.23

Based on the average scores presented in Table 1, the immediate direct corrective feedback
group had higher scores in learning engagement, while the delayed direct corrective
feedback group had lower scores. Preliminary analysis suggests that immediate direct
corrective feedback may have a more positive impact on participants' learning engagement
than delayed direct corrective feedback. Additionally, there were differences in the standard
deviation and standard error between the two groups, indicating that the learning
engagement scores in the immediate direct corrective feedback group were more stable,
while those in the delayed direct corrective feedback group were more variable. Further
analysis using the micro-genetic approach will be conducted to validate these findings.

5. Results and Discussion
5.1 Comparison of Mean Score Changes in Learning Engagement for Two Direct
Corrective Feedback Methods
As shown in Figure 1, the micro-analysis method was used to compare the average scoresof learning engagement between the immediate and delayed direct corrective feedback
groups in each class. The immediate direct corrective feedback group (3.25, 2.33, 2.67,2.75) had slightly higher score changes throughout the process than the delayed direct
corrective feedback group (1.25, -0.25, 4, 3), and the change trend was relatively stable.
However, the score changes in the delayed direct corrective feedback group fluctuated in thefirst two classes, but increased significantly in the third and fourth classes, with averagescore changes higher than the immediate direct corrective feedback group throughout the773

process. This indicates that the learning engagement of the immediate direct corrective
feedback group is more stable than that of the delayed direct corrective feedback group in
the learning process, while the delayed direct corrective feedback group's learning
engagement improves significantly in the later stages of learning.
It can be seen that the immediate and delayed direct corrective feedback groups
showed completely different trends in the average score changes of learning engagement ineach class. The learning engagement of the immediate direct corrective feedback groupsteadily increased as the course progressed, while that of the delayed direct corrective
feedback group fluctuated and was unstable. These data results may be related to thecharacteristics of the feedback modes of the immediate and delayed corrective feedback
groups. The immediate direct corrective feedback group corrected students' errors in atimely manner and provided corresponding suggestions, which may help students makemore correct responses in subsequent learning. However, the delayed direct corrective
feedback group may not make students perceive their errors in a timely manner, leading tolonger-lasting responses to errors and adverse effects on learning engagement.

Figure 1. Average score changes in learning engagement between two different
corrective feedback groups.

5.2 Comparison of Rate Changes in Learning Engagement for Two Direct Corrective
Feedback Methods
As shown in Figure 2, a microscopic analysis of the change rate reveals that the immediate
direct corrective feedback group exhibited a higher change rate in the first two sessions,particularly with a change rate of 0.14 between the first and second session, while thedelayed direct corrective feedback group showed a more stable change rate throughout theprocess with minimal fluctuations.
From this, it can be inferred that the immediate direct corrective feedback groupdemonstrated a gradually increasing trend in learning engagement scores throughout thecourse, with a higher change rate in the first two sessions. In contrast, the delayed direct
corrective feedback group also showed improvement in learning engagement throughout thecourse, but with a more stable change rate. These findings suggest that immediate direct
corrective feedback is more effective in producing positive effects on students' learning
engagement in a shorter amount of time, while delayed direct corrective feedback may
require a longer learning process to achieve significant improvement.

774

Figure 2 Rate of changes in learning engagement between two different corrective
feedback groups.

5.3 Variations in Individual Learning Engagement for Two Direct Corrective
Feedback Methods
5.3.1 Individual Learning Engagement Changes in Immediate Direct Corrective Feedback
As illustrated in Figure 3, it demonstrates a general trend of improvement in the learning
engagement of the four research participants over the course of the four sessions.
Specifically, the first participant exhibited a score increase from 83 to 88, with relatively
consistent change patterns. In the interview, the participant reported that the immediate
direct corrective feedback facilitated a more profound understanding of incorrect language
points in their verbal expression, thereby enhancing their motivation to learn, as evidenced
by the reduced frequency of errors and a sense of achievement. Participant 2 demonstrated
significant changes in scores between the first and second sessions, but subsequently
displayed relatively stable changes, ultimately achieving a score of 68. During the interview,
this participant indicated that the experiment had a positive impact on their English verbal
skills. However, due to the establishment of a new company, they lacked the energy to
devote to English learning and experienced interruptions from work during class. Participant
3's score continued to rise steadily throughout the four sessions, increasing from 69 to 81.
The participant reported that their work frequently involved the use of English and that theimmediate direct corrective feedback helped them to deepen their understanding of
language points, thus increasing their motivation to learn by reducing errors in verbal
expression. Participant 4 demonstrated relatively stable score changes during the first three
sessions but exhibited significant improvement during the fourth session, with a scoreincrease from 77 to 84. During the interview, the participant stated that they initially felt
pressure and lacked confidence when receiving instant direct corrective feedback, which
resulted in hesitancy to express themselves. However, after adapting to this feedback
method, their perspective changed, and they recognized it as an integral part of the learning
process. They gained more confidence as they encountered and addressed challenges,
ultimately becoming more willing to invest additional effort into learning.

Figure 3 Individual changes in learning engagement for immediate direct corrective
feedback group.

775

5.3.2 Individual Learning Engagement Changes in Delayed Direct Corrective Feedback
As shown in Figure 4, the learning engagement scores of the fifth participant decreased from74 to 65 between the first and fourth lessons. During the interview, Participant 5 reported
having to work on a project that required a report to be submitted at the end of theexperiment, resulting in overtime work and interruptions during the lessons due to workrelated phone calls. This left the participant with insufficient energy to focus on learning. The
participant had enough time for English language learning in the early stages of theexperiment and perceived an improvement in language proficiency. However, the participant
recognized the need to dedicate more time to learning. Participant 6 showed an increase inlearning engagement scores for the first three lessons and a decrease to 63 in the fourth
lesson. During the interview, Participant 6 reported feeling that their English verbal
proficiency had improved with delayed and immediate corrective feedback. However, during
the final week, the participant was occupied with end-of-quarter preparations for annual
summaries and meetings, leaving little energy for language learning. Participant 7 showed aslight decrease in learning engagement scores for the fourth lesson, with a change from 64
in the first lesson to 70 in the fourth lesson. The participant reported being less confident
during the first lesson, which led to reluctance to speak, but as the course progressed, theparticipant gained more confidence and willingness to learn. However, the participant's
workload increased during the final week, leading to a reduction in English language learning
time. Participant 8 showed a continuous increase in learning engagement scores across all
four lessons, with a change from 69 in the first lesson to 79 in the fourth lesson. During thefirst lesson, Participant 8 expressed doubts about the effectiveness of delayed and
immediate corrective feedback and believed that learning English grammar systematicallywould be more meaningful. However, in the subsequent lessons, the participant found that
their English verbal expression and language proficiency had gradually improved, resulting
in greater learning motivation and willingness to dedicate more time to learning.

Figure 4 Individual changes in learning engagement for delayed direct corrective
feedback group.
5.3.3 Comparison of Changes in Learning Engagement of Individuals for Two Corrective
Feedback Methods
Overall, there was a positive trend in the learning engagement of all participants, indicatingthat both types of direct corrective feedback had a positive impact on their learning
engagement. However, immediate direct corrective feedback had a more significant positiveeffect. Notably, the significant decrease in learning engagement in the final week observedin participants 5 and 6 was attributed to work-related factors, whereas the continuous
increase in learning engagement in participants 1, 3, and 4, and 8 was influenced by theirlearning motivation and sense of achievement. These findings suggest that learners'
learning engagement can be influenced by various factors, such as learning motivation,
work, and cognitive awareness, etc., resulting in fluctuating changes in their learning776

engagement. However, given the limited sample size of this study, further research andvalidation are necessary to generalize these conclusions.

6. Conclusion
This study compared the effects of immediate corrective feedback and delayed corrective
feedback on learners' learning engagement and found that both types of feedback
significantly increased learning engagement. However, the immediate corrective feedback
group had higher scores in learning engagement than the delayed corrective feedback
group. The changes in learning engagement also differed between the two groups, with theimmediate corrective feedback group showing more significant changes than the delayedcorrective feedback group, which is consistent with previous research findings. Masià andRamon's study (2013) found that immediate feedback can improve learning engagement
and learning outcomes. Meanwhile, Winstone et al. (2017) emphasized the importance oftimely feedback for student learning. The results of this study suggest that immediate
corrective feedback can improve the average score of learning engagement and can change
learning engagement in a relatively short period. However, delayed corrective feedback isalso effective in improving learning engagement, especially for students who need more timeto understand and digest information. Additionally, in the process of learners' learning,learning engagement can be influenced by other factors, especially for working EFL adultlearners who are affected by work-related factors. Learners should reflect on their learning
situation in a timely manner and plan for frequent learning in their free time to improve theirlearning engagement and learning outcomes. Teachers should also consider otherinfluencing factors in the teaching process and provide timely feedback to students so thatthey can understand their learning situation and make corresponding adjustments.
Future research can increase the sample size to further verify the generalizability of
the conclusions. Additionally, it is possible to further explore the micro-level effects ofdifferent types of feedback on various sub-dimensions of learners' learning engagement. On
the other hand, it is also possible to investigate how different types of feedback affectlearners' learning engagement in different scenarios, such as small or large classes.
Furthermore, it is possible to further investigate the effects of different types of feedback on
the learning outcomes and learning engagement of learners with different personalities.
These studies will help to better understand the role of feedback in student learning andprovide better support for improving student learning outcomes.

References
Bitchener, J., & Knoch, U. (2009). The relative effectiveness of different types of direct written corrective
feedback. System, 37(2), 322–329.
Cahyono, B. (2016). The Efficacy of Comprehensive Corrective Feedback in Improving Grammatical
Accuracy of EFL Learners’ Writing Pariyanto Universitas Islam Negeri Sunan Ampel Surabaya
Indonesia. International Journal of Language and Linguistics, 3(6), 51–59.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning,teaching, assessment. Cambridge University Press.
Dixson, M. D. (2015). Measuring student engagement in the online course: The online student
engagement scale (OSE). Online Learning, 19(4), n4.
Ellis, R. (2009). Corrective feedback and teacher development. L2 Journal, 1(1), 3–18.
Estaji, M., & Farahanynia, M. (2019). The Immediate and Delayed Effect of Dynamic Assessment
Approaches on EFL Learners’ Oral Narrative Performance and Anxiety. Educational
Assessment, 24(2), 135–154.
Falhasiri, M. (2021). Is Less Really More? The Case for Comprehensive Written Corrective Feedback.
Canadian Journal of Applied Linguistics, 24(3), 145–165.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept,
state of the evidence. Review of Educational Research, 74(1), 59–109.
Harmer, J. (2007). How to teach English. Pearson Education Limited.

777

Hartshorn, K. J., & Evans, N. W. (2015). The effects of dynamic written corrective feedback: A 30-week
study. Journal of Response to Writing, 1(2), 2.
Hunter, J. D. (2012). A multi-method investigation of the effectiveness and utility of delayed corrective
feedback in second-language oral production [PhD Thesis]. University of Birmingham.
Jiang, Y. (2019). A microgenetic analysis of task-based language learning: An investigation of changes
in engagement over time. Innovation in Language Learning and Teaching, 13(2), 167–181.
Kaivanpanah, S., & Miri, M. (2018). Inspecting Task-Induced Involvement from the Perspective ofSociocultural Theory. Journal of Teaching Language Skills, 37(1).
Li, S., & Zhang, S. (2018). Investigating the validity of IELTS speaking test scores: A mixed-methods
study. Language Testing, 35(4), 529–554.
Li, S., Zhu, Y., & Ellis, R. (2016). The Effects of the Timing of Corrective Feedback on the Acquisitionof a New Linguistic Structure: The Modern Language Journal. The Modern Language Journal,
100(1), 276–295.
Lyster, R., & Ranta, L. (1997). Corrective feedback and learner uptake: Negotiation of form incommunicative classrooms. Studies in Second Language Acquisition, 19(1), 37–66.
Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In
A. Mackey (Ed.), Conversational interaction in second language acquisition: A series of
empirical studies (pp. 407–452). Oxford University Press.
Nassaji, H., & Kartchava, E. (2017). Corrective feedback in second language teaching and learning:
Research, theory, applications, implications. Routledge, Taylor & Francis Group.
Nassaji, H., & Swain, M. (2000). A Vygotskian perspective on corrective feedback in L2: The effect ofrandom versus negotiated help on the learning of English articles. Language Awareness, 9(1),34–51.
Quinn, P. (2014). Delayed versus immediate corrective feedback on orally produced passive errors in
English [PhD Thesis]. University of Toronto Toronto.
Rahimi, A., & Dastjerdi, H. V. (2012). Impact of immediate and delayed error correction on EFL learners’
oral production: CAF. Mediterranean Journal of Social Sciences, 3(1), 45–54.
Sakiroglu, H. Ü. (2020). Oral Corrective Feedback Preferences of University Students in English
Communication Classes. International Journal of Research in Education and Science, 6(1),
172–178.
Sancho-Vinuesa, T., Escudero-Viladoms, N., & Masià, R. (2013). Continuous activity with immediate
feedback: A good strategy to guarantee student engagement with the course. Open Learning:
The Journal of Open, Distance and e-Learning, 28(1), 51–66.
Sarifah, I. (2019). Thai Efl Learners’ Preferences on Delayed Versus Immediate Corrective Feedback
in English Learning: A Case Study on a Primary School in Songkhla, Thailand. Bogor English
Student and Teacher (BEST) Conference, 1, 146–151.
Shabani, K., & Safari, F. (2016a). Immediate vs Delayed Correction Feedback (CF) and Accuracy ofOral Production: The Role of Anxiety. Theory & Practice in Language Studies, 6(11).
Shabani, K., & Safari, F. (2016b). Immediate vs Delayed Correction Feedback (CF) and Accuracy ofOral Production: The Role of Anxiety. Theory and Practice in Language Studies, 6(11), 2222.
Sheen, Y.-C. (2007). The effect of focused written corrective feedback and language aptitude on ESL
learners’ acquisition of articles. TESOL Quarterly, 41(2), 255–283.
Siegler, R. S., & Crowley, K. (1991). The micro-genetic method: A direct means for studying cognitive
development. American Psychologist, 46(6), 606–620.
Tesnim, O. (2019). Oral Corrective Feedback and Its Impact on Learners’ Speaking Skills: Tunisian
EFL Students as a Case Study. International Journal of Language and Linguistics, 7(3), 138.
Van Beuningen, C. G., De Jong, N. H., & Kuiken, F. (2012). Evidence on the effectiveness ofcomprehensive error correction in second language writing. Language Learning, 62(1), 1–41.
Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic
engagement with feedback: A systematic review and a taxonomy of recipience processes.
Educational Psychologist, 52(1), 17–37.
Yakisik, B. Y. (2021). EFL Learners’ Preferences and Emotions about Oral Corrective Feedback at
Secondary Education in Turkey: Are There Gender and Grade-Level Differences? Indonesian
Journal of English Language Teaching and Applied Linguistics, 6(1), 103–122.

778

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Using Self-Regulated Digital Storytelling inPrimary Students’ English Learning: An
Exploratory Factor Analysis
Yunsi Tina MAa,b*, Siu-Cheung KONGa & Daner SUNa
Department of Mathematics and Information Technology,bCentre for Learning, Teaching and Technology, The Education University of Hong Kong,
Hong Kong*yma@eduhk.hkaAbstract: Successful learners are cultivated by good learning habits and essential
skills. Among them, self-regulated learning (SRL) is proven to be an effective learning
strategy and pedagogical approach to develop independent learning, enhance learning
outcomes, improve learning motivation, and finally achieve lifelong success. In
language education, Digital Storytelling (DST) has been an emerging strategy forteachers to guide their students to learn a language by expressing ideas and meanings.
However, little research has been done to explore the effectiveness of such integration
on students’ language learning. The present study designed and developed a selfregulated digital storytelling (SRDST) curriculum for teachers to teach and students tolearn English writing. About 110 primary grade 4 students of 9-10 years old participatedin this study. A SRDST scale with 13 items in forethought, performance, and reflection
phases was developed and validated. The results of exploratory factor analysis (EFA)
from the primary students (N=70) on a pre-/post-test design yielded three factors thatwere similar to the three phases of SRL, namely, forethought, performance andreflection stages. The paired-samples t-test results indicated a significant increase inforethought, suggesting that the students realized certain preparation needs to be done
before their study, and they also preferred autonomy on managing their own learning.
Further, the data demonstrated a significant result in the performance phase where thestudents can seek for help when encountering difficulties, and determine to keep up
with the overall learning progress and complete the tasks on time. The results also
revealed that the students tended to reflect on whether there were better ways toimprove their final projects. Finally, the research found that young learners preferred to
set learning goals during the performance stage which is different from adult learners.
Keywords: Self-regulated learning (SRL), digital storytelling, English learning,
instruments, exploratory factor analysis (EFA)

1. Introduction
Self-regulated learning (SRL) is not only a learning strategy to support student learning but
also a pedagogy to facilitate teacher teaching. SRL can be used as a stand-alone pedagogy
or combined with other pedagogical approaches to facilitate student-centred learning (Barrett,2006). While many researchers investigated the effectiveness of SRL in higher education andadult education both offline and online in the past years (Carter Jr et al., 2020; Jansen, 2019;
Vanslambrouck et al., 2019;), more studies have been probing the impact of SRL in online or
blended learning mode after the outbreak of COVID-19 pandemic when researchers andeducators uphold the importance of SRL abilities. Previous studies argued that SRL can
improve young learners’ learning outcomes, learning strategies and motivation (McClelland &Cameron, 2011; Hung et al., 2012; Järvelä et al., 2012). Dignath et al. (2008) pointed out thatprimary school students can also acquire SRL and benefit from it. Researchers believe that
SRL can foster sustainable lifelong learning skills as SRL can cultivate learners’ genericabilities, such as problem-solving skills, digital competencies and learning autonomy(Anthonysamy et al., 2020; Lüftenegger et al., 2012). Recently, the application of SRL in

779

language learning is emerging, however, limited studies focused on primary EFL/ESL
students’ language learning guided by SRL approach.
In the last four decades, educators and researchers probed the impact of digitalstorytelling as a pedagogy in various subject disciplines across primary, secondary, and higher
education levels (Wu & Chen, 2020). Digital Storytelling (DST) is a technology-support
strategy for teachers to guide their students to learn a foreign or second language byexpressing ideas and meaning through integrating the use of audio, texts, and visual artefacts
to develop language ability and communication skills (Wang & Zhan, 2010; Torres et al.,2012). Besides, it is also an innovative narrative tool for promoting digital competencies andrelevant skills (Del-Moral-Pérez et al., 2019). Moreover, DST can also be a learning strategy
for improving learners’ academic achievement, problem-solving skills, learning motivation andcollaboration skills (Hung et al., 2012; Niemi & Multisilta, 2016). In the field of language
learning, DST has proved to be one of the most popular and effective strategies for improving
students English writing (Yang & Wu, 2012; Burke & Kafai, 2010). However, the research on
innovative pedagogies or tools for facilitating primary students’ DST is limited. Furthermore,aiming to equip students with digital competence in language learning, the current study aims
to explore whether the SRDST enabled by technologies is effective in grooming young
learners’ digital competency development. Thus, for addressing the above issues, theintegration of programming tools in DST, for instance, Scratch, will have great potential for
engaging students in SRL learning environment.

2. Literature Review
2.1 Self-regulated Learning
Zimmerman (2002) concluded that self-regulation is neither a mental ability nor an academic
performance skill. Instead, it is a process that learners self-direct themselves to transform theirmental capabilities into academic skills and achievements. Self-regulated learners can apply
multiple strategies including cognitive strategies (e.g. organization and elaboration),
metacognitive strategies (e.g. planning and monitoring), resource management strategies(e.g. time management), and collaborative learning to improve their learning (Bai et al., 2021).
Self-regulated learning can also develop positive thinking by guiding students to have an
optimistic future planning which can guide them to overcome obstacles and succeed
academically (Zimmerman, 2002). Therefore, SRL is essential for grooming lifelong learning
skills in a digital world (Anthonysamy et al., 2020). Further, Loyens et al. (2008) found that the“self” aspect is important in the developmental processes of SRL. Therefore, emphasizing
student-centred learning also aligns with the gist of SRL.
Riding on Bandura’s social cognitive theory (1986) interplaying person, behavior andenvironment, researchers developed and entailed various notions into SRL. Pintrich (2004)
added metacognition and social context to the SRL framework and expanded the regulating
behaviors to the forethought phase. Forethought includes task analysis and self-motivation
belief, performance includes self-control and self-observation, and self-reflection includes selfjudgement and self-reaction (Zimmerman, 2008). Meanwhile, Schunk (1990) pointed out thatstudents shall be taught to set realistic upper and lower goals to enter learning activities forgoal attainment with self-efficacy. In the SRDST process, the students have little idea about
the activities in the forethought period, therefore, the current research attempted to put goal
setting under the early performance stage so that the students can set and adjust the upper
goals or main goals, and lower goals or sub-goals throughout the performance stages. There
have been various SRL models working on Zimmerman’s three phases, namely, forethought,
performance, and self-reflection (Panadero, 2017). In our study, we adopted an adapted
Zimmerman’s phases (Zimmerman & Moylan, 2009) entailing different self-regulated
behaviours aiming to propose one framework that fits the primary school level. In theframework for the primary level, the forethought phase entails environment, learner autonomy
and strategic planning, performance phase entails goal-setting, time management, selfefficacy, help-seeking and progress monitoring, and the reflection phase entails self-reflection

780

with or without peer assessment, learner engagement, improvement and well-being (Figure
1).

Figure 1. SRL framework proposed by the current study for primary levels based on
Zimmerman and Moylan’s framework (2009)

2.2 Self-regulated Digital Storytelling
DST facilitates a series of student-centred learning strategies, namely, student engagement,
reflection for deep learning, project-based learning and effective integration of technology withlearning and teaching by using technologies in a meaningful way in classroom settings(Barrett, 2006; Stale & Freeman, 2017). Wu and Chen (2020)’s systematic review on DST for
educational purposes revealed that DST had been applied in various levels of educations
ranging from primary, secondary to higher education levels as either a stand-alone pedagogy
or in combination with other pedagogies, usually in culture, gender, language, social
psychology, and social studies and 21st century skills among which language learning is themain subject where DST is employed. Lim et al. (2022) observed that little research offereddetailed definitions of related research topics, concepts and methods of digital storytelling forlanguage learning. Therefore, the current study proposes an SRDST framework and model toguide the research activities aiming to fill in the research gaps and probe the impacts of
SRDST on EFL/ESL primary students’ learning in English writing (Ma et al., 2022). In gist,
SRDST is a digital storytelling approach steered by SRL for enhancing learning outcomes andimproving learning motivation.
A few studies found that digital storytelling on Scratch had a positive impact onenhancing learner’s motivation and academic achievement in English learning. Digitalstorytelling on Scratch offers middle school learners the opportunity to better understand theprocess of developing a story from an idea to a digital story (Burke & Kafai, 2010; Parsazadeh
et al., 2021). Smith and Burrow (2016) suggested ways to integrate DST into the classroom
to support reasoning, creative thinking and problem-solving. By partnering with teachers fromlanguage arts, technology and math subjects, Wolz et al. (2011)’s study revealed that teachers
can infuse computational thinking in language learning in technology-enhanced classroom
settings so as to enrich the language arts curriculum. Burke and Kafai (2012) affirmed that
there is a great deal of learning potential in this intersection of the formal writing practices
taught during the school-day and the “informal” activities of digital creation.

2.3 Background of the SRDST Scale
Previous research attempted to develop and validate SRL scale embedded with other learning
strategies in different contexts while mainly in the realm of higher education (Roth et al., 2016).
Barnard et al. (2009) developed a six-dimension questionnaire to measure SRL in online or
hybrid learning modes demonstrating environment structuring, goal setting, time
management, help-seeking, task strategies, and self-evaluation. Although this six-dimension

781

questionnaire is slightly different from Zimmerman’s three phases by subdividing forethought
and performance phases, the current research adopted and adapted both frameworks as they
match the results and findings from the previous curriculum design using a design-based
research approach (Ma et al., 2023; Zimmerman, 2002).
Pintrich (2004) and Magno (2011) developed a framework to classify the phases andrelevant scales based on the Motivated Strategies for Learning Questionnaire (MSLQ)(Pintrich et al., 1991) in higher education. Retnawati (2016) developed an SRL instrument
consisting of a Likert scale and multiple-choice questions for college students who majored inmathematics education. Based on their self-efficacy for self-regulated learning scale forelementary to high school learners, Usher and Pajares (2008) found that elementary students
attained higher self-efficacy for self-regulated learning than those in higher levels. Roth andher colleagues (2016) also reported that since 1980s there had been an emerging
development and deployment of subject-specific questionnaires over the past decades.
Besides, they also claimed that MSLQ seemed to be the most verified instrument in SRL
research, however, the MSLQ may not be able to assess post-performance behaviours suchas self-reflection and regulation. Riding on the literature review and findings, the present
research developed a SRDST scale to evaluate students’ performance and improvement inlearning English writing guided by SRDST strategies.

2.4 Research Purposes and Research Questions
The current research, on the one hand, aims to develop and evaluation an innovative
pedagogy, i.e. SRDST, for facilitating primary students’ DST in English learning, on the other
hand, it aims to equip students with digital competence in language learning by deploying the
SRDST strategy. To achieve these research objectives, the current study attempts to address
the following research questions (RQs):(1) What are the factors related to SRDST as perceived by the primary 4 students? Are they
similar to the factors identified in the literature?
(2) Do the students perceive an improvement in the factors after attending the SRDST for
English learning curriculum?

3. Method
3.1 Participants
About 110 EFL/ ESL Chinese students of 9 – 10 years old from a Hong Kong local primaryschool participated in this study. The Human Research Ethics Committee of The Education
University of Hong Kong considered and approved this study by reviewing the researchmethods, procedures and instruments prior to the start of the research activities. The
participating students provided approved written consent from their parents. The students
were divided into one experimental group and one control group. The experimental group were
instructed based on SRDST approach while the control group were instructed based on paperformed worksheets and presentations slides.

3.2 The Self-Regulated Digital Storytelling Curriculum
Figure 2 demonstrates the rationales of the lesson design and the cycling process of the
SRDST approach. The SRDST emphasizes the three phases of SRL: forethought,
performance I and II and reflection to guide students to acquire the vocabulary, grammar andwriting skills. Guided by SRDST approach, Table 1 presents a lesson plan for a primary grade
4 English Language unit “We Love Festivals” where SRDST strategies were embedded
throughout the curriculum and teaching and learning resources. The experiment was
conducted in four weeks in the first semester of 2022/23 academic year.The development of782

the curriculum followed a design-based research approach and was guided by the SRDST
innovative pedagogy proposed by this research (Ma et al., 2023).

Figure 2. Using SRDST approach to learn and teach the P4 English Language unit “WeLove Festivals”
Table 1. A lesson plan of the P4 English Language unit “We Love Festivals”
LessonDuration*
Activities
Pre-test (30 mins)
Lesson 0
25 mins
Vocabulary
Lesson 1
45 mins
Grammar 1
Homework
Lesson 1 Take-home Assignment
Lesson 2
45 mins
Grammar 2
Homework
Lesson 2 Take-home Assignment
Lesson 3
45 mins
Writing: To create your own story
Homework
Lesson 3 Take-home Assignment
Lesson 4
45 mins
Writing: To create your own story
Homework
Lesson 4 Take-home Assignment
Lesson 5
45 mins
To create the story on Scratch
Lesson 6
45 mins
To create the story on Scratch
Lesson 7
45 mins
Presentation
Post-test (30 mins)
Total
340 mins

RemarkFlipped Learning
English Class
Homework
English Class
Homework
English Class
Homework
English Class
Homework
Computer Class
Computer Class
English Class

3.3 Instruments and Procedure
Riding on the literature review, the current study developed and employed an SRDST scalewith 13 items of forethought, performance and reflection to evaluate students’ performanceand improvement in learning English writing guided by SRDST strategies. In this study, thesamples would be considered as two equivalent groups because (1) the distribution of gender,
age, and class size were similar, (2) the teachers, textbooks, and time spent on the English
classes for both groups were all the same, and (3) other important confounding variables,
including students’ DST experience, years of learning English, coding experience on Scratch
were not significantly different based on the results of a chi-square test for homogeneity.

4. Results
4.1 Descriptive Statistics
Among the 110 participants, 70 completed both the pre- and post-tests and regarded as thedata sample. The sample students consisted of 38 females and 32 males from primary grade
4 whose ages ranged from 9-10 years old (M = 9.21; SD = .41). The English language learning783

experience of the students ranged from 6 to 7 years. About half of the students have somecoding experience on Scratch and the other half has no coding experience at all (M = .51; SD= .50). It is assumed that there is no significant difference between the two groups of students.
The descriptive statistics of the two groups are presented in Table 2. The chi-square test of
independence showed that there was no significant difference between the two groups interms of gender, X2 (1, N = 70) = 1.490, p = .22, age, X2 (1, N = 70) = 1.775, p = .18, andexperience of coding, X2 (1, N = 70) = .525, p = .47. Besides, the English teachers’ year ofexperience were also similar. Therefore, any differences between the experimental group andthe control group could be regarded as independent variables in the current research.
Table 2. Descriptive Statistics of the Two Groups
N = 70All Students(N = 70)
Control(n = 34)
Experimental(n = 36)

GenderAge

Coding Experience
Mean
SD

Mean

SD

Mean

SD

.46

.50

9.2

.41

.51

.50

.38

.49

9.1

.35

.56

.50

.53

.50

9.3

.45

.47

.50

Note. N = Gender (1 = male; 0 = female); Coding Experience (1 = Yes; 0 = No)

4.2 Reliability Test and Factor Analysis
A Cronbach’s alpha reliability test was conducted to measure the reliability of the SRDST scalewith the pre-test data. According to the test results presented in Table 3, the reliability of thequestionnaire was robust (Cronbach’s α = .858). Its 13-item scale examined three phases of
SRDST in an English writing class: a) forethought (three items); b) performance (six items);
and c) reflection (four items). It was measured using a 5-point Likert scale, ranging from 1,‘Strongly disagree’ to 5, ‘Strongly agree’. In this study, the reliability of the scale wassatisfactory (forethought: Cronbach’s α = .610; performance: Cronbach’s α = .822; reflection:
Cronbach’s α = .822) as presented in Table 4.
To explore the factor structure of the SRDST instrument, a factor analysis was conducted on
the pre-test data collected from the students (N = 70) of the 13 items using SPSS, with
Principal Component Analysis and Varimax with Kaiser Normalization as the extraction androtation methods, respectively, which yielded three factors that were similar to the threephases of self-regulated learning, namely, forethought, performance and reflection stages, as
shown in Table 4. The sample size met the minimum requirement of delivering an EFA (Bryant& Yarnold, 1995; Winter et al., 2009). The Kaiser–Meyer–Olkin value was 0.83, and the chisquare value for Bartlett’s test of sphericity was 375.11 (df = 78, p < .001), indicating that thethree factors had good explanatory power. The total variance accounted for by these threefactors was 62.79%. The factor loadings of the items are presented in Table 4.
Table 3. Reliability and factor loading of the SRDST scale
Self-regulated Digital Storytelling Scale (N = 70)
SRDST Scale
Forethought
1.
I know where and when I can learn English most efficiently.
2.
I consult or discuss with someone when I need help in my learning.
3.
I have the autonomy to achieve my learning goals in my own way.
Performance4.
I proactively seek for relevant information to understand the selfregulated digital storytelling activities on Scratch if I have difficulties.
5.
I am clear about the main learning goals of self-regulated digitalstorytelling on Scratch.
6.
I have set sub-goals (e.g. read the digital stories on Scratch on timeand complete the worksheets) leading to the main learning goals.
7.
I have enough time to learn English.
8.
I make sure I can keep up with the overall progress of self-regulated
digital stories on Scratch for learning English (e.g. read the digitalstories on Scratch on time and complete the worksheets).

784

Mean

SD

Factorloading

2.81
3.17
3.01

1.572
1.296
1.367

.697
.756
.633

3.14

1.289

.646

Cronbach’s
α
.858
.610
.876
.861
.840
.822
.848

3.39

1.376

.694

.850

3.24

1.408

.767

.842

3.07
3.16

1.220
1.519

.576
.664

.847
.839

9.

I can schedule a suitable time to read the digital stories on Scratch forlearning English.
Reflection
10.
I receive guidance on how to achieve the goals in stages.
11.
I enjoy the parts of Scratch digital stories and worksheets that lead me
to reflect on my English learning.
12.
I will reflect on what I have learned after I finish the self-regulated digitalstorytelling activities.
13.
After I finish the self-regulated digital storytelling activities, I will reflect
on whether there are better ways of creating my digital stories.

3.20

1.336

.762

.846

3.20
2.97

1.336
1.251

.850
.640

.822
.855
.842

3.14

1.311

.758

.840

3.11

1.368

.734

.839

Note. A 5-point Likert scale is used, where 5 = “strongly agree”, and 1 = “strongly disagree”. *p ≤ .05; **p ≤ .01; ***p ≤ .001

4.3 Paired-samples T-test
Among the participants, 52 of them completed the whole experiment process by attending all
the learning activities and finished the pre-/post-test surveys. The others who were absent
from one or some classes were excluded from the paired-samples T-test analysis. The means,standard deviations and t-values are presented in Table 4.
Table 4. Means, standard deviations and t-values of the Paired-samples T-Test
Self-regulated Digital Storytelling Scale (N = 52)
Forethought
1.
I know where and when I can learn English mostefficiently.
2.
I consult or discuss with someone when I need help in
my learning.
3.
I have the autonomy to achieve my learning goals inmy own way.
Performance4.
I proactively seek for relevant information tounderstand the self-regulated digital storytelling
activities on Scratch if I have difficulties.
5.
I am clear about the main learning goals of selfregulated digital storytelling on Scratch.
6.
I have set sub-goals (e.g. read the digital stories on
Scratch on time and complete the worksheets) leading
to the main learning goals.
7.
I have enough time to learn English.
8.
I make sure I can keep up with the overall progress of
self-regulated digital stories on Scratch for learning
English (e.g. read the digital stories on Scratch ontime and complete the worksheets).
9.
I can schedule a suitable time to read the digitalstories on Scratch for learning English.
Reflection
10.
I receive guidance on how to achieve the goals instages.
11.
I enjoy the parts of Scratch digital stories andworksheets that lead me to reflect on my English
learning.
12.
I will reflect on what I have learned after I finish theself-regulated digital storytelling activities.
13.
After I finish the self-regulated digital storytelling
activities, I will reflect on whether there are betterways of creating my digital stories.

Pre-test
Mean
SD

Post-test
MeanSD

t-value

2.670

1.642

3.620

1.207

-3.424**

3.080

1.296

3.830

1.115

-2.819**

2.920

1.326

3.710

1.126

-3.308**

3.060

1.211

3.790

1.304

-2.919**

3.330

1.396

3.460

1.335

-0.555

3.170

1.410

3.270

1.315

-0.467

3.060
2.960

1.243
1.533

3.440
3.420

1.145
1.194

-1.853-2.299*

3.100

1.347

3.190

1.253

-0.437

3.130

1.329

3.290

1.242

-0.704

2.790

1.160

3.120

1.215

-1.761

3.080

1.296

3.270

1.254

-0.919

3.100

1.376

3.630

1.189

-2.442*Note. A 5-point Likert scale is used, where 5 = “strongly agree”, and 1 = “strongly disagree”. *p ≤ .05; **p ≤ .01; ***p ≤ .001

5. Discussions
According to the factor analysis results, we calculated the pre-test and post-test means of thethree factors and carried out a paired-samples T-test to examine whether students havesignificant improvements after the completion of the course. The t-test results of the threefactors, together with those of individual items, were as shown in Table 4. On the one hand,
the results showed a significant increase in forethought, suggesting that the students realizedcertain preparation needs to be done before their study. With learning activities facilitated bytechnologies and internet, the students learned to find a suitable place and a good time to785

attend the activities at home. SRDST approach encouraged students to seek for help when
they were in need. They also appealed for autonomy in managing their own learning andachieving their goals in their own ways. SRL has a positive impact on learners’ autonomy andunder some conditions can support learners’ autonomy (Papamitsiou & Economides, 2019;
Yamauchi & Tanaka, 1998). Sierens et al. (2009) argued “when teachers want their students
to evaluate themselves, to plan their study activities, and to think about themselves as learners,
the teachers are encouraged to provide help, instructions, and expectations in an autonomysupportive way.” On the other hand, the data demonstrated a significant result in theperformance phase where the students can seek relevant information when they encounter
difficulties and determine to keep up with the overall progress for learning English andcomplete the tasks on time. The abilities and habits of keeping up with the learning progress
are very critical to ensure learning effectiveness (Hong et al, 2021).
Although Zimmerman’s framework (2002) puts goal setting in the forethought phase, inthe current study, it is found that the young learners set and adjusted sub-goals during theperformance phase to guide them to achieve the main learning goals. The importance of such
behaviours was echoed by many studies (Huh & Reigeluth, 2017; Kizilcec et al., 2017; Schunk,1990). Huh and Reigeluth (2017) argued that goal setting is an iterative procedure in SRL thatwas supported by overarching self-efficacy and self-motivation. Kizilec et al. (2017) highlighted
that goal seeing and strategic planning positively predict academic achievements. Further, theresults also revealed that with self-reflection, peer assessment rubrics and teachers’ feedback
provided, the students were willing to reflect on whether there were better ways to improvetheir final projects, namely their English writing, storyboard and their own digital stories. It isbelieved that self-reflection can be cultivated when reflection prompts and feedback fromteachers and peers are provided to engage students in self-reflection (Masui & De Corte, 2005;
Schunk & Zimmerman, 1998; Van den Boom et al, 2004, 2007).

6. Conclusion, Limitations and Future Studies
In the study, an innovative pedagogy, namely SRDST, was employed in primary school
students’ formal English learning. In order to understand its effectiveness, an instrument witha factor analysis yielding three factors 1) forethought. 2) performance 3) reflection wasdeveloped. The factors were similar to the ones identified in the literature (Zimmerman, 2002).
Based on the SRL framework for guiding teaching and learning, a curriculum with teaching
and learning resources were developed for implementing SRDST in English learning. The ttest results suggested a significant improvement in the forethought phase where the students
believed they know where and when they can learn English most efficiently, they reckonedthat they can consult others when they are in need, and they possessed the autonomy toachieve learning goals in their own ways. Besides, the students managed to monitor their own
progress as long as clear guidelines and guidance were given to them. Further, after theyfinished the first draft of English writing and their first digital stories based on their own English
writing storyboards, they would reflect on whether there were better ways for refinement
according to the self-reflection rubrics as well as the feedback from teachers and peers. The
positive results revealed the feasibility of using SRDST to teach and learn English language.
Further, the SRDST has much potential in innovating teaching and learning. For example, thispedagogy can be used in other disciplines, e.g. science, history, for steering the developmentof curriculum resources. Besides, this learning strategy can be taught to young learners forcultivating good learning habits, developing digital competencies, and preparing them forbecoming lifelong learners in a digital world.
Admittedly, this research may have some limitations due to the limited sample size. First,
the experiment time only lasted for 3 weeks and covered only one English unit due to theschool’s class planning and limited curriculum space. A longer duration with more suitabletopics to be covered would be ideal for further exploration and later a broader implementation.
Second, the current study was conducted in one primary school and the sample size only met
the minimum requirement of delivering an exploratory factor analysis (Bryant & Yarnold, 1995).
To better understand the relationship between students’ gender, interests and motivations, a786

large sample size with participants from different schools was desired to examine the factor
structure and access the effectiveness of this pedagogy and learning strategy. Third, theteachers’ perspectives on the SRDST pedagogy were not clear. Further studies need to bedone by organizing in-depth teacher and student interviews, class observation and content
analysis on students’ works.

References
Anthonysamy, L., Koo, A. C., & Hew, S. H. (2020). Self-regulated learning strategies in higher education: Fostering
digital literacy for sustainable lifelong learning. Education and Information Technologies, 25(4), 2393-2414.
Bai, B., Wang, J., & Zhou, H. (2021). An intervention study to improve primary school students’ self-regulated
strategy use in English writing through e-learning in Hong Kong. Computer Assisted Language Learning, 123.
Bandura, A. (1986). Fearful expectations and avoidant actions as coeffects of perceived self-inefficacy. American
Psychologist, 41(12), 1389-1391.
Barnard, L., Lan, W. Y., To, Y. M., Paton, V. O., & Lai, S. L. (2009). Measuring self-regulation in online and blended
learning environments. The internet and higher education, 12(1), 1-6.
Barrett, H. (2006, March). Researching and evaluating digital storytelling as a deep learning tool. In Society for
information technology & teacher education international conference (pp. 647-654). Association for the
Advancement of Computing in Education (AACE).
Bryant, F. B., & Yarnold, P. R. (1995). Principal-components analysis and exploratory and confirmatory factor
analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 99–
136). Washington, DC: American Psychological Association.
Burke, Q., & Kafai, Y. B. (2010, June). Programming & storytelling: opportunities for learning about coding &composition. In Proceedings of the 9th international conference on interaction design and children (pp. 348351).
Burke, Q., & Kafai, Y. B. (2012, February). The writers' workshop for youth programmers: digital storytelling withscratch in middle school classrooms. In Proceedings of the 43rd ACM technical symposium on Computer
Science Education (pp. 433-438).
Carter Jr, R. A., Rice, M., Yang, S., & Jackson, H. A. (2020). Self-regulated learning in online learning environments:
strategies for remote learning. Information and Learning Sciences.
de Winter*, J. C., Dodou*, D. I. M. I. T. R. A., & Wieringa, P. A. (2009). Exploratory factor analysis with small sample
sizes. Multivariate behavioral research, 44(2), 147-181.
Del-Moral-Pérez, M. E., Villalustre-Martínez, L., & Neira-Piñeiro, M. D. R. (2019). Teachers’ perception about thecontribution of collaborative creation of digital storytelling to the communicative and digital competence inprimary education schoolchildren. Computer Assisted Language Learning, 32(4), 342-365.
Dignath, C., Buettner, G., & Langfeldt, H. P. (2008). How can primary school students learn self-regulated learning
strategies most effectively?: A meta-analysis on self-regulation training programmes. Educational Research
Review, 3(2), 101-129.
Hong, J. C., Lee, Y. F., & Ye, J. H. (2021). Procrastination predicts online self-regulated learning and online learning
ineffectiveness during the coronavirus lockdown. Personality and individual differences, 174, 110673.
Huh, Y., & Reigeluth, C. M. (2017). Self-regulated learning: The continuous-change conceptual framework and avision of new paradigm, technology system, and pedagogical support. Journal of Educational Technology
Systems, 46(2), 191-214.
Hung, C. M., Hwang, G. J., & Huang, I. (2012). A project-based digital storytelling approach for improving students'
learning motivation, problem-solving competence and learning achievement. Journal of Educational
Technology & Society, 15(4), 368-379.
Jansen, R. S., Van Leeuwen, A., Janssen, J., Jak, S., & Kester, L. (2019). Self-regulated learning partially mediates
the effect of self-regulated learning interventions on achievement in higher education: A metaanalysis. Educational Research Review, 28, 100292.
Järvelä, S., Järvenoja, H., & Malmberg, J. (2012). How elementary school students' motivation is connected to selfregulation. Educational Research and Evaluation, 18(1), 65-84.
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner
behavior and goal attainment in Massive Open Online Courses. Computers & education, 104, 18-33.
Lim, N. Z. L., Zakaria, A., & Aryadoust, V. (2022). A systematic review of digital storytelling in language learning inadolescents and adults. Education and Information Technologies, 1-31.
Loyens, S. M., Magda, J., & Rikers, R. M. (2008). Self-directed learning in problem-based learning and its
relationships with self-regulated learning. Educational psychology review, 20(4), 411-427.
Lüftenegger, M., Schober, B., Van de Schoot, R., Wagner, P., Finsterwald, M., & Spiel, C. (2012). Lifelong learning
as a goal–Do autonomy and self-regulation in school result in well prepared pupils?. Learning andInstruction, 22(1), 27-36.
Ma, Y. T., Kong, S.C., & Sun, D. (2022, November). Using Digital Storytelling on Scratch to Support Primary School
EFL/ESL Students’ Writing: A Self-regulated Learning Approach. In S. Iyer, et al. (Eds.), Proceedings of the
30th International Conference on Computers in Education ICCE 2022 (pp. 714-718). Kuala Lumpur, Malaysia:
Asia-Pacific Society for Computers in Education.

787

Ma, Y. T., Kong, S.C., & Sun, D. (2023, May). Design and Develop a Scratch-Enabled Digital Storytelling
Curriculum for Primary Students in Hong Kong: A Design-Based Research. In Chen, H., et al. (Eds.),
Workshop Proceedings of the 27th Global Chinese Conference on Computers in Education GCCCE 2023
(pp. 285-291). Beijing, China: Global Chinese Society for Computers in Education.
Magno, C. (2011). Validating the academic self-regulated learning scale with the motivated strategies for learningquestionnaire (MSLQ) and learning and study strategies inventory (LASSI). The International Journal of
Educational and Psychological Assessment, 7(2).
Masui, C., & De Corte, E. (2005). Learning to reflect and to attribute constructively as basic components of selfregulated learning. British Journal of Educational Psychology, 75(3), 351-372.
McClelland, M. M., & Cameron, C. E. (2011). Self-regulation and academic achievement in elementary school
children. New directions for child and adolescent development, 2011(133), 29-44.
Niemi, H., & Multisilta, J. (2016). Digital storytelling promoting twenty-first century skills and student
engagement. Technology, Pedagogy and Education, 25(4), 451-468.
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in
psychology, 422.
Papamitsiou, Z., & Economides, A. A. (2019). Exploring autonomous learning capacity from a self-regulated
learning perspective using learning analytics. British Journal of Educational Technology, 50(6), 3138-3155.
Parsazadeh, N., Cheng, P. Y., Wu, T. T., & Huang, Y. M. (2021). Integrating computational thinking concept into
digital storytelling to improve learners’ motivation and performance. Journal of Educational Computing
Research, 59(3), 470-495.
Pintrich, P. R., Smith, D., Garcia, T., & McKeachie, W. (1991). A manual for the use of the Motivated Strategies for
Learning Questionnaire (MSLQ), The University of Michigan, Ann Arbor, MI.
Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college
students. Educational psychology review, 16, 385-407.
Retnawati, H. (2016). Proving content validity of self-regulated learning scale (The comparison of Aiken index andexpanded Gregory index). REiD (Research and Evaluation in Education), 2(2), 155-164.
Roth, A., Ogrin, S., & Schmitz, B. (2016). Assessing self-regulated learning in higher education: A systematic
literature review of self-report instruments. Educational Assessment, Evaluation and Accountability, 28, 225250.
Schunk, D. H. (1990). Goal setting and self-efficacy during self-regulated learning. Educational psychologist, 25(1),71-86.
Schunk, D. H., & Zimmerman, B. J. (1998). Self-Regulated Learning: From Teaching to Self-reflective Practice.
New York: Guilford Press.
Sierens, E., Vansteenkiste, M., Goossens, L., Soenens, B., & Dochy, F. (2009). The synergistic relationship ofperceived autonomy support and structure in the prediction of self-regulated learning. British Journal of
Educational Psychology, 79(1), 57-68.
Smith, S., & Burrow, L. E. (2016). Programming multimedia stories in Scratch to integrate computational thinking
and writing with elementary students. Journal of Mathematics Education, 9(2), 119-131.
Staley, B., & Freeman, L. A. (2017). Digital storytelling as student-centred pedagogy: empowering high school
students to frame their futures. Research and Practice in Technology Enhanced Learning, 12(1), 1-17.
Torres, A. R., Ponce, E., & García, P. M. D. (2012). Digital storytelling as a pedagogical tool within a didactic
sequence in foreign language teaching. Digital Education Review, 2012, vol. 22, p. 1-18.
Usher, E. L., & Pajares, F. (2008). Self-efficacy for self-regulated learning: A validation study. Educational andpsychological measurement, 68(3), 443-463.
Van den Boom, G., Paas, F., Van Merrienboer, J. J., & Van Gog, T. (2004). Reflection prompts and tutor feedback
in a web-based learning environment: Effects on students' self-regulated learning competence. Computers in
Human Behavior, 20(4), 551-567.
Van den Boom, G., Paas, F., & Van Merrienboer, J. J. (2007). Effects of elicited reflections combined with tutor or
peer feedback on self-regulated learning and learning outcomes. Learning and Instruction, 17(5), 532-548.
Vanslambrouck, S., Zhu, C., Pynoo, B., Lombaerts, K., Tondeur, J., & Scherer, R. (2019). A latent profile analysisof adult students’ online self-regulation in blended learning environments. Computers in Human Behavior, 99,
126-136.
Wang, S., & Zhan, H. (2010). Enhancing teaching and learning with digital storytelling. International Journal of
Information and Communication Technology Education (IJICTE), 6(2), 76-87.
Wolz, U., Stone, M., Pearson, K., Pulimood, S. M., & Switzer, M. (2011). Computational thinking and expository
writing in the middle school. ACM Transactions on Computing Education (TOCE), 11(2), 1-22.
Wu, J., & Chen, D. T. V. (2020). A systematic review of educational digital storytelling. Computers & Education,
147, 103786.
Yamauchi, H., & Tanaka, K. (1998). Relations of autonomy, self-referenced beliefs, and self-regulated learning
among Japanese children. Psychological reports, 82(3), 803-816.
Yang, Y. T. C., & Wu, W. C. I. (2012). Digital storytelling for enhancing student academic achievement, criticalthinking, and learning motivation: A year-long experimental study. Computers & education, 59(2), 339-352.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into practice, 41(2), 64-70.
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodologicaldevelopments, and future prospects. American educational research journal, 45(1), 166-183.
Zimmerman, B. J., & Moylan, A. R. (2009). Self-regulation: Where metacognition and motivation intersect.
In Handbook of metacognition in education (pp. 311-328). Routledge.

788

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Constructing a Natural Conversation
Learning Activity to Improve Students’English-Speaking Skills
Hong-Min TUa*, Chang-Yen LIAOb, Te-Yang CHOUc
Graduate Institute of Network Learning Technology, National Central University, TaiwanbDepartment of Hakka Language and Social Sciences, National Central University, TaiwancGraduate Institute of Network Learning Technology, National Central University, Taiwan*benjamintu1514@gmail.com
aAbstract: As a medium of communication, language encapsulates four corecompetencies: listening, speaking, reading, and writing. Notably, speaking, whichfacilitates direct communication, is critical. Yet, Taiwan's English pedagogy heavily
emphasizes reading, inadvertently fostering a deficit in speaking skills among students.
Addressing this, the current study proposes an English conversational practice systemthat adheres to language acquisition tenets, integrating 10-minute mini-lessons forEnglish listening and speaking exercises. In these activities, students utilize a natural
approach, engaging with the system-provided English conversational audio to refine
their listening and speaking skills. The system, furnished with a recording function,
empowers students to engage in self-paced English listening and speaking practiceduring their free time. The process of "listening to system audio, followed by speaking
practice (recording), and re-recording" offers students corrective feedback from theirrecordings, enhancing their pronunciation, grammar, and fluency - vital elements of
conversational competency. Additionally, the system incorporates a voice-based
chatbot, facilitating situational speaking exercises based on the practice topics. The
overarching goal of this study is to instill in students the skill to transfer their learning todiverse conversational contexts they might encounter, fostering habitual speaking to
mitigate the disparity in the development of the four English language skills.
Keywords: conversation learning, mini-lessons, language acquisition, voice recording,natural approach

1. Introduction
The English education system in Taiwan has been overly focused on the "reading" skill(understanding English article content and grammar structures), which has resulted in
Taiwanese students performing significantly better in "reading" skills than in "speaking" and"listening" skills (National Development Council of Taiwan, 2021).
Our language skills are acquired through the acquisition, while "acquisition" is a part ofthe learning process, referring to learning non-deliberately or unconsciously (Mani, 2016).
Children acquire their native language skills through constant exposure to conversations indaily life (Clark, 2018). Similarly, students' speaking skills develop through listening to a large
amount of external input in daily life and imitating what they hear and understand.
Based on the above, our aim is to develop an English conversation practice system forstudents that provides a conducive language learning environment and designs 10-minutemini-lessons. This will allow them to acquire English naturally through the imitation of theprocess of native language learning (Krashen & Terrell, 1983). This system enables students
to engage in "self-paced English listening and speaking practice" or "situational conversation
practice with a voice-based chatbot", using tablets or mobile devices during their free time.
Afterwards, students can listen to different conversation topics and practice listening
and speaking independently through the "listening to system audio, followed by speaking
practice (recording), and re-recording" mode. By utilizing recordings, students can improve789

their pronunciation, grammar, fluency, intonation, and more, thereby increasing theirconfidence and courage to use English in spoken interactions (YanJu, Mei & Mohamed, 2017).
Voice-based chatbots enable students to practice spoken language, meeting their self-paced
needs (Chen, Vicki Widarso & Sutrisno, 2020), and are not limited by time or location (Han,2020). Additionally, these chatbots provide students with an authentic language environment,
contributing to enhancing their speaking skills (Huang, Hew & Fryer, 2022), ultimately forming
the habit of speaking English.

2. System Introduction
In order to improve the current English education system in Taiwan, which finds it difficult tocultivate students' speaking skills in English class, the design philosophy of this study system
is based on the "Natural Approach" (Krashen & Terrell, 1983) and "Interest-Driven Creator
Theory" (Chan et al., 2018), using the habit loop's “cuing environment” and "routine activities".
This allows students to immerse themselves in a language acquisition environment
constructed by the English conversation practice system, and by being exposed to significant
amounts of English language input, cultivates students' speaking skills and develops the habitof speaking English.
The research team has compiled 36 English conversation learning topics, including
socializing at school, family interaction, and time concepts. They have collected related
English conversation content for each topic, which has been categorized into five levels ofdifficulty. So far, the system has collected approximately 2,000 English conversations.
In the mini-lessons, teachers can select an English conversation topic that matches
their students’ English level and learning progress from the English conversation topic page.
The blackboard page displays the English text and Chinese text (including Mandarin phonetic
symbols) of the conversation content, and also allows for the playback of the English
conversation audio. This enables students to practice speaking along with the English
conversation content they hear.
This system enables students to engage in "self-paced English listening and speaking
practice" or "situational conversation practice with a voice-based chatbot", using tablets ormobile devices during their free time. The voice-based chatbot will employ OpenAI as theconversation partner for students to practice speaking with. However, as OpenAI is based on
text-based artificial intelligence, Microsoft Azure Speech Services will be utilized as thecommunication bridge between the learners and OpenAI. The process of situational
conversation practice will begin with OpenAI presenting a topic related to the theme, and then
the text will be converted to speech via Azure Speech Services and played back. Students will
listen to the content and provide oral responses, which they will record. The recorded speech
will be converted back to text using Azure Speech Services and sent to OpenAI. Through thisprocess, students can apply the conversations they have practiced to real-life scenarios,
thereby enhancing their speaking skills.

3. Preliminary Evaluation
3.1 Research participants
The research participants for this study were 55 students from a certain experimental primaryschool in Taoyuan city, Taiwan, ranging from grades three to six. Of the participants, 24 wereboys and 30 were girls. Other participants included researchers and the English teachers fromeach class.

3.2 Preliminary Results and Conclusions
The objective of this study is to examine the impact of English listening and speaking
exercises, implemented through mini-lessons and self-paced activities, on the oral proficiency

790

and habits of elementary students in grades three to six. Observations within the classroom
setting have demonstrated that, with teacher guidance, students actively participate in English
listening and speaking practice activities. Interestingly, after engaging in these mini-lessons,students show a growing willingness to communicate in English with their educators andclassmates during the sessions.
Initial findings of this study suggest that English speaking and listening practice,delivered through mini-lessons and self-paced activities, positively impacts students'
inclination and habits to communicate in English. As the research is set to continue over thefull semester, we will maintain our classroom observations and accumulate data on students'
midterm and final English grades. At the end of semester, we intend to carry out an Englishspeaking competency assessment, utilizing evaluative criteria from a preceding study on
English speaking performance (West Hartford Public Schools, 2012). This assessment willencompass six dimensions: "accuracy," "comprehensibility and pronunciation," "fluency,""comprehension," "content," and "maturity of the language."

Acknowledgements
This study was funded by the National Science and Technology Council of Taiwan (112-2410H-008 -020 -MY3) and by Research Center for Science and Technology for Learning, National
Central University, Taiwan.

References
Clark, E. V. (2018). Conversation and language acquisition: A pragmatic approach. Language
Learning and Development, 14(3), 170-185.
Chan, T. W., Looi, C. K., Chen, W., Wong, L. H., Chang, B., Liao, C. C., ... & Ogata, H. (2018).
Interest-driven creator theory: Towards a theory of learning design for Asia in the twentyfirst century. Journal of Computers in Education, 5(4), 435-461.
Chen, H. L., Vicki Widarso, G., & Sutrisno, H. (2020). A chatbot for learning Chinese: Learningachievement and technology acceptance. Journal of Educational Computing
Research, 58(6), 1161-1189.
Han, D. E. (2020). The effects of voice-based AI chatbots on Korean EFL middle school
students’ speaking competence and affective domains. Asia-pacific Journal ofConvergent Research Interchange, 6(7), 71-80.
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really
useful? A systematic review of chatbotƮsupported language learning. Journal of Computer
Assisted Learning, 38(1), 237-257.
Krashen, S. D., & Terrell, T. (1983). Natural approach (pp. 20-20). New York: Pergamon.
Mani, R. S. (2016). The natural approach. Journal of English Language And Literature(JOELL), 3(1), 8-16.
National Development Council (2021). Bilingual 2030. National Development Council.
https://www.ndc.gov.tw/en/Content_List.aspx?n=BF21AB4041BB5255
West Hartford Public Schools. (2012). World Language Department’s middle school speaking
rubric.
YanJu, S., Mei, S. Y., & Mohamed, Y. (2017). Implementation of voice recording activities in
improving Mandarin oral fluency. European Journal of Interdisciplinary Studies, 3(4), 5661.

791

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on
Computers in Education. Asia-Pacific Society for Computers in Education

Analysis of topic sentences ofclassification paragraphs anddevelopment of a diagnostic function
Kandai ISHIKAWAa*, Hidenobu KUNICHIKAba
Graduate School of Computer Science and Systems Engineering,
Kyushu Institute of Technology, Japanb
Faculty of Computer Science and Systems Engineering,
Kyushu Institute of Technology, Japan*k.ishikawa@minnie.ai.kyutech.ac.jp
Abstract: In a paragraph, the topic sentence is an important component that clearly
expresses the author's argument and indicates the direction of the entire paragraph. It
is difficult for beginners to learn English to write appropriate topic sentences. In orderto solve the problem, we have focused on classification paragraphs and developed adiagnostic function of the topic sentences. As the result of an evaluation of the function,
we have found that the function correctly diagnosed 70.6% of appropriate topic
sentences taken from text books for paragraph writing.
Keywords: Support in English Language Learning, Paragraph Writing, Topic
Sentence, Classification Paragraph

1. Introduction
Paragraph writing is one of writing techniques for expressing opinions andarguments. In a paragraph, the topic sentence is the most important component that clearly
expresses the author's argument and indicates the direction of the entire paragraph.
Moreover, a well-written topic sentence determines whether a paragraph can be written well.
There are some conditions for writing topic sentences, and they differ according to types ofparagraphs. In order to write paragraphs appropriately, it is necessary to understand suchrequirements, but beginners of English learning often lack sufficient knowledge. Therefore,
we focus on supporting the writing of topic sentences.
There are some studies on assessing writing; e.g., (Su, Lin & Lai 2023). Althoughsuch systems will give learners feedback that is somewhat vague and abstract, they will not
be able to generate concrete feedback according to types of paragraphs. In order to solve
the problem, we are aiming to implement a diagnostic function of topic sentences as one of
the functions of an English language paragraph writing support system. In this study, we
focus on classification paragraphs, and have implemented a prototype of a diagnostic
function of topic sentences (Ishikawa & Kunichika, 2023). This paper describes development
and an evaluation of the diagnostic function.

2. Topic Sentences of Classification Paragraphs
Classification paragraphs classify things, that correspond to the topic of theparagraph, into several groups according to their characteristics or features. A topic
sentence basically consists of two parts: a topic that indicates what the entire paragraph isabout, and an assertion that expresses the author's argument or opinion on the topic. In thisstudy, we have investigated the conditions of assertion in topic sentences of classified
paragraphs by referring to textbooks on paragraph writing, and classified the conditions into
three types. Also, we have categorized the forms of topic sentences into two types. The

792

three types of conditions differ in their applicability depending on the form of a topic
sentence. The forms of topic sentences and the conditions of assertions are as follows.
(Form 1) The topic is appeared as the subject of a topic sentence.
The subject of a topic sentence corresponds to the topic, and is followed by a verb used
for classification; e.g., “Popular music can be divided into three types."(Form 2) An expletive is used as the subject of a topic sentence.
An expletive such as “there” is used as the subject of a topic sentence, and is followed bya be-verb; e.g., "There are three kinds of skiing."(Condition 1) The verb of a topic sentence needs to mean classification.
This condition is applied to only Form 1. The topic sentence of a classification paragraphshould indicate that the topic is classified into several groups. So, the verb of a topic
sentence needs to mean classification. There are some frequent verbs such as "bedivided into” and “fall into" (Kitao & Kitao, 1998).
(Condition 2) A criterion for classification needs to be provided.
A criterion for classification needs to be clearly stated in a topic sentence for classifying a
topic based on the criterion in a paragraph. There are frequent words which express
criteria; e.g., type, kind and category (Kitao & Kitao, 1998).
(Condition 3) The number of classified categories need to be indicated.
The details of each category are presented in supporting sentences of a paragraph.
Therefore, the number of classified categories needs to be indicated in a topic sentence(Donald et al.,1998). There are two ways to express the number of items: one is toindicate both the criteria for classification and the number of categories, e.g., "divided into
three types", and the other is to list the name of each category, e.g., "classified as A andB."

3. Development of a Diagnostic Function of Topic Sentences
Our diagnostic function analyzes a topic sentence by referring to the result of word
dependency analysis performed by CoreNLP (Stanford, 2022). Figure 1 shows the result ofword dependency analysis of “Popular music can be divided into three types.” Then, thefunction diagnoses if a topic sentence satisfies all the conditions. The diagnosis of topic
sentences consists of two steps: identifying the form of a topic sentence and judging theconditions of the assertion of the topic sentence. First, for identifying the form of a topic
sentence, the function confirms whether or not the main verb, which is shown as a root in the
output from CoreNLP, is a be-verb and whether or not the subject is an expletive. Next, thefunction judges each condition of an assertion. For the judgment, the function extracts
important words from the result of word dependency analysis performed by CoreNLP.
Important words correspond to the main verb, a criterion for classification, and the number ofclassified categories. If the topic sentence is “Form 1”, the function confirms whether or not
the verb means classification by referring to a list of frequent verbs (e.g., Howard & Tracz,
1976; Donald et al.,1998) for condition 1. For condition 2, the function confirms whether or

Figure 1.An Example of the Result of Word Dependency Analysis performed by
CoreNLP

793

not a word corresponding to the object of the topic sentence can be used for a criterion ofclassification by referring to a list of frequent words which express criteria (e.g., Howard &Tracz, 1976; Donald et al.,1998). For condition 3, the function confirms whether or not there
are numeral modifiers for the words which express criteria or whether or not the names ofcategories are listed. Finally, the function outputs the result of the diagnosis.
When an input sentence satisfies all the conditions, the function shows the format of
the topic sentence, the verb which means classification, the criteria for the classification, andthe number of classified categories. Figure 2 shows an example of the result of an
acceptable sentence. If an input sentence does not satisfy the conditions, the function shows
the conditions that are not satisfied.



Figure 2.An Example of the Result of an Acceptable Sentence (Ishikawa & Kunichika,2023)

4. AnEvaluation of the Diagnostic Function of Topic Sentences
As an evaluation, we prepared 17 topic sentences of classified paragraphs taken from eighttextbooks (e.g., Howard & Tracz, 1976; Donald et al.,1998; Oshima & Hogue 2017) on
paragraph writing, and used the sentences as input for the diagnostic function. As the result,the system correctly diagnosed 12 (70.6%) of the 17 sentences as appropriate, while the other5 sentences were judged as unacceptable sentences. We have found two patterns ofsentences that were misdiagnosed: one is omitting the criteria for classification and the otheris listing categories in clauses or sentences connected by a conjunction.

5. Conclusion
In this study, we have developed a diagnostic function of the topic sentences forclassification paragraphs. Also, this paper has presented an evaluation of the function. As
the result of the evaluation, we have found that the function correctly diagnosed 70.6% of
appropriate topic sentences taken from textbooks for paragraph writing. In the future, weplan to evaluate the function by using sentences including inappropriate topic sentences.

References
Donald, B. R., Moore, D. J., Moorow, R. B., Wargetz, L. G., & Werner, K., (1998) Writing Clear
Paragraphs (6th Edition). Prentice-Hall.
Howard, C. J., & Tracz, R. F., (1976). Writing Effective Paragraphs. Winthrop Publishers.
Ishikawa, K., & Kunichika, H., (2023). Development of a Diagnostic Function of the Topic Sentence fora Classified Paragraph. Annual Student Meeting 2023, Japanese Society for Information and
Systems in Education. Retrieved March 7, 2023, from https://www.jsise.org/wpcontent/uploads/2023/01/2022_kyusyu_okinawa_a03.pdf (In Japanese)
Kitao, S. K., & Kitao K, (1998) Basic English Paragraphs -Improving Reading and Writing SkillsEichosha Co., Ltd.
Oshima, A., & Hogue, A., (2017) Longman Academic Writing Series (3rd Edition). Pearson
Education Inc.
Stanford NLP Group, (2022) Overview – CoreNLP. https://stanfordnlp.github.io/CoreNLP/
Su, Y., Lin, Y., Lai, C., (2023). Collaborating with ChatGPT in argumentative writing classrooms.
Assessing Writing, 57(2). https://doi.org/10.1016/j.asw.2023.100752

794

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Construction of an English Grammar Quiz
Recommendation System Using
Explanation by a Knowledge Map
Kensuke TAKIIa*, Naomichi TANIMURAa, Brendan FLANAGANb & Hiroaki OGATAca
Graduate School of Informatics, Kyoto University, Japanb
Center for Innovative Research and Education in Data Science, Kyoto University, Japanc
Academic Center for Computing and Media Studies, Kyoto University, Japan*kensuke.takii96@gmail.com
Abstract: Many systems to assist in learning English grammar have been developedin the field of learning material recommendation systems (LMRSs). Compared withthose based on knowledge models, recommendations based on data tend to cause thecold-start problem, and it is said that explainable LMRSs may be able to enhancelearners’ motivation. In our research, we propose an explainable English grammar quiz
recommendation system using a knowledge map to support students’ learning ofEnglish grammar with trust in the system and motivation. The learning effect of the
explanation of the system was evaluated in an experiment, in which 349 high school
students in Japan participated. This experiment showed that there was little learning
effect of the explanation, but the system reliability and motivation improved by the
explanation. The limitation and our future work regarding the validity and learning
effects of the system are also indicated.
Keywords: English grammar learning, e-learning recommender, knowledge map,learning path, explainable recommendation

1. Introduction
There has been increasing need for learning material recommendation systems (LMRSs),
because of the difficulty for learners to select appropriate materials due to the increasing
options (Liu, 2019). Although existing LMRSs can be classified into systems based on
educational data and those based on domain knowledge models, the former is more likely tocause the cold-start problem for new users than the latter (Sun et al., 2019). Additionally,
recent studies have developed explainable LMRSs such as Takami et al. (2022), which can
explain the rationale of the recommendations. Since feedback from systems improves
learners’ motivation and achievement (Duffy and Azevedo, 2015), explainable LMRSs basedon knowledge models may improve the learning effect and motivation in learning.
In the field of LMRSs, many systems to assist in learning English grammar have beendeveloped (Fang et al., 2018). English is an important second language in many regions of
the world, and the grammatical structures present in various languages pose challenges forreading comprehension in foreign languages (Alderson et al., 1984). According to previous
research, teaching grammar items in a specific order is necessary to gain maximum
educational effect (Izumi and Isahara, 2004). Hence, the order in which grammar items shouldbe learned is vital in learning English as a second language (ESL).
In our research, we propose an explainable English grammar quiz recommendation
system using a graph structure called a knowledge map. We developed an English grammar
knowledge map as a directed graph structure that can represent English grammar items, theirrelations, and the order in which they should be learned. Besides, the recommendation using
the knowledge map can explain its rationale by tracking learners’ learning logs andrecommended quizzes. In addition, we implemented the system into an existing web-basedlearning system and conducted an experiment targeting Japanese high school students who795are studying ESL to verify the effect of the explainable recommendation in learning English
grammar. Our research focuses on the explanation of the rationale of the recommendations
and the following research question is set: Does the explanation of the rationale of therecommendation for learning English grammar have a learning promotion effect?

2. Proposed Approach
2.1 English Grammar Knowledge Map
We developed an English grammar knowledge map, which represents English grammar items,
the relations between them, and the learning order as a graph structure. We referred to CEFRJ Grammar Profile (Ishii and Tono, 2018), as a list of English grammar items. It includes 501
grammar items, and each item is classified into the CEFR (Council of Europe, 2011) level.
Each item in the profile has a regular expression, which is used to judge if the item is used ina certain sentence. For example, the sentence “I am a student.” includes the grammar item
“Personal pronoun nominative (I)+be: I am”, and the system can automatically determine thatthe sentence includes the grammar item by using the regular expression.
The knowledge map was generated based on the method by Flanagan et al. (2019):
The nodes representing grammar items in CEFR-J Grammar Profile are connected by directed
weighted edges, which represent the similarity of the regular expressions of the items andtheir proper learning order. The order was based on “English Grammar/Syntax Vintage”(Shinoda and Yoneyama, 2010), a workbook used for ESL learning in Japanese high schools.
The knowledge map is structured as a directed maximum spanning tree by pruning the edges.

2.2 Explainable Quiz Recommendation with Knowledge Map
To recommend quizzes, our system first judges if each grammar item is learned or
misunderstood by learners. The system creates two grammar sets using learners’ learning
logs: The first is Incorrect Grammar Set (IGS), including grammar items in quizzes that theuser has answered incorrectly, and the second is Unanswered Grammar Set (UGS), including
grammar items in quizzes that the user has not yet answered. Then, the system recommends
quizzes based on the following three conditions regarding the number of grammar items
included in the quizzes. The higher these numbers, the more highly the quizzes arerecommended; however, these 3 numbers are used in order of priority from 1 to 3. Besides, if
there exists a directed edge from A to B in the knowledge map, we call A “a prerequisite” of B.
1. The number of grammar items that are prerequisites for other grammar items in IGS, andthat are also included in IGS
2. The number of grammar items that are prerequisites for other grammar items in IGS, andthat are included in UGS
3. The number of grammar items that are not prerequisites for other grammar items in IGS,
but which themselves would be included in IGS
In generating an explanation in the proposed system, when the system recommends
grammar quizzes, the grammar items are distinguished by the following criteria: whether thelearner has incorrectly answered quizzes including these items before, or the learner has not
answered these quizzes yet. Furthermore, the system notifies the learner whether a certain
grammar item is a prerequisite to be known for another item that the learner has
misunderstood, i.e., the learner has incorrectly answered quizzes including the grammar item.

2.3 User Interface of the Platform
As an interface for answering quizzes, we adopted BookRoll (Flanagan and Ogata, 2018), a
multifunctional e-book reader system, and registered English grammar quizzes in a workbook
being used in Japan for learning English. BookRoll has many functions to realize e-learning
environments; one of which is a quiz function, with which learners can answer quizzes

796registered to materials. We made e-books for answering the grammar quizzes, by editing thecontents of the workbook so that each page has one quiz. Users can solve the quizzes on
BookRoll and register whether their answer for each quiz was correct or incorrect.
The user interface of our recommendation system provides learners with informationon (1) quizzes recommended by the system, (2) links to open the quizzes on BookRoll, and(3) explanations of why the recommendation was made for the learner. The interface displays
5 recommended quizzes, each of which has at most 3 types of explanations for why the quiz
was recommended. These explanations clarify what kind of grammar items the user can learn
with the recommended quizzes.

3. Experiment
3.1 Method Overview
We conducted an online A/B test, targeting high school students in Japan who are learning
ESL by the curriculum in Japan. The target students were divided into an experimental group
and a control group. Students in the experimental group used a recommendation system that
displayed the explanation of the recommendation rationale, and students in the control groupused a system that did not. Furthermore, we set the following 4 perspectives from which theeffect of the explanations according to the RQ shown in the Introduction section:z The number of recommendation usage: How highly did the students engage in learning?
z Scores of the pre- and post-test: Change of the learners’ performancez Answers for the poll: Willingness to learn with the system and trust in it

3.2 Settings
The experiment targeted 349 first graders in 9 classes at high school (about 40 students perclass, the students who did not agree to participate were excluded). The participants weredivided into the experimental group (4 classes, 152 students) and the control group (5 classes,
197 students), both of which used the recommendation system while learning for 14 days.
This experiment was conducted during a winter vacation, and the students were encouraged
in doing the learning activity with the system as a review, after learning English grammar witha paper workbook. This online learning activity was done with BookRoll, and 219 grammar
quizzes were registered to it. Table 1 shows an example of grammar quizzes.
Table 1. Examples of the quizzes registered to BookRoll
Questionೖຌ‫ޢ‬͹қັͶ߻͑Γ͑Ͷ ( ) Ͷన‫ޢ‬Νॽ͘͵͠͏ɽ
(Write an appropriate word to match the Japanese sentence.)͍͵ͪ͗οʖϞͶ͏ͱ͚ΗͱࢴͪͬͺϧρΫʖͲͤɽWe are ( ) ( ) have you in our team.

Answerlucky to3.3 Experimental Procedure
Step 1. The English teacher explains how to use the system to the participants.
Step 2. Before starting the learning activity, the participants take an English grammar pre-test.
Step 3. The participants solve some quizzes with BookRoll.
Step 4. The participants work on the quizzes using the recommendation and BookRoll.
Step 5. After ending the learning activity, the participants take an English grammar post-test.
Step 6. The participants answer the poll.
The contents of the pre and post tests were 5 grammar quizzes, which were randomly
selected from the quizzes registered to BookRoll. The contents of both the quizzes were same.
The poll was intended to an evaluation of the motivating effects of the system from the797

viewpoints of trust and motivation in learning. This poll consists of 11 questions with a 5-Likert
scale and 2 descriptive questions. For measuring the trust and motivation in using the system,
we adopted Hoffman et al. (2018)’s “Trust Scale Recommended for XAI” and Keller (1987)’s
ARCS model. The perspective of the questions is shown in the Table 3 in Result section.

4. Result
4.1 Usage of the System
Table 2. Numbers of students who used the system
With explanation
Without explanation
Total

Participant
152
197
349

Recommendation
16 (10.5%)
18 (9.1%)
34 (9.7%)

BookRoll
10 (6.6%)
10 (5.1%)
20 (5.7%)Both
26 (17.1%)
28 (14.2%)
54 (15.5%)

Table 2 shows the numbers of students who used both the recommendation system and
BookRoll, who accessed the recommendation interface, and who used BookRoll only. This
table indicates that about 10% of the participants used the recommendation and that about
6% of them used BookRoll only, without using the recommendation.

4.2 Result of the Poll
Table 3. Result of Mann-Whitney’s U test on the questions in a 5-Likert scale of the poll
Question Perspective
Q1
Trust
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9

Trust
(predictability)
Trust(reliability)
Trust(efficiency)
Trust(endorsement)
Motivation(alert)Motivation(relation)
Motivation(confidence)
Motivation(satisfaction)

Group
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation
With explanation
Without explanation# of Users
14
18
14
18
14
18
14
18
14
18
14
18
14
18
14
18
14
18

Mean
3.29
3.06
3.64
3.06
2.93
2.78
2.86
2.61
2.57
2.56
2.57
2.50
3.14
2.94
4.07
4.00
2.79
2.61

Std.
0.61
0.94
1.01
0.99
1.00
1.00
1.17
1.04
1.16
0.86
0.94
0.86
1.10
0.87
0.73
0.69
1.12
0.98

P-value
P=0.59P=0.58P=0.69P=0.64
P=0.98P=0.79P=0.56
P=0.61
P=0.66

Table 3 summarizes the result of the poll. It shows that from both the perspectives of trust andmotivation, the impression of the students in the experimental group was more favorable thanthat of the students in the control group. In terms of Trust, there are differences of more than
0.2 points in the mean scores in the items of trust and efficiency. In terms of Motivations, there
are differences of more than 0.2 points in the items of relation. In the free-text items, theanswers regarding the efficiency in review and finding weak points were the most common
response which was favorable for the recommendation system. On the other hand, as the
response that was not favorable, we found the answers that doubted the correctness of thediagnosis by the system and found selecting the quizzes troublesome. In addition, somestudents found the system interface inefficient.

798

4.3 Result of the Pre- and Post-test

Figure 1. Result of the pre- and post-test and the change of the scores
Figure 1 shows the distribution of the results of the pre- and post-test and the change in thescores. The scores of the post-test were higher than those of the pre-test, but this difference
was not statistically significant. Comparing the experimental and control groups, the scores ofthe pre- and post-test and the change in the scores did not indicate significant differences.

5. Discussion
5.1 Learning Promotion Effect by the Explainable Recommendation
In the post-poll, there were relatively large differences on the items that asked about trust,
predictability, reliability, and efficiency in the Trust items. This implies that the explanation of
the recommendation rationale improved the reliability of the system as perceived by the users.
On the other hand, the item that asked about endorsement showed less difference.
Considering that many answers for the free-text questions referred to the difficulty in using thesystem, the use of the system itself was a burden in learning. Also, the reliability of the
explanation would be less influential on the system acceptance than the system’s userfriendliness. In the question about Motivation, there were relatively large differences on theitems about relation and satisfaction, which implies that the display of explanation of therecommendation rationale made users feel the relationship between their proficiency andlearning contents proposed by the system. Furthermore, in the free-text questions, some users
in the control group doubted the reliability of the diagnosis by the system, and such answers
could not be found in the experimental group. This implies that it is difficult for users to judge
the correctness of the recommendation without an explanation of the rationale.
The result of the grammar tests shows no significant differences between the twogroups, and the post-test scores were higher than the pre-test scores but not significantly
different from them. This implies that the student did not use the system often enough for thedifference in the condition to influence the scores. Also, the number of quizzes in pre and post
tests may be so small that no significant differences in the grammar tests were found.

5.2 Limitation
Our system was used infrequently regardless of conditions. We argue that the main reasons
were lack of usability, and the burden in reading the explanation. When the students solve thequizzes with BookRoll and the proposed system, they must switch their screens many times.
Second, if reading the explanation requires long time, it may be the users’ burden to propose
the explanation. Next, our system does not consider the level of importance of each grammar
item in learning. Each quiz includes several grammar items, some of which are necessary tosolve it, but others are not. Without considering the importance of each grammar item, thesystem might recommend quizzes that include items not important for solving them.

799

6. Conclusion and Future WorkOur experimental result did not show a learning effect from the explainable recommendation
but implied that the system reliability and motivation were improved. Therefore, the answer toour research question was that the explainable recommendation improved the users’ learningin the aspects of (1) reliability and motivation and (2) the frequency of the system use.
Our future work will focus on the validity and learning effects of the system. First, weshould verify the accuracy of the knowledge map and the learning effects by consideringlearning paths. Second, the sufficient result has not been gained about whether the
explanation of the recommendation rationale has a learning effect. The burden of using thesystem and understanding the explanation should also be investigated.

Acknowledgements
This work was partly supported by JSPS Grant-in-Aid for Scientific Research (B) 20H01722,
JSPS Grant-in-Aid for Scientific Research (Exploratory) 21K19824, and NEDO JPNP20006.

References
Alderson, C. J., Alderson, J. C., & Urquhart, A. H. (Eds.). (1984). Reading in a foreign language.
Longman Publishing Group.
Council of Europe. Council for Cultural Co-operation. Education Committee. Modern Languages
Division. (2001). Common European framework of reference for languages: Learning, teaching,assessment. Cambridge University Press.
Duffy, M. C., & Azevedo, R. (2015). Motivation matters: Interactions between achievement goals andagent scaffolding for self-regulated learning within an intelligent tutoring system. Computers in
Human Behavior, 52, 338-348.
Fang, L., Tuan, L. A., Hui, S. C., & Wu, L. (2018). Personalized question recommendation for English
grammar learning. Expert Systems, 35(2), e12244.
Flanagan, B., Chen, M. A., Lecailliez, L., Majumdar, R., Akçapinar, G., Ocheja, P., & Ogata, H. (2019,
December). Automatic vocabulary study map generation by semantic context and learning material
analysis. In Proceedings of the 27th International Conference on Computers in Education(ICCE2019) (pp. 698-702).
Flanagan, B., & Ogata, H. (2018). Learning analytics platform in higher education in Japan. Knowledge
Management & E-Learning, 10(4), 469-484.
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018). Metrics for explainable AI: Challenges andprospects. arXiv preprint arXiv:1812.04608.
Ishii, Y., & Tono, Y. (2018). Investigating Japanese EFL learners’ overuse/underuse of English grammar
categories and their relevance to CEFR levels. In Proceedings of the 4th Asia Pacific Corpus
Linguistics Conference (pp. 160-165).
Izumi, E., & Isahara, H. (2004, December). Investigation into language learners’ acquisition order basedon an error analysis of a learner corpus. In IWLeL 2004: an interactive workshop on language elearning (pp. 63-71).
Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of
instructional development, 10(3), 2.
Liu, T. Y. (2009). A context ̺ aware ubiquitous learning environment for language listening andspeaking. Journal of Computer Assisted Learning, 25(6), 515-527.
Shinoda, S., Yoneyama, T. (2010). Eibunpo Goho Vintage New Edition, Iizuna Shoten. (in Japanese)Sun, Z., Guo, Q., Yang, J., Fang, H., Guo, G., Zhang, J., & Burke, R. (2019). Research commentary on
recommendations with side information: A survey and research directions. Electronic Commerce
Research and Applications, 37, 100879.
Takami, K., Dai, Y., Flanagan, B., & Ogata, H. (2022, March). Educational explainable recommenderusage and its effectiveness in high school summer vacation assignment. In LAK22: 12th
International Learning Analytics and Knowledge Conference (pp. 458-464).

800

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Educational System of English Tense forJapanese Learners by Forming Temporal
Constraints on Tense
Haruto NAGATAa*, Tatsuhiro KONISHIb, Makoto KONDOc
Graduate School of Integrated Science and Technology, Shizuoka University, Japanb,cFaculty of Informatics, Shizuoka University, Japan*nagata.haruto.18@shizuoka.ac.jpaAbstract: In this study, we developed an English-tense learning and education
system that provides training in constructing tense constraints from given
sentences. Temporal constraints refer to the condition that the correct choiceshould be met in English tense-choice questions. In this study, we organized
the patterns of temporal constraints and guided learners to assemble thembased on these patterns. We also introduced a visual representation toenhance the intuitive understanding of temporal constraints. We conducted anexperimental evaluation of the proposed system to investigate its learning
effectiveness. Additionally, in the experimental evaluation, we developed asystem using graphic representation and a system without graphic
representation and investigated the differences between them.
Keywords: Learning Support System of English grammar, Tense in English,
Temporal Constraints, Graphic Representation
1. Introduction
In English language learning among Japanese native speakers, errors in verb tense largely
depend on knowledge outside the text, including real world knowledge. As a result, correcting
verb tense and aspect errors has been suggested to be difficult (Lee, 2008).
In this study, we focused on verb tense and aspect in the context of English tense
multiple-choice exercises in which learners are required to select the appropriate verb form
from a set of choices to fill in a blank. The process of solving English tense problems can beconsidered as (1) constructing constraints about the word that should fill in the blank in the
problem sentence and (2) determining whether each choice satisfies all the constraints. We
call these constraints “temporal constraints.” In English tense problems, temporal constraints
are frequently encountered and are important among those constraints. Several errors ofEnglish are believed to occur in the (1). Therefore, we developed a learning and educational
support system that trains learners to construct temporal constraints from the information
contained in problem statements. To assist in the construction of tense constraints, we break
down temporal constraints into patterns. By conducting the learning process of constructing
temporal constraints based on each pattern, we aim to improve the accuracy of answering thequestions and enhance the ability to explain the cognitive reasoning underlying the answers.
As a learning method for constructing temporal constraints, we envision an interactive
process between learners and an educational support system that guides them in following(1) and (2). In Step (1), the system guides learners to construct temporal constraints from the
problem text based on the temporal constraint patterns. In Step (2), the system allows learners
to apply constraints to each choice and determine whether the constraints are satisfied. The
constraints can be expressed in text format. However, we are afraid that some types of
temporal constraints in text are difficult to intuitively apply to the choices in the Step (2).
Therefore, we introduce a graphical-representation method to express these constraints.
The reasons for introducing the use of graphical representation are as follows.

801

Problems with English tenses require an understanding of the temporal relationships between
events in English sentences. In general, time is recognized as a one-dimensional space fromthe past to future. Thus, by representing the timeline as a one-dimensional space anddisplaying a diagram that positions events on the timeline, the temporal relationship between
the events can be clarified.
In this study, to verify the effectiveness of this graphical representation, we constructed
and conducted an experimental evaluation of both a system with and without graphic
representations. The following results were obtained for both systems. Before and after
receiving support from the systems, a significant improvement was observed in both theaccuracy of problem solving and the quality of explanations for the answers. By using systems
to practice exercises repeatedly, learners can internalize the methods for constructing
temporal constraints without relying on the systems.
2. Related works
Many educational systems of English grammar for Japanese learners have been developed(Le,2011)(Ozaki,2004)(Ozaki,2007). However, these systems don’t aim to teach usage ofverbs. The misuse of verbs such as tense and aspect is a typical error category for nonnative
speakers of English. Correcting errors related to tense and aspect, which are directly related
to the English verb tense, is considered difficult. This is because tenses and aspects heavily
depend not only on information within the target sentence, such as adverbs and coordinating
conjunctions, but also on the global context, such as knowledge of the real world outside thetarget sentence (Lee,2008). In the KJ Corpus, which annotates grammatical errors andperforms syntactic analysis, article errors are the most frequent, followed by errors insingular/plural nouns, prepositions, and verb tenses.
Various methods have been proposed to support learning and education through theautomatic detection and correction of errors(Page,2003)(Burstein,2003)(Liao,2016).
However, there have been few developments in the creation of systems that specifically focus
on thought processes involved in solving problems. As an educational system that focus on
thought process, the system establishes thought process for arithmetic problems by having
students select and arrange sentences from a provided collection of short sentences and posequestions in an appropriate order(Hirashima,2007). A system also focuses on developing
thought process of programming using code puzzles(Ito,2020). Code puzzles are having
students form programs by arranging statements that are ordered randomly. These systems
are designed to have students form something in order to focus on thought process in solving
problems. But they aren’t systems for English tense. In terms of learning of English tense,
what should be formed by students? We think forming temporal constraints is effective forstudents to learn thought process of English tense problems. So we developed educational
systems that teach students how to form the temporal constraints required for correct answer
choices in English tense choice questions.
As a method for teaching students how to form temporal constraints, we focused onusing graphical representations. Some studies use them to assist Japanese in understanding
English tenses(Tanaka, 2018)(Miura, 2010). However, these graphical representations are notintended for application in learning English tenses; therefore, we thought an extension of thegraphical method is necessary. In this study, we developed graphical representations suitable
for learning and educational support, as described in Section 3.2. We developed learning andeducational support systems to help learners judge whether each choice is valid by comparing
the temporal constraints and events represented by each choice graphically.
3. Basic Ideas
3.1 Temporal constraints
We analyzed the relationship between the validity of choices and temporal constraints for 24
English tense choice questions from standard university entrance exam questions in Japan,
including past exam questions from the National Center for University Entrance Examinations

802

(2000-2018) and commercial study materials in Japan (Nakao,2016). Consequently, we foundthat temporal constraints can be classified into the following three types:
1. Constraint by specific words or phrases: “already”, ”by + time phrase”, and other specificadverbial or prepositional phrases determine the tense and aspect of the verb they modify.
2. Constraint by tense agreement: When the main clause is in the past tense, the subordinate
clause also becomes past tense or past perfect tense.
3. Constraint based on the relationship between events: Constraints based on the context ofthe problem and common sense of the real world determine tense and aspects of verbs.
The systems that we developed allow learners to form these constraints in theirsentences. For a system that uses graphical representations, the generated constraints canbe incorporated into graphical representations.
3.2 Extending Graphical Representation
Based on the above problem analysis, we organized the necessary components forexpressing the temporal constraints in a diagram as follows:i) Event node: The element representing an event is expressed as a node, and the aspect of
the event is expressed by the node type. These are shown in Figure 1.
ii)Timeline: We express the timeline using a horizontal arrow.
iii)Time of utterance: This is the time point when the problem statement was uttered.
In the timeline, the right side represents the future and the left side represents the pastwith respect to the point when the problem statement is uttered. For each event in the problem
statement, we set the following components: i) ii) iii) For example, the graphical representationof the sentence “By the time the 2002 World Cup was held, soccer had already become aleading sport in Japan.” is shown in Figure 2.

ii)

Figure1. Aspect GraphicalRepresentationsiii

Figure2. Example of The Graphical Representation

3.3 Learning process in the system
The learning process in the system using graphical representations is shown in Table 1. On
the other hand, system without using graphical representations, formation of graphicalrepresentations of temporal constraints and problem statement containing the choice of words
and comparison of them are omitted, as described in Section 3.3.1.
Table1. Learning Process in the System Using Graphical Representations803

4. Implementation
In this study, we constructed systems and implemented 10 problems that contained two typesof constraints as a prototype: constraints by specific words or phrases and constraints by tense
agreement. The constructed systems followed the learning flow described in Section 3.3.
Systems are implemented using the C# language in Visual Studio 2022.
4.1 Forming and comparing graphical representations of temporal constraints with graphicalrepresentations of each choice
Temporal constraints were created as text by filling in the fill-in-the-blank template, as shownin Figure 3. Constraint types are displayed on the menu as guidelines to make learners awareof the pattern of constraints. By following a certain abstract pattern, learners learn to constructtemporal constraints. If the learner makes a mistake, the system provides advice tailored tothe blank space to help the learner arrive at the correct answer.
Constraint by specific word
specific word
Word modified by specific word
Word modified by specific word
Constraint by tense agreement
Temporal relationship between
clauses based on tense
agreement

Figure3. Screen for Creating Temporal Constraints (as a text)

4.2 Forming and comparing graphical representations
Learners visualize the problem statement, including temporal constraints and eachchoice. They form graphical representations by specifying the temporal constraints andaspects of the node, as indicated by the red boxes in Figures 4 and 5. By visually comparing
them, learners can judge whether each choice satisfies temporal constraints.

Learners select the tense andaspect from these choices

Figure4.Forming GraphicalRepresentations (Temporal Constraints)

Figure5. Forming Graphical Representations(problem statement including choices)

5. Experimental Evaluation
5.1 Hypothesis of the experiment
In the experiment, we validated the following experimental hypotheses.
Hypothesis 1: Prompting learners to consider temporal constraints improves the accuracy of804their responses to questions and increases the quality of their explanations for judging thevalidity of their choices.
Hypothesis 2: Repeated practice exercises will help establish a consistent approach to judge
the validity of each choice.
5.2 Experimental procedure
Participants were divided into two groups: Group A used a system with graphicalrepresentations, and Group B used a system without graphical representations. Five
Japanese university students were recruited for each group and were given 10 English tense
choice questions. Experimental procedure was performed according to Section 3.3 above.
From the data obtained Step 1 and Step 7 in Section 3.3, aggregate how the reasons for eachchoice of “ʕ”, ”×”, and “?” changed. Thus, it can be said that experimental Hypothesis 1 is
supported if the accuracy rate improves and the justifications for the validity of each choiceprovided by the learners become clearer, or if there is a reduction in ambiguous reasoning. In
addition, if the reasoning for each choice in the system before learning becomes clearer
through the repeated practice of problem-solving, Hypothesis 2 is supported.
5.3 Results
5.3.1 The correct answer rate
The correct answer rates for the questions in Groups A and B are shown in Figures 6 and 7.
Here, the correct answer to a question means that all choices for that question are correctly
answered with “ʕ” or “×”. In both groups, “the correct answer rate improved significantly before
and after using systems”(result1).

Figure6. Accuracy of the Problems (group A)

Figure7. Accuracy of the Problems (group B)

5.3.2 Changes in reasoning before and after system learning
We scored the validity reasoning for each choice before and after learning withsystems as follows: Items that provide clear reasoning(2 points). Items that describe the typeof temporal constraints, but do not provide specific measures to address the issue(1 point).
Items that are clearly ambiguous, such as “I thought it seemed like it.”(0 point).
The total number of choices for the 10 questions was 165. Assuming that all 165
justifications for the choices are clear (2 points), the percentage of clarity in justifications before
and after system learning for each group is shown in Figures 8 and 9.
In the justifications after system learning, it is clear that both Group A and B haveclearer justifications than before system learning for all 10 questions. Therefore, we find that“using the system improves quality of explanations for the reasoning of the validity of the
choices”(result2). Thus, the Experimental Hypothesis 1 is supported by the result1 and theresult2.Furthermore, the changes in learners' understanding through repeated practice can beseen from the score changes in questions ձ-պ before system learning in Figures 8 and 9. As
a result, “gradual improvement can be observed as the number of questions
increases”(result3). Although this experiment used only 10 questions, we can expect sufficient805improvement if we increase the number of questions. Therefore, we can say that theExperimental Hypothesis 2 is supported by the result3.

Figure8. Justifications Before and After System
Learning (group A)

Figure9. Justifications Before and After System
Learning (group B)

6. Conclusion
In this study, we developed learning education support systems that teach learners toconstruct temporal constraints based on information on English tense-choice problems. The
experimental results showed that we could get “the correct answer rate improved significantly
before and after using systems”(result1), “using the system improves quality of explanations
for the reasoning of the validity of the choices”(result2) and “gradual improvement can beobserved as the number of questions increases”(result3) in Section 5.3. However, based on
their responses to the questionnaire, some students noted that the graphic representations
were effective. Therefore, we plan to conduct experiments to assess the effectiveness ofgraphics tailored to the learners’ English proficiency levels.
References
Lee, S. (2008). Correcting misuse of verb forms, Proceedings of the 46th ACL,174-182.
Page, E. B. (2003) Project Essay Grade: PEG. In M. Shermis, & J. Burstein (Eds) , Automatedessay scoring: A cross-disciplinary perspective. New York: Routledge.43-54.
Burstein, J. (2003). The e-rater‫ ט‬scoring engine: Automated essay scoring with natural
language processing. In M. Shermis, & J. Burstein (Eds), Automated essay scoring: A
cross-disciplinary perspective. New York: Routledge, 113-121.
Liao, H. (2016). Using automated writing evaluation to reduce grammar errors in writing.
ELT Journal, 70(3), 308-319.
Le, H. Brook, J. (2011). Using Praat to teach intonation to ESL students. Hawaii Pacific
University TESOL Working Paper Series 9(1), 2:2-15.
Ozaki, M. Koyama, K. Takeoka, S. Adachi, Y. (2004). Development of Teaching Materials
Which Dynamically Change in Learning Process, Proceedings of International
Conference on Knowledge-Based Intelligent Engineering Systems KES’ 2004, 77-82.
Ozaki, M. Adachi, Y. Takeoka, S. Sugimura, A. Ishii, N. (2007) Educational System Using Selfmonitor Study and Streaming. Proceedings of 11th International Conference on
Knowledge-Based Intelligent Engineering Systems KES’ 2007. 1037-1044.
Hirashima, T. Yokoyama, T. Okamoto, M. Takeuchi, A. (2007). Learning by Problem-Posing
as Sentence-Integration and Experimental Use. Proc. of AIED2007, 254-261.
Ito, H. Shimakawa, H. Harada, F. (2020). Comprehension analysis considering programming
thinking ability using code puzzle. Proceedings of the Federated Conference on Computer
Science and Information Systems. 609-618.
Tanaka, K., Honda, K., Hatayama, Y. (2018). Tense and aspect. Asakura Shoten (inJapanese).
Miura, Y. (2010). Language process of English tenses: An attempt to illustrate with rail-camera
diagrams. Journal of Humanities 24, 143-159 (in Japanese).
Nakao, K. (2016). Conquering 261 frequently-tested items in Scramble English Grammar andSyntax 3rd Edition. Obunsha (in Japanese).

806

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring the use of chatbot to promote
online EFL students’ behavioral, cognitive,and emotional engagements
Xinyi LUOa, Weijiao HUANGa*, Khe Foon HEWa, Chengyuan JIAb & Xiangjie CAOca
Faculty of Education, The University of Hong Kong, Hong Kong SAR, Chinab
College of Education, Zhejiang University, Hangzhou, Chinac
College of Foreign Languages, Northwest University of Political Science and Law, Xi’an,China*wjhuang1@connect.hku.hk
Abstract: The implementation of online learning is often beset with the challenge ofstudent disengagement, especially in online language learning where active learner
practice is expected. To address this problem, we explored the use of a chatbot tosupport students with their English as a Foreign Language (EFL) listening learning in
an online undergraduate course. The Self-determination theory was adopted to design
the chatbot. Students were invited to interact with the chatbot for ten days to practicetheir English listening skills. A mixed-methods design was used to measure students’behavioral engagement, cognitive engagement, and emotional engagement. The
results indicated students’ positive online learning experiences with the chatbot. We
also conducted follow-up interviews with students to determine their perceptions oflearning with the chatbot.
Keywords: Self-determination theory, chatbot, online learning, language learning

1. Introduction
Over the past few decades, online learning has been viewed as a weaker alternative, andoften stigmatized as a lower-quality form of education, compared to traditional face-to-face
instruction (Hodges et al., 2020). However, due to the disruption caused by Covid-19, many
countries are forced to switch to online learning (Barron Rodríguez et al., 2020). Educators
had to employ online learning approach during the pandemic to sustain school education.
Today, despite the end of the pandemic, the increasing of online learning has exceeded our
expectations and grown in importance. Nonetheless, the persistent issue of student
disengagement in online learning should not be overlooked (Dumford & Miller, 2018). Student
disengagement refers to students’ low participation or interest in the learning activities, which
negatively impacts their online learning performance. This problem also exists in online
language learning (Ji et al., 2022), where students need to master the language skills by
actively practicing the specific skill. Previous studies have identified several challenges that
affect students’ engagement in online language learning, such as the lack of autonomy (Jiang& Peng, 2023), insufficient interactions (Tran, 2018), and inadequate teacher instruction(Indartono, 2019). One way to address this issue is to provide students with chatbots in their
online language learning (Hew et al., 2023).
Chatbots, dialogue systems that interact with users via text or voice, are used primarily
for language learning in education, particularly English learning. Previous research has
demonstrated the positive effects of chatbots as conversational partners for students’language learning, with experimental groups showing significant performance improvements(e.g., Jeon, 2021). While some studies (e.g., Hew et al., 2023) highlight the potential of
chatbots to enhance student engagement, others (e.g., Liu et al., 2022) found no significant
impact on student interest in English reading activities. Given these mixed results, further
research is needed to determine the effectiveness of using chatbots in language learning.

807

Existing studies (e.g., Yang et al., 2022) have primarily focused on chatbot applications in
speaking, reading, and writing, with limited empirical research on the use of chatbots toimprove students' EFL listening comprehension.
In this study, we used a chatbot as an interlocutor for EFL listening learning in an online
listening course. The present study mainly aimed to investigate the impact of the chatbot on
student engagement. Student engagement has broadly been described as a multidimensional
concept that encompasses three main domains: behavioral, cognitive, and emotional
engagement (Cooper, 2014). Behavioral engagement refers to students' participation inlearning and academic assignments (Fredricks et al., 2004). Cognitive engagement concerns
the level of students’ desire to go beyond the course requirements in the learning (Fredricks
et al., 2004). Emotional engagement was described as students’ affective responses, likeinterest, satisfaction, and sense of anxiety (Fredricks et al., 2004). Additionally, we exploredsuggestions from students to further optimize the effectiveness of the chatbot in online
language learning. This study is guided by the following research question: What is the effectof the chatbot on students’ engagement (behavioral, cognitive, and emotional
engagements) in online English listening learning?

2. Design and development of the chatbot
In this study, we designed the chatbot based on self-determination theory (SDT) aiming toengage students in online English listening. Engagement is considered an observableoutcome of motivation (Reeve, 2012). SDT is a widely used theory for motivational
development and psychological needs to deepen understanding of how and why individual
behavior occurs (Deci & Ryan, 1985). SDT assumes three vital and inherent human needs
that drive action: Autonomy, Competence, and Relatedness (Ryan & Deci, 2000). The needfor autonomy is concerned with the desire of people to be causal agents and to have volitionin their behavior (Ryan & Deci, 2000). More specifically, students’ demand for autonomy canbe satisfied if they can freely choose to make an extra effort or not. Competence pertains towhether one’s pursuits or learning can be effectively mastered (Helme & Clarke, 1998). In the
educational setting, students feel more competent when they can address the challenges ofassigned tasks. Relatedness can be defined as students’ desire to feel associated with others
and to own a sense of belonging (Ryan & Deci, 2000). In teaching and learning, students havean imperative desire to interact with their peers and instructors. Figure 1 shows the chatbot
interface.

Figure 1. Chatbot interface.
The autonomy in students’ learning was achieved by an optional learning journal
activity in the chatbot system, where students could reflect on their daily learning experiences.
The chatbot prompted students to express their learning difficulties or reflections, by saying“How’s your learning today? Any difficulties?”. We considered students’ journal writing as theindicator of their autonomy because they controlled the learning behaviors (deciding whether
or not to write and what materials to reflect upon).
Competence was attained through three approaches: instructor-designed explanation
videos, targeted feedback, and task instruction. First, the chatbot would deliver an instructor-

808

designed video that included the key points of each listening practice, helping students tocomprehend the listening materials. Second, the chatbot would provide targeted feedback interms of students’ input. For example, if students misspelled a required word, the chatbot
would respond with target feedback, such as “The spelling of this word is not complete.” Third,
the chatbot could provide step-by-step instructions on each listening task during the interaction.
This aimed to alleviate any confusion that students may encounter during the learning process
and enhance their competence to master the listening skills. For example, the chatbot
introduced the learning objectives and guided students to check learning content by saying“You may check the explanation video below.”
To enhance relatedness, the chatbot used communication indicators such as emojis
and encouraging prompts for human-like interaction. Emojis, which are widely used in digitalcommunication (Boutet et al., 2021), can help the chatbot to express emotional responses.
For example, the use of the celebratory emoji in a message, "Well done⦦⦯", could help the
⦮
⦭
⦬
⦫
⦪
⦩
⦨
⦧
chatbot convey more friendly features to students. In addition, encouraging messages wereused to foster a positive learning environment. When students made a mistake, the chatbot
would respond with, “Please try again. I believe you can do it!”

3. Research Design
Our target participants were 41 year-two undergraduate students at a large public university
in northwestern China. The present study was conducted in a compulsory English course. The
whole intervention lasted two weeks, and students needed to interact with the chatbot in tendays to practice English listening. Ethical approval to carry out the study was obtained fromthe author’s university and consent was received from all participants. Table 1 summarizes
the quantitative data collection methods used in this study. To further examine students’
perceptions of using the chatbot, individual semi-structured interviews were conducted.
Twelve participants attended the interviews voluntarily. A sample interview question was “Howdo you think about your learning with the chatbot?”
Table 1. Data collection methods
Measurements
Quantitative data collection
Behavioral
Students’ listening task completion rate, which refers to the number
engagementof students who complete the required task to the total number ofstudents. Students learning records were collected.
Cognitive
Students’ learning journal completion rate. This is an optional
engagement
learning record.
Emotional
A 5-point Likert scales survey was conducted to examine students’
engagementsocial presence and interpersonal attraction in the chatbot (adopted
from Li et al., 2016).

4. Results and Discussion
4.1 Students’ Behavioral Engagement
There were forty-one students who participated in the online listening practice every day using
the chatbot. As previously mentioned, students’ behavioral engagement was measured by thecompletion rate of the required task. The descriptive statistics of students’ behavioral
engagement are depicted in Figure 2. The results revealed that the daily completion rate
among students was relatively high and remained stable over the ten days (Mean = 82%, Min= 71%, Max = 88%).
Three key factors have been identified in students’ interviews as the main drivers ofstudents’ high levels of behavioral engagement. First, all 12 students mentioned that theimmediate feedback given by the chatbot encouraged them to complete the listening tasks,
helping them to reflect and revise their previous inputs during the learning process. Besides,
the chatbot alleviated the pressure students might feel while doing the assignments compared809

with receiving feedback from a human instructor. For example, Student K mentioned that if
they misunderstood the question or gave a wrong answer in front of their instructor, they may
feel “anxious and shame”. When students experienced the chatbot providing them withimmediate targeted feedback and multiple chances to answer a question, they showed apreference for “interacting with a chatbot while completing tasks” (Student C). Second,
students were engaged by the task completion function. After completing the listening tasks,
they were able to click a “completed” button and check their task completion status in thechatbot system. Clicking on this button every day provided students with “a sense ofaccomplishment and motivates students to continue by completing the tasks” (Student B).
Clear task instructions from the chatbot helped students understand the learning goals in thedaily tasks and “navigate their online learning easily” (Student D), which gave them a senseof mastery over the learning process.
These features were helpful in supporting students’ competence in performing thelearning activities, which in turn facilitated students’ continued participation in the online
listening learning. This finding is consistent with Niemiec and Ryan (2009) who indicated thatstudents are more engaged in learning when their competence is supported.
100%

80%

83% 88% 85% 88% 88% 85% 78% 78% 78%
71%

100%
80%

60%

60%

40%

40%

20%

20%

0%

29%

20% 15% 17% 17% 15% 17% 15% 15%
12%0%

Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10

Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 Day 7 Day 8 Day 9 Day 10

Figure 2. Student Task-completion Rate.

Figure 3. Journal Completion Rate.

4.2 Students’ Cognitive Engagement
We analyzed the learning journal completion rates to examine student cognitive engagement.
Figure 3 shows a downward trend over ten days, with the rate dropping from 29% to 15% on
the third day and then remaining low. There were 71% students not participating in writing thelearning journal from the beginning.
Interestingly, despite the low level of cognitive engagement in the objective data,
students in the interviews commented positively on the learning journal activities. Students
valued the learning journal in the chatbot because of the increased sense of interaction in acommunity and the insights they could gain from others' experiences. For example, when
students recorded their learning experiences, the chatbot quoted from previous students'
learning journals, such as “Your seniors last semester also mentioned ‘Today's task wasparticularly challenging.’” In this case, the learning journal activity fostered a sense ofconnection with previous students that went beyond just interacting with the chatbot. They
also indicated that sharing others’ learning experiences in the chatbot “served as a helpful
reminder of important subject knowledge” (Student A) and provided “valuable insights into
English listening learning” (Student K).
One explanation for this contradiction between the quantitative and qualitative data
could be that students are not aware of the purpose of writing learning journals and its benefits.
It should be noted that only 29% of the students wrote learning journals at the very beginning.
The low participation indicates that a large portion of students (71%) may not have known that
this activity existed. Nonetheless, more than half of the students who initially participated inthis activity continued to write learning journals over time. The students who wrote the learningjournals appreciated that the chatbot fostered a sense of a learning community by showing
the experiences of previous students. Future designs should include an introductory session
to the chatbot features and closer collaboration between teachers and the chatbot, withteachers explaining the pedagogical benefits of the chatbot learning activity.

810

4.3 Students’ Emotional Engagement
A total of 33 students completed the surveys. Cronbach's alpha results for the social presence
and interpersonal attraction scales were 0.96 and 0.95, respectively. Mean scores for social
presence (M = 3.72, SD = 1.02) and interpersonal attraction (M = 3.61, SD = 1.06) wererelatively high (see Table 2), indicating students perceived the chatbot as intelligent andinteracted effectively. However, the third item of interpersonal attraction had the lowest meanof 3.48 (SD = 1.06), suggesting students held neutral attitudes towards spending additionaltime conversing with the chatbot after task completion.
Table 2. Social Presence and Interpersonal Attraction Scales
ScalesSocialpresence(n = 33)

Interpersonal
attraction(n = 33)

Items
1. I felt as if I were interacting with an intelligent being.
2. I felt as if I were accompanied by an intelligent being.
3. I felt that the chatbot was able to respond to me effectively.
4. I felt involved with the chatbot.
5. I felt that I was able to communicate effectively with thechatbot.
1. I liked the chatbot.
2. I think I could work with the chatbot.
3. I would like to spend more time with the chatbot.
4. I think the chatbot could be a friend of mine.

Mean (SD)3.76 (1.17)
3.76 (1.06)
3.67 (1.05)
3.67 (0.96)
3.73 (0.94)
3.70 (1.07)
3.76 (0.97)
3.48 (1.06)
3.52 (1.15)

The interviews with students revealed several factors that were emotionally attractive
and aversive. Students enjoyed interacting with the chatbot because the “conversations withthe chatbot were relaxed and fun” (Student H) and the chatbot provided “a sense ofcompanionship” (Student E). Two design elements contributed to their emotional engagement:
Emojis and the storage of prior learning records. Emojis made the chatbot lively, while
reviewing learning records provided students a sense of coherence in their learning journey.
A major shortcoming, however, was the chatbot's repetitive responses. All 12 students
reported the pre-set repetitive responses that “made the difference between interacting withthe chatbot and a real human” (Student F). The chatbot used the same prompts daily to guidestudents through the tasks, potentially diminishing their willingness to invest additional timeinteracting with the chatbot.
For future designs of using chatbots to facilitate students’ emotional engagement,teachers are advised to employ a variety of prompts for the learning tasks and avoid repetitive
instructions. The topics of the interaction can be beyond subject knowledge. For instance,
chatbots can start the conversation with casual conversation rather than directly introducing
the learning tasks. Non-verbal cues (emojis) can be used to express the chatbot's feelings.

5. Limitation and Conclusion
The current study has several limitations that suggest opportunities for future research.
First, the short study duration of ten days could be extended in future research to examine
student engagement in chatbot-supported learning over the long term. Second, the small
sample of 41 students may not be representative of other contexts. Therefore, future studies
should include larger and more diverse samples to evaluate the effectiveness of chatbots inonline language learning.
In the present study, we designed a chatbot grounded on the SDT components forstudents’ online English listening learning. Overall, students hold a positive attitude toward theimplementation of the chatbot, which was suggested by their behavioral engagement,
cognitive engagement and emotional engagement. However, the chatbot is not yet mature
enough. Students proposed valuable insights in terms of both pedagogical and technologicalaspects, which will further help us to develop chatbots in the next round.

811

Acknowledgement
This research was supported by the Research Grants Council of Hong Kong RGC Research
Fellow Scheme (Reference no: RFS2223-7H02).

References
Barron Rodríguez, M., Cobo, C., Munoz-Najar, A., & Sánchez Ciarrusta, I. (2020). Remote Learning
during the Global School Lockdown: Multi-Country Lessons. World Bank.
Boutet, I., LeBlanc, M., Chamberland, J. A., & Collin, C. A. (2021). Emojis influence emotional
communication, social attributions, and information processing. Computers in Human
Behavior, 119, 106722. https://doi.org/https://doi.org/10.1016/j.chb.2021.106722
Byrne, D., & Griffitt, W. (1973). Interpersonal attraction. Annual review of psychology, 24(1), 317-336.
Cooper, K. S. (2014). Eliciting engagement in the high school classroom: A mixed-methods
examination of teaching practices. American Educational Research Journal, 51(2), 363-402.
Deci, E. L., & Ryan, R. M. (1985). The general causality orientations scale: Self-determination inpersonality. Journal of Research in Personality, 19(2), 109-134.
Dumford, A. D., & Miller, A. L. (2018). Online learning in higher education: exploring advantages anddisadvantages for engagement. Journal of Computing in Higher Education, 30(3), 452-465.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of theconcept, state of the evidence. Review of educational research, 74(1), 59-109.
Helme, S., & Clarke, D. (1998). We really put our minds to it: Cognitive engagement in themathematics classroom, Teaching Mathematics in New Times. In: Brisbane.
Hew, K. F., Huang, W., Du, J., & Jia, C. (2023). Using chatbots to support student goal setting andsocial presence in fully online activities: learner engagement and perceptions. Journal ofComputing in Higher Education, 35(1), 40-68
Hodges, C. B., Moore, S., Lockee, B. B., Trust, T., & Bond, M. A. (2020). The difference between
emergency remote teaching and online learning.
Indartono, S. (2019). The Influence of Instructional Syntax to Engage EFL Students in Online
Learning Environment. Indonesian Journal of English Language Teaching and Applied
Linguistics, 4(1), 121-132.
Jeon, J. (2021). Chatbot-assisted dynamic assessment (CA-DA) for L2 vocabulary learning anddiagnosis [Article]. Computer Assisted Language Learning, 1-27.
Ji, H., Park, S., & Shin, H. W. (2022). Investigating the link between engagement, readiness, andsatisfaction in a synchronous online second language learning environment. System, 105,
102720.
Jiang, Y., & Peng, J.-E. (2023). Exploring the relationships between learners’ engagement, autonomy,
and academic performance in an English language MOOC. Computer Assisted Language
Learning, 1-26.
Li, J., Kizilcec, R., Bailenson, J., & Ju, W. (2016). Social robots and virtual agents as lecturers forvideo instruction. Computers in Human Behavior, 55, 1222-1230.
Liu, C.-C., Liao, M.-G., Chang, C.-H., & Lin, H.-M. (2022). An analysis of children’ interaction with an
AI chatbot and its impact on their interest in reading. Computers & Education, 189, 104576.
Niemiec, C. P., & Ryan, R. M. (2009). Autonomy, competence, and relatedness in the classroom:
Applying self-determination theory to educational practice. Theory and research in Education,
7(2), 133-144.
Reeve, J. (2012). A self-determination theory perspective on student engagement. In Handbook of
research on student engagement (pp. 149-172). Springer.
Ryan, R. M., & Deci, E. L. (2000). Self-Determination Theory and the Facilitation of Intrinsic
Motivation, Social Development, and Well-Being. The American psychologist, 55(1), 68-78.
Tran, P. (2018). Enhancing Online Language Learning Task Engagement through Social Interaction.
Australian Journal of Applied Linguistics, 1(2), 88-101.
Vazquez-Cano, E., Mengual-Andres, S., & Lopez-Meneses, E. (2021). Chatbot to improve learning
punctuation in Spanish and to enhance open and flexible learning environments. International
Journal of Educational Technology in Higher Education, 18(1), Article 33.
Yang, H., Kim, H., Lee, J. H., & Shin, D. (2022). Implementation of an AI chatbot as an English
conversation partner in EFL speaking classes. ReCALL, 34(3), 327-343.

812

" !   !" " "   #" !
#" !"$  #" ! #"       
       $'&""" !"$ &%*"&"!($%&*#!
  !&$"$ " #'&!!&'%*"&"!($%&*#!
D.;4G6EB 609<B1 0<::.7B:1.? ?D6A.76A .8F<A<B .0 7=56?<.86 <4.A.4:.69 0<:   In the 21st century, self-directed learning (SDL) has become an essential
skill for personal growth and success. This applies to the context of language learning
as well. To enhance students' SDL skills, we previously proposed the DAPER (Data,
Analyze, Plan, Execution, Monitoring, Reflect) model and developed the GOAL (Goaloriented Active Learner) system to support DAPER. In this study, we focused on the
Analysis phase of GOAL. 113 students tracked their scores in 39 weekly tests (19 forEnglish vocabulary and 20 for English grammar) over one year in GOAL. Their selfanalysis behaviors after studying and taking tests were also prompted in GOAL. We
investigated the relationship between analysis behavior and test scores, finding asignificant difference in learning outcomes between students who conducted selfanalysis and those who did not. While the current result cannot definitively conclude
that self-analysis improves study performance in language learning, it does show that
it is meaningful to promote students' SDL behavior tends to perform better. 7052<1*<8;.207*70>*0.$.5/*7*5B<2<!# $.5/
-2;.,=.-5.*;7270

   
(29316?20A21"2.?;6;4("6@.;2@@2;A6.9@86996;A52 @A02;AB?F& ?.:2D<?8 (64;6360.;A.1C.;02@5.C2/22;:.126;A52:2A5<1@D2.0>B6?28;<D92142.;1A52?26@.;2E=9<@6<;6;A52C<9B:2<3?2@<B?02@  ;A56@0<;A2EA=?<:<A6;4("@8699@D6A5
A205;<9<4F@B==<?A5.@/20<:20?B06.9)2< ;.9F@6@@8699@.;2@@2;A6.9=.?A<3("0.;529=92.?;2?@B;12?@A.;1A526?92.?;6;4=?<4?2@@/2AA2?6;09B16;4@AB1F5./6A@=?<4?2@@
<BA0<:2@.;12C.9B.A6;4A5223360.0F<3A526?@AB1F@A?.A2462@$<4B056#0.?A5F
 #.;F=?<4?.:@3<0B@6;4<;12C29<=6;4@AB12;A@("5./6A@<3A2;16@0B@@A2.056;4
:2A5<1@.;1@A?.A2462@<C2?9<<86;4<A52?C6A.93.0A<?@6;("9682=2?@<;.9;221@@293
=9.;;6;4.;1@2932C.9B.A6<;#6.+60A<?6,.9A2?"<085.?A  ,5692.@.0<:=<;2;A<3("@293.@@2@@:2;A5.@/22;@5<D;A<529=@AB12;A@6:=?<C2A526?@=2.86;4@8699@6;
;496@552;
@293.;.9F@6@2E=9<?6;4=2?@<;.9/25.C6<?=.AA2?;@<3.0562C2:2;A5.@;<A/22;@AB16213<?6A@23320A@<;9.;4B.4292.?;6;4 )<3699A56@4.=D2.;.9FG21A526:=.0A
<3@293.;.9F@6@<;A52<BA0<:2@<3;496@5C<0./B9.?F.;14?.::.?92.?;6;46;C2@A64.A6;4
A523<99<D6;4?2@2.?05>B2@A6<;,5.A6@A52?29.A6<;@56=/2AD22;@AB12;A@@293.;.9F@6@
5./6A@.;1A526?=2?3<?:.;026;;496@5C<0./B9.?F.;14?.::.?92.?;6;4





#!#


)52<.9%?62;A210A6C2"2.?;2?%"@F@A2:#.7B:1.?2A.9 6@.@AB12;A
<?62;A211646A.9=9.A3<?:A5.A3<0B@2@<;2;5.;06;4@AB12;A@H(29316?20A21(8699@(( *@6;4A52%"@F@A2:@AB12;A@0.;A?.08A526?.0A6C6A62@.;.9FG2.0A6C6AF1.A.@2A4<.9@

813

2E20BA2:<;6A<?6;4.;1?23920A<;.D612?.;42<3.0A6C6A62@?29.A21A<A526?@AB162@.;1
52.9A52;0<B?.46;4A52:A<A.82<D;2?@56=<3A526?92.?;6;4=?<02@@2@ )52%"@F@A2:
0.;?20<?1.;1C6@B.96G2@AB12;A@=.@A.0A6C6AF1.A..99<D6;4A52:A<0<;36?:A526?=?<4?2@@.;12.@69F4?.@=A526?@6AB.A6<;     @.;2@@2;A6.9@A2=6;&'B@2?@0.;0<;1B0A@6:=92.;.9F@6@A.@8@<;A520<9920A21
1.A.?29.A21A<.;.0A6C6AF )5?<B45A56@B@2?@0.;B;12?@A.;1A526?/25.C6<?.;1612;A63F
6;16C61B.9=?</92:@ )52%"@F@A2:<332?@1.A.6;=BA3<?:3<?D2289FA2@A@0<?2@<?
@F;05?<;6G2@1.A.3?<:<;96;2=9.A3<?:@3<?2<<8@?2.16;4.0A6C6AF3<?A521.A.0<9920A6<;
=5.@2 <?A52.;.9F@6@=5.@21.A.6@C6@B.96G21A<@B==<?AB@2?@A<.;.9FG2:<?2092.?9F
.;1@6:=9F ?<.19FA52@F@A2:0<:=.?2@0B??2;A1.A.D6A5=.@A1.A.<?D6A54?<B=
.C2?.42@ ;9.;4B.4292.?;6;4A52%"@F@A2:<332?@A5?22=.?.:2A2?@.C2?.42A6:2A<A.9
A6:2.;11646A.96G2192.?;6;4<BA0<:2@A<2C.9B.A2A52A2@AA6:2@0<?2.;1@AB1FA6:2 (0<?2@.?22C.9B.A21/F0<:=.?6;40B??2;AA2@A?2@B9A@D6A5=.@A?2@B9A@.;109.@@
.C2?.42@ (AB12;A@0.;0<;36?:A52@21.A.6;=BA0<::2;A@.;12E=?2@@2E=20A.A6<;@3<?
=2?3<?:.;026;A52;2EAA2@A  

;A529.;4B.4292.?;6;4=.?A<3A52%"@F@A2:@AB12;A@0.;0<;36?:A526?("@869992C29@
.;16;3<?:.A6<;?29.A21A<A526?@0<?2@.;1=?2=.?.A6<;A6:26;=.@AA2@A@.@@5<D;6;64B?2


'$ ! ""   #      )52@AB12;A@0.;6;=BAA526?D2289FA2@A@0<?2@.;1A52A6:2A52F@=2;A=?2=.?6;43<?
.=.?A60B9.?A2@A )52;A52F0.;05208A52.C2?.42A2@A@0<?2<3A526?09.@@.;142A3221/.08
3?<:A52@F@A2: )52F0.;.;.9FG2A52@0<?2.;1A6:2D6A5A52@F@A2:=?<:=A@.@@5<D;6;
64B?2 

814




'$)52C<0./B9.?FA2@A@0<?2.;.9F@6@6;A2?3.02-.;42A.9    ;A56@@AB1FA524<.992.?;2?:<129?20<?1@.99@AB12;A1.A.3?<:2.05C<0./B9.?F.;1
4?.::.?A2@A6;09B16;4A52@AB12;A@** *;6C2?@.99F*;6>B2 12;A6362?.@2?6.9;B:/2?
.BA<:.A60.99F.@@64;21D6A56;A52@F@A2:=?2=.?.A6<;A6:23<?A52A2@AA2@A@0<?2@3221/.08<;A2@A163360B9AF.;.9F@6@<3A2@A@0<?2@.;15<=2@3<?A52;2EAA2@A <D2C2?A56@1.A.6@
<;9F?20<?12163A52@AB12;A0<:=92A21A52@2.0A6C6A62@ )52@2.0A6C6A62@.?2;<A:.;1.A<?FD52;.@AB12;A6@./@2;A3<?.A2@AA526?@0<?26@?20<?121.@;B99 





"

        
)56@?2@2.?05D.@0<;1B0A216;.!.=.;2@25645@05<<9 (AB12;A@D2?246C2;;496@5
C<0./B9.?F.;14?.::.?92.?;6;4A.@8@6;12=2;12;A<3A52?24B9.?09.@@?<<:0B??60B9B: )52:.A2?6.93<?92.?;6;4;496@5C<0./B9.?FD.@.D<?1/<<80.9921"&).82<8..;1A52:.A2?6.93<?92.?;6;44?.::.?D.@0.9921+6;A.42(56;<1.-<;2F.:. +<0./B9.?F.;14?.::.?A2@A@D2?20<;1B0A212C2?FD228 .05C<0./B9.?F.;14?.::.?
A2@A0<;@6@A21<3.==?<E6:.A29F >B2@A6<;@.;1A52A<A.9@0<?2D.@ =<6;A@ (AB12;A@
;22121A<@AB1F6;12=2;12;A9FA<.0562C2/2AA2??2@B9A@6;B;6C2?@6AF2;A?.;022E.:6;.A6<;@ )2.052?@6;A?<1B021A52%"@F@A2:.;16A@B@.42.AA52/246;;6;4<3A52A2?: <D2C2?
A52B@2<3%"D.@;<A0<:=B9@<?F/BA7B@A.1C6@21A<822=A?.08.;1.;.9FG2@AB12;A@H
@0<?2@.;1233<?A@0<;02?;6;4A6:2@=2;A=?2=.?6;43<?A52A2@A ,20<9920A211.A.3?<:
3<?D2289F;496@5A2@A@3<?
@20<;1F2.?5645
@05<<9@AB12;A@6;!.=.; )52@2A2@A@6;09B121 C<0./B9.?FA2@A@.;1 4?.::.?A2@A@ .05A2@A5.16A@@0<=212A2?:6;21/23<?25.;1.99<D6;4@AB12;A@A<=?2=.?23<?A52A2@A6;
.1C.;02 3A2?296:6;.A6;46;C.9611.A.@B05.@1B=960.A26;=BA@.;1A2@A1.A.3?<:A52
.1:6;6@A?.A<?D2</A.6;21.A<A.9<3;496@5A2@A@0<?22;A?62@3?<:5645@05<<9@AB12;A@.;1 9<4@?29.A21A<@AB12;A@H.;.9F@6@<3A526?2E.:@0<?2@ %30<B?@2@<:2@AB12;A@:.F5.C2@AB16213<?A2@A@/BA161;A?20<?1.;1:.;.42A526?@AB1FA6:21.A.B@6;4
%" 



815 %&$'&"!"&"&&!$!$%! %*%& 

9<4@92.?;2?@@293.;.9F@6@9<4@92.?;2?@

C<0./B9.?FA2@A@0<?2

 





4?.::.?A2@A@0<?2





 


?<:)./92 0<:=.?21A<A520<9920A21.0A6C6AF1.A.:B0532D2?1.A.D2?2
.;.9FG213<?A52@2.0A6C6A62@/FA52@AB12;A@A52:@29C2@ ,2</@2?C21A5.A<;9F.@:.99=?<=<?A6<;<3@AB12;A@.;.9FG21A521.A..3A2?=?2=.?6;43<?A52A2@A.;1A.86;4A52A2@A 
64B?2 6@.3?2>B2;0F16@A?6/BA6<;05.?AD52?2A52C2?A60.9.E6@?2=?2@2;A@A52;B:/2?<3
=2<=92D5<A<<8A52.0A6C6AFA5.A:.;FA6:2@6;3<B?.0A6C6A62@.;1A525<?6G<;A.9.E6@
?2=?2@2;A@A52;B:/2?<3A6:2@.;.0A6C6AFD.@A.82;6;3<B?.0A6C6A62@           
.@21<;64B?2 D216C61216;A< 4?<B=@@AB12;A@D5<5.C2.;.9FG21A526?
@0<?2@.3A2?/<A5A52C<0./B9.?F.;14?.::.?A2@A@:<?2A5.; A6:2@.;1.;<A52?/26;4
@AB12;A@D5<5.C21<;292@@A5.; A6:2@  3.@AB12;A5.@.;.9FG21A526?@0<?2@:<?2
A5.; A6:2@6;A524?.::.?A2@A@<BA<3.A<A.9<3 A2@A@.;1:<?2A5.; A6:2@6;A52
C<0./B9.?FA2@A@<BA<3.A<A.9<3 A2@A@D20.;@.FA52F5.C212C29<=21.5./6A<3@293
.;.9F@6@ )<.;.9FG2D52A52?.@AB12;AD5<5.@12C29<=21.5./6A<3@293.;.9F@6@D699
6;39B2;02A526?@AB1F=2?3<?:.;026;9.;4B.4292.?;6;4<?;<AD20.90B9.A21A52.C2?.42<3
A52@0<?2@<3?<B=D56055.@.;.9FG21:<?2A5.; A6:2@.;1?<B=D5605D6A5
32D2?A5.; A6:2@<;A52C<0./B9.?F.;14?.::.?A2@A@2C2?FA6:2.;10<;1B0A21.AA2@A           

)5216332?2;026;A2@A@0<?2@<3@AB12;A@D6A5.;1D6A5<BA@293.;.9F@6@<3A2@A@0<?2@.3A2?
A52@.:2A2@A@.?2.@3<99<D@ '$$!!&%&%"$%"%&'!&%)&!)&"'&&%!*%%

816

$#"$&!
 !"$ &"!"%"$%"&%&'!&%)&!)&"'&&&"!*%%
&%!*%%
&

&"'&%!*%%
& 

&&%&%

#('        

         













;A56@@AB1FD2.;.9FG21A526:=.0A<3@293.;.9F@6@<;A52<BA0<:2@<3;496@5C<0./B9.?F
.;14?.::.?92.?;6;4.;13<B;1.@64;6360.;A6:=.0A6;;496@5C<0./B9.?F92.?;6;4A5?<B45
A52?2@B9A@ )52=2?3<?:.;02<3@AB12;A@D6A5@293.;.9F@6@5./6A@6@56452?A5.;A5<@2
D6A5<BA@293.;.9F@6@6;C<0./B9.?F92.?;6;4 (<D20.;@.FA5.A5.C6;4.5./6A<3@293
.;.9F@6@D699529=@AB12;A@92.?;;496@5C<0./B9.?F/2AA2? %;A52<A52?5.;12C2;63A52
.C2?.42<3@AB12;A@D6A5@293.;.9F@6@5./6A@<;4?.::.?A2@A@6@56452?A5.;A5<@2D6A5<BA
/FA52?2@B9A<3A52AA2@A6A0.;;<A/21236;6A6C29F@.61A5.AA52?26@.16332?2;02 ;1,2A56;8
A5.AA529.08<3@64;6360.;A6:=.0A<;4?.::.?92.?;6;4:645A/21B2A<A523.0AA5.A4?.::.?
92.?;6;46@=<@@6/9F.:<?20<:=92E=?<02@@6;C<9C6;4A52B;12?@A.;16;4<3?B92@.;1
@A?B0AB?2@ +<0./B9.?F.;14?.::.?:.F6;C<9C216332?2;A0<4;6A6C2.;192.?;6;4=?<02@@2@ (293.;.9F@6@:645A/2:<?2@B6A./923<?C<0./B9.?F92.?;6;4D5605?2>B6?2@:2:<?F.;1
?20.99?.A52?A5.;?B92/.@2192.?;6;46;4?.::.?  A0<B91.9@</2=<@@6/92A5.AA52:2A5<1@
B@21A<.@@2@@4?.::.?@8699@.?2;<A@2;@6A6C22;<B45A<12A20AA526:=.0A<3@293.;.9F@6@ )52@2?2.@<;@5.C246C2;B@16?20A6<;.;16;@645A@3<?3BAB?2?2@2.?05 )5236;16;4@<3A56@@AB1F.964;D6A5A52=?<=<@.9<3A52&':<1292:=5.@6G6;4
A526:=<?A.;02<3@293.;.9F@6@6;A5292.?;6;4=?<02@@ F.;.9FG6;4A526?<D;1.A.@AB12;A@
0.;@2A@=2063604<.9@=9.;A526?92.?;6;4@A?.A2462@2E20BA2A52::<;6A<?A526?=?<4?2@@.;1?23920A<;A526?<BA0<:2@ )52%"@F@A2:@B==<?A@A56@=?<02@@/F=?<C616;4.
=9.A3<?:3<?@AB12;A@A<?20<?1.;1?23920A<;A526?@29316?20A2192.?;6;4.0A6C6A62@ )52
@64;6360.;A6:=.0A</@2?C216;C<0./B9.?F92.?;6;4A5?<B45@293.;.9F@6@12:<;@A?.A2@A52
23320A6C2;2@@<3A56@.==?<.05D5692A526;0<;09B@6C2?2@B9A@6;4?.::.?92.?;6;456459645A
.?2.@3<?3B?A52?6;C2@A64.A6<; 3A2?12:<;@A?.A6;4A5.A@293.;.9F@6@.61@@AB12;A@6;92.?;6;4C<0./B9.?F.;15.@@<:26:=.0A<;92.?;6;44?.::.?D23<0B@<;12C29<=6;4A525./6A<3@293.;.9F@6@6;
@AB12;A@ )52;D216@0<C2?21.;6;A2?2@A6;4=52;<:2;<;  3D20<;@612?A5.A@AB12;A@.;.9FG21:<?2A5.; A6:2@5.C2.5./6A<3@293.;.9F@6@A52;A5<@2.;.9FG213?<: A< 
A6:2@6;2.05.;.9F@6@.0A6C6AF.?2@AB12;A@D5<.?2A?F6;4A<0B9A6C.A2A526?5./6A@<3@293
.;.9F@6@ ,25.1.5F=<A52@6@A5.AA52@2@AB12;A@@5<B915.C2A5?22@A.42@6;A526?@293
.;.9F@6@;<A0<;1B0A6;4.;.9F@6@.AA2:=A6;4.;.9F@6@.;13?2>B2;A9F0<;1B0A6;4.;.9F@6@
.;1A52;A?.;@3<?:6;4A<A52;2EA@A.42.@A6:205.;42@ <D2C2?A52.;.9F@6@1.A.3?<:
A52.0A6C6A62@<3@AB12;A@@5<D@A5.AA52FA2;1A<0<;1B0A.;.9F@6@.0A6C6A62@1B?6;4.@=206360
=2?6<1 )56@6;09B12@/<A5A52@AB1FA6:2.;1A2@A@0<?2@3<?C<0./B9.?F.;14?.::.?A2@A@ 
BAA52F0.;A@A608D6A56A )52@=206.9=2?6<152?21<2@;<A?232?A<.36E21A6:2 2.05
@AB12;A5.@.@=206.9=2?6<1/BA6A@.9@<=<@@6/92A5.AA52F1<;A 3A2?:.6;A.6;6;4.02?A.6;
=2?6<1<3@293.;.9F@6@.0A6C6A62@A52FD699@A<=A52.;.9F@6@.0A6C6AF )52=?.0A60.96:=960.A6<;@<3A56@@AB1F<332?6;@645A@6;A<@293.;.9F@6@A5.A0.;
2;5.;02C<0./B9.?F.0>B6@6A6<;2;0<B?.46;4@AB12;A@A<.0A6C29F2;4.426;A52=?<02@@ (6:B9A.;2<B@9FA52.==960.A6<;<3A52%"@F@A2:?26;3<?02@A56@=?<02@@3<@A2?6;4A52
12C29<=:2;A<3@AB12;A@@29316?20A2192.?;6;4.;1@293.;.9F@6@@8699@ )5236;16;4@.9@<56459645AA52;2213<?3B?A52??2@2.?05A<B;12?@A.;1D5F@AB12;A@:.F@A?B4492A<:.6;A.6;
@293.;.9F@6@5./6A@92.16;4A<=<A2;A6.96;A2?C2;A6<;@A<@B==<?A@B@A.6;212;4.42:2;A 

817

<?3BAB?2D<?8D2.6:A<.;.9FG2D5F@293.;.9F@6@@5<D@.56452?6:=.0A<;
529=6;4@AB12;A@6;C<0./B9.?F92.?;6;4A5.;4?.::.?92.?;6;4 ,2.9@<D.;AA<C2?63F
3B?A52?@293.;.9F@6@@6:=.0A<;9.;4B.4292.?;6;4<BA0<:2@.;1<A52?@92.?;6;4<BA0<:2@ AA52@.:2A6:2D25.C2.9@<0<9920A211.A.<;A52.:<B;A<3A6:2@AB12;A@@=2;A
=?2=.?6;43<?2E.:@.@D299.@A526?.;.9F@6@.;1?23920A6<;@<;A526?=?2=.?.A6<;A6:2 ,2
5<=2A<.;.9FG2D52A52?A52?26@.?29.A6<;@56=/2AD22;A56@.;1A526?@AB1F=2?3<?:.;02 
116A6<;.99FD2D.;AA<.;.9FG2A52?2.@<;@D5F@AB12;A@0.;;<A@A608D6A5.;.9F@6@
.0A6C6A62@.;1/.@21<;A52?2.@<;@=?<=<@2:2A5<1@A<2;0<B?.42@AB12;A@A<12C29<=6;
@293.;.9F@6@D2.6:A<=?<C612:<?2@B6A./92@B==<?AA<=?<:<A2@AB12;A@H12C29<=:2;A<3@293.;.9F@6@.;1("@8699@ !

)56@D<?8D.@@B==<?A216;=.?A/FA523<99<D6;44?.;A@!(&(!&!&$&.;1!&$& 








.;1$%



;*6.@8;4/8; <=,.7=>;B5.*;72701==9@@@9 8;08>;@8;49 /;*6.@8;4
%*4.84*
! 
..%.8%
8<=.;270<.5/-2;.,=.-5.*;7270@2=1%7$1*2"'*70-<
&()#"(*%&##&(*",#(%"% 99 D $270*98;.!.*;<87*3>6-*;#)>*7)>*7)*70>2B87024C*9F7*;5*7*0*7 0*=*
 $B<=.6=8$>998;=.*;7.;E<,:>2<2=2878/$.5/2;.,=287$4255<(&"% )&*! *!
%*(%*"&%#&%(%&%&$'+*()"%+*"&%  99    
2*&2,=8;2'*5=.;8,41*;= 71*7,2706.=*,8072=28727<.5/-2;.,=.-5*70>*0.5.*;7270-)*$&#+$ ))+ 99 
80>,12,*;=1B%
#./5.,=2?.$.5/<=>-B8<=.;270.*;7.;>=8786B7$=84.
-  &%(%(&"% )%84B8*9*7
!*8*718>'.2*71.7
A958;*=8;B$=>-B8/=1.#.5*=287<129+.=@..7
$.5/2;.,=.-.*;7270*7-,*-.62,!.;/8;6*7,.27*'.+*<.-.*;72707?2;876.7=
%#"%&+(%#&")*%(%"% $"%")*(*"&%&85>6.( >6+.;$9;270
$1278-*$%*4.84*
 "%*  ("*"&% )>*7B>*7)*70>2B8702#@2=*32=*3>6-*;2;8*42 0*=*
 <B<=.6/8;87527.
<.5/-2;.,=2879;*,=2,..A958;270<=>-.7=<E+.1*?28;*59*==.;7<*7-=1.269*,=87*,*-.62,
*,12.?.6.7=27=1.1201<,1885,87=.A= &+(%#&&$'+*()"%+*"&% 818Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

AI-powered Collaborative Activities for
Chinese Vocabulary Learning
Xinyu GUO, Yun WEN*
National Institute of Education, Nanyang Technological University, Singapore* yun.wen@nie.edu.sg
澔
Abstract: In recent years, Artificial Intelligence (AI) has significantly increased in digitalsecond language (L2) learning, particularly in supporting vocabulary acquisition.
However, research on how AI might facilitate collaborative vocabulary learning is stillin its new stage. This study works on investigating the effectiveness of AIrecommended contexts in fostering collaborative language learning among young
learners. The research employed a self-developed AI-empowered Chinese vocabulary
learning system called ARCHe, which was implemented in primary schools in
Singapore. A mixed-methods case study approach was conducted with the 2nd-gradestudents who spoke English as their first language. Preliminary findings indicate thatlearning Chinese with ARCHe effectively enhances the academic performance ofyoung learners, with the AI-empowered self-generated contexts feature exhibiting a
positive impact on collaborative language learning performance. The study offers
insights into the integration of AI in digital language learning, with the potential toenhance L2 learning outcomes for young learners.
Keywords: Artificial intelligence, Collaborative learning, Vocabulary learning, Learnergenerated Context澳

1. Introduction
In recent years, there has been an unprecedented increase in the use of Artificial Intelligence(AI) in digital language learning, with the pandemic further accelerating its adoption. AI
technologies such as machine learning, automatic speech recognition, and natural language
processing (NLP) have been utilized to provide automated feedback on student writing (Wu,
Wu & Zhang, 2021; Godwin-Jones, 2022) and personalized reading recommendations (Hsu,
Hwang, & Chang, 2013; Xiao & Hu, 2019). In the context of Chinese as an L2, AI has shown
significant potential in supporting vocabulary learning through character pronunciation,
recognition, and writing (Kuo et al., 2022). While the importance of learning contexts invocabulary acquisition is widely recognized (Atkinson, 2002; Wen, 2021), AI may facilitate
interactions among students, recommends learning resources, and automatically generates
contexts to foster collaborative learning. However, there is limited research on investigating
the role of AI in enriching contextual information, promoting interactions, so as to enhancing
collaborative learning for L2 learners.
An AI-enhanced Chinese vocabulary learning system, ARCHe, was designed and developedby our research team. In the ARCHe-supported collaborative learning activity, group students
are instructed to construct sentences using provided scenarios. The system can automatically
recommend students’ artifacts, by which group students will be triggered to create, review,
and improve their artifacts continuously. The purpose of this paper is to investigate whether
and how AI-recommended contexts can help to promote young learners’ collaborative Chinese
vocabulary learning. The research questions are as follows:
1. Did the use of ARCHe help to improve young learners’ Chinese character learning?
2. How did AI-recommended contexts promote students’ collaborative vocabulary learning?

2. AI-powered vocabulary learning

819

Vocabulary learning, as one of the fundamental language skills has received considerable
attention in recent studies on technology-enhanced language learning (Burston, 2015). The
advent of mobile devices has provided L2 learners with real-time, convenient, and context-richlearning opportunities, particularly in the realm of vocabulary acquisition. Notably, AI tools
have emerged as effective means to enhance vocabulary learning. These tools empower
students to streamline their language learning journey by delegating specific tasks andreceiving personalized learning experiences tailored to their individual needs and progress(Roxana & Fabián, 2023).Considering the potential of AI language generation systems forvocabulary practice and providing diverse assistance (Woo & Choi, 2021), we hypothesizethat incorporating this feature into L2 learning tools can promote collaborative interaction.

3. ARCHe Design
3.1 ARCHe collaborative learning activities
The ARCHe system comprises two components: a home-based pre-collaborative activity(asynchronous) and a class-based collaborative activity (synchronous) (Wen, 2022). In thehome-based activity, students use the ARCHe learning system to complete assigned
foundation and creation tasks. The class-based activity involves scenario-based tagging andartifact construction as a group. During the class-based activity, students work in small groups,
labeling points of interest in scenario pictures and generating group artifacts. The system's
automatic recommendation feature enriches contexts with self-generated artifacts (Figure 1).
As students submit group artifacts, the AI-powered system extracts keywords andrecommends related learner-generated artifacts for reference, fostering discussions, peer
learning, and continuous refinement and improvement of group artifacts.

Figure 1. Self-generated contexts recommendation enabled by ARCHe.

4. Methodology
This research employed a mixed methods approach with the objective of examining andcontextualizing the effectiveness of AI-powered collaborative activities for Chinese vocabulary
learning by enriching self-generated-contexts.

4.1 Participants
Table 1. Details of Participants
Teacher
Class
T1

C1

Number of
students
20

T2

C2

17

820

Number ofgroups7
6

Targetgroups
T1G1
T1G4
T2G1
T2G5

Ability level
Lower
HigherHigher
Lower

The study was conducted in two 2nd-grade classes at School A, with a total of 37 student
participants. Among them, we selected two groups from each class as the target groups.

4.2 Intervention Procedure
Each class participated in the project for one semester, engaging in class-based collaborative
learning activities conducted once every two weeks. Before each class-based session,
students were required to complete specific home-based learning tasks. During these classbased collaborative sessions, teachers summarized the performance of the home-based tasks
and initially assigned group tasks. With the assistance of the teachers, each group completed
a scenario-based tagging and sentence-making task, subsequently reviewed the work of othergroups, and provided comments and ratings. The teacher circulated as a facilitator andcollaborator throughout the learning process.

4.3 Data Source and Analysis
In this mixed methods study, quantitative data included pre- and post-tests, while qualitative
data was collected through various means. Screen recordings, accompanied by audio feedsof student discussions, were captured on mobile devices within each group. Additionally, indepth structured post-interviews were conducted with teachers and focus groups to provide
qualitative insights for in-depth analysis and to interpret the quantitative data. The focus groupinterviews with young students covered three main areas: group learning task completion,
learning mode preferences, and collaboration satisfaction with ARCHe. Moreover, postinterviews were conducted with teachers to gather feedback on teaching using ARCHe and to
capture their overall user experiences.

5. Findings
5.1 The use of ARCHe helped to improve learners’ Chinese character learning
The comparison of the pre & post-tests demonstrates that ARCHe benefits student learning.
A total of 37 students from School A. Class 1 and Class 2 completed all the pre-and posttests. In Class 1, the mean pre-test score was 11.85 (SD=3.45, n=20) out of a total of 30
points. Following the intervention, the mean post-test scores significantly increased to 16.15(SD=6.09, n=20). The paired t-test revealed a significant improvement in performance among
Class 1 students (t=3.91, p<.001). Similarly, in Class 2, the mean post-test score was 16.118(SD=5.86, n=17), which was higher than the mean pre-test score of 12.82 (SD=4.23). The ttest also indicated a significant difference between the two scores (t=3.18, p < 0.05).
Table 2. Comparison of pre- and post-tests
Class
Test
No. ofEstimated Marginal
students
Means
Class 1 Pre-test
20
11.850Post-test
16.150Class 2 Pre-test
17
12.824Post-test
16.118

SD

t

P

3.453
6.089
4.231
5.862

3.857

<.001*

3.182

.003*

The results of the group artifacts scores provide evidence that ARCHe's learning activities
effectively improved the quality of group artifacts. As shown in Table 2, both Class 1 and Class
2 students showed progress in creating artifacts throughout the intervention process. While
comparing language skills, we found that students made more significant progress in thecontent of artifacts.

5.2 The recommendation enabled by ARCHe promoted self-context generation
To explore how AI-recommended contexts promote students’ collaborative vocabulary
learning, we examined how group interaction took place with the AI-recommended context.
The results revealed that the AI-recommendation effectively motivated young students to821

review and learn from their peers' artifacts, so as to promote self-context generation in thecollaborative learning activity.
Under the guidance of the teachers, intragroup members began exploring peers' artifacts
through the self-generated context recommendation, which contained keywords relevant totheir own artifacts. During the first lesson in intervention, Teacher 1 and Teacher 2 ascollaborators, actively encouraged every group to utilize the self-generated context
recommendation. For example, after group 6 of class 1 completed generating the artifacts "ㆹ㚱ᶨ᷒⥸⥸ (I have a sister), " Teacher 1 suggested: "Let's look at other sentences about '⥸⥸(sister).'" Then all group members agreed with this suggestion, reviewed and read out one
sentence from the self-generated contexts recommendation: "⥡⥡㊧䛨᷄大⛐㇧斜慴䍑 (mysister is playing with something in her room)" and appraised that was a good sentence濁To facilitate further discussion, we observed that high-level and low-level learning groups
employed different strategies in utilizing the AI-recommended contexts. The verbal exchanges
from four individual groups are presented in Table 3. For the low-level groups, therecommendation proved more helpful in providing inspiration for generating additional groupartifacts. On the other hand, the high-level group, leveraging their judgment skills based on
their language proficiency, preferred to comment on and evaluate their peers' artifacts using
the recommendation. Their group work through the recommendation primarily focused on
improving existing artifacts rather than generating new ones.
Table 3. Examples of intragroup interactions took place in 4 target groups when using the AIrecommended contexts
Focus
group Level
Group Talk Content
Analysis
T1G1

Low

T1G4

High

T2G1

Low

T2G5

High

(Click” ↧䅨 [Animal]” from self-generated context
recommendation)¾ Group member 1: ᷕ䐧䖃㔮 ‘䘞⡆⼩’ȼ
[They used ‘so weird’]¾ Group member 2: 廘ᴩ↧䅨䖃⋤⩏⡼῎㘈䘊廆ȼ[I saw this sentence before]¾ Group member 3: ㄐᷫ⋮ᷤ⩥廘ᶚₘȼ
[We can learn from these sentences and write
similar one]
(Click” 旧ḝ [Umbrella]” from self-generated context
recommendation)¾ Group member 1: ㄐᷫ㮠㓵擳ᶅȼ[We don’t have enough time]¾ Group member 2: 媨ㄐᷫ₌䘊䘊㘈㮠㘈㗳⡼䖃⋤⩏[let's see whether others have better sentence.]¾ Group member 1: ㄐ壈⺖ㄐᷫ䖃㫓庂⡼ȼ
[I think our sentences are better](Click” 㕙ᴉ [Night]” from self-generated context
recommendation)¾ Group member 1: ⒓漓⺇⠙⺇⠙⋤⩏漓䩈᳿ᴊㄐᷫ⋮ᷤ copy ᷕᷫ䖃⋤⩏ȼ
[wow, lots of sentences. Later we can copy theirsentences.]¾ Group member 2,3: ⡼ȼ[Ok]All three students ofthis group were
engaged in the talk.
They prefer to write
more similarsentences bybrowsing peers'
artifacts.

(Click” 䈳⩏ [Monkey]”,”㘊⋊[Friend]”,”⺇⠙[very]” fromself-generated context recommendation)

All students in thisgroup enjoyed822This group reviewed
and evaluated peers'
artifacts bycomparing with theirown artifacts, which
promote moremeaningful
discussion during
collaboration.
Since this group
have limited
language ability, they
tried to findinspiration of
artifacts generation
by reviewing peers’
artifacts.

¾
¾
¾

¾

Group member 1: 㘈⺇⠙⁲ᶍ “㘊⋊” 䖃⋤⩏ȼ
[Lots of sentences about “friend”]
Group member 2: ᷕᷫ䖃⋤⩏㮠㘈ㄐᷫₘ⺖⡼ȼ
[Their sentences are not as good as ours]
Group member 1: ṅ廘ᴩ⋤⩏⡼ȼ[But this sentence is good](review and read out lots of sentences)Group member 3: ⠩⠙⋤⩏ᶅ漓㮠㓵擳ᶅ漓ㄐᷫᴌ墀䘊ᶅȼ[Two much sentences, the task time is notenough for us to review more.]

reviewing andcommenting peers’
artifacts review andevaluation. By
comparing peers’
artifacts, theydecided if they takefurther action in theirown artifacts’improvement.

In the post-interview, regarding the AI-recommended feature, all students expressed theirenjoyment of clicking on labels and reviewing their peers' sentences. The two teachers also
agreed that this feature stimulated group discussions, idea sharing, and peer learning,enhancing collaborative learning among young learners.
Furthermore, the enthusiasm for utilizing AI-recommended contexts varied between the focus
groups from Class 1 and Class 2. Class 1 students displayed curiosity about their peers'
artifacts and enjoyed discussing these sentences within their groups. This observation
indicates potential differences in teaching approaches between Teacher 1 and Teacher 2.
Post-interview findings revealed that Teacher 1 had a consistent focus on developing students'
collaborative learning skills from primary 1 up to the current stage. In contrast, Teacher 2 did
not extensively discuss collaborative learning during the interview. These findings underscorethe importance of considering students' motivation levels and teachers' technological andpedagogical knowledge when integrating AI into language classrooms.

6. Conclusion and limitations
The study demonstrated that the collaborative vocabulary learning activities facilitated by
ARCHe effectively promoted young learners' academic performance and showed progress increating group artifacts throughout the intervention process. The positive feedback received
from both students and teachers indicates that young learners enjoyed engaging incollaborative vocabulary learning with ARCHe, and teachers observed improvements instudents' collaborative learning performance.
The utilization of AI-recommended contexts promoted young students to review and learn fromtheir peers' artifacts. It also enhanced collaborative performance by stimulating self-context
generation by idea sharing and discussion. The teachers' guidance in encouraging students
to utilize the recommendation function played a vital role in this process. Additionally, thevariation in enthusiasm observed between the focus groups from different classes highlighted
the importance of effective teaching pedagogy and the role of teachers in fostering successful
collaborative learning experiences.
Moreover, the study revealed that different strategies were employed by high-level and lowlevel learning groups in utilizing the recommendation, with the focus being on artifact
generation for the low-level groups and artifact improvement for the high-level groups. This
indicates that the effectiveness of using automatic recommendation to modify group artifacts
depends on the learners' level of competence. Compared to have students reviewing the selfgenerated contexts recommendation sentences during the collaborative learning process bythemselves, the part of teacher's comments and summary of all groups’ artifacts were more
efficient for helping lower-level students’ artifacts improvement.
There are some limitations to this study. Firstly, due to the page constraints, we were unable
to provide extensive details about the students’ group learning processes. Secondly, the823

effectiveness of students’ character learning was only reported based on students’ pre-and
post-test scores. Additional details will be reported in our forthcoming studies.

Acknowledgements
This study is funded by Education Research Funding Programme, National Institute of
Education (NIE), Nanyang Technological University, Singapore, with project no. DEV 05/21
WY.

References
Atkinson, D. (2002). Toward a sociocognitive approach to second language acquisition,
Modern Language Journal, 86, 525–545.
Burston, J. (2015). Twenty years of MALL project implementation: A meta-analysis of learning
outcomes. ReCALL, 27(1), 4-20.
Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructedlanguagelearning.
Language
Learning&
Technology,
26(2),
5–24.
http://doi.org/10125/73474
Hsu, C. K., Hwang, G. J., & Chang, C. K. (2013). A personalized recommendation-based
mobile learning approach to improving the reading performance of EFL students.
Computers and Education, 63, 327–336. https://doi.org/10.1016/j.compedu.2012.12.004
Kuo, Y.-C., Yao, C.-B., & Wu, C.-Y. (2022). A Strategy for Enhancing English Learning
Achievement, Based on the Eye-Tracking Technology with Self-Regulated Learning.
Sustainability, 14(23), 16286. MDPI AG.
Roxana, R. F. D. L. V., & Fabián, G. A. (2023). Exploring the Benefits and Challenges of AILanguage Learning Tools. International Journal of Social Sciences and Humanities
Invention 10(01): 7569-7576, 2023, 2349–2031.
https://doi.org/10.18535/ijsshi/v10i01.02
Wu, L., Wu, Y., & Zhang, X. (2021). L2 Learner Cognitive Psychological Factors About Artificial
Intelligence Writing Corrective Feedback. English Language Teaching, 14(10), 70.
https://doi.org/10.5539/elt.v14n10p70
Wen, Y. (2021). Augmented reality enhanced cognitive engagement: Designing classroombased collaborative learning activities for young language learners. Educational
Technology Research & Development, 69, 843–860
Wen, Y. (2022). Design of an AI-powered seamless vocabulary learning for young learners.
In Sridhar Iyer et al. (Eds.), Proceedings of the 30th International Conference on
Computers in Education Conference (Vol. 2, pp. 659-661). Asia-Pacific Society for
Computers in Education. https://icce2022.apsce.net/proceedings/volume2
Woo, J. H., & Choi, H. (2021). Systematic Review for AI-based Language Learning Tools.
http://arxiv.org/abs/2111.04455.
Xiao, Y., & Hu, J. (2019). Assessment of Optimal Pedagogical Factors for Canadian ESL
Learners’ Reading Literacy Through Artificial Intelligence Algorithms. International
Journal of English Linguistics, 9(4), 1. https://doi.org/10.5539/ijel.v9n4p1

824

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Design and Development of a Sentence
Construction Game for Deaf and Hard of
Hearing (DHH) Users: A Qualitative
Usability Study
Arjun PRASADa*, Sunny PRAJAPATIa* , Utkarsha, Vishwas BADHEa
aIndian Institute of Technology Bombay, India*22d1204@iitb.ac.in, 22d1201@iitb.ac.in
Abstract: Concerns about the writing proficiency of deaf students in the English
language are well known and persistent. Many of them prefer to write concisesentences with fewer adjectives and adverbs and exhibit a variety of syntax usage
errors. The purpose of this study was to evaluate the usefulness of a learningintervention created to help DHH students who struggle with sentence formation. The
users played with the digital prototype and the perception of the usability of theintervention was analysed and represented using a thematic analysis. Primary insights
on the usability and design of this game-based early intervention revealed that thecontent was well received but more gamification components must be added for theintervention to be both enjoyable and pedagogically valuable. Incorporating moregamification components, conjugating verbs, and studying the effectiveness of theintervention in addressing the difficulty of DHH students in framing sentences will be
the focus of the future extension of the study.
Keywords: DHH students, sentence construction, learning intervention, gamification

1. Introduction
The educational experiences of Deaf and Hard of Hearing (DHH) students are significantly
diverse and unique compared to students in general education classrooms and students fromother disability groups. The Indian educational system has to be strengthened to cater theireducational needs as language acquisition remains a major challenge in their life (Mohanty &Mishra, 2020). Studies reported that many DHH individuals struggle with the process of writing
and different aspects of writing including phonology, morphology, lexicon, grammar and syntax(Mayer & Trezek, 2019).
Further, DHH children who have limited or no access to spoken language sounds may
also find it difficult to learn the rules and patterns of a language like English, particularly
because English orthography uses phonemic alphabets (Cannon & Kirby, 2013). Many
students also engage in gestures right from a young age to communicate their ideas, which
may also contribute to their difficulty in sentence construction during formal education.
Teachers employ a variety of resources to enhance instruction and learning inclassrooms, including the use of YouTube videos, guided activities and experience-basedlearning. Teaching tools and artefacts, such as flashcards, diagrams and rate of speech andhearing charts are used in many classrooms. In contrast to traditional classrooms, it isfrequently challenging to teach a full lesson to the students using just speech and gestures.
As a result, the key issues facing educators of DHH children continue to be broadening thescope of what may be communicated to the pupils and bridging the gap between what is stated
and what is understood (Mohanty & Mishra, 2020). Hence, to meet their educational demands,
in addition to better teacher training programs, the teaching materials must be carefully
designed. Studies have shown that early intervention helps youngsters develop their linguistic,
communication, cognitive, and social abilities more successfully (Marschark et al., 2011). DHH

825

children who receive early intervention programmes perform better than their peers who do
not in terms of language, social development, and early academic achievement (Mohanty &Mishra, 2020).

2. Literature review
A need identification study conducted by Adnyani et al., 2021 highlights the emphasis on thedevelopment of multimedia-based English learning materials in addition to highlighting thestudies that have put forth the need for incorporating images, visual scaffolds, fingerspelling,etc. to enhance the learning. In another study by Chan et al., 2022, a systematic investigation
on French literacy was done for Special Education Needs (SEN) and DHH students withserious games. The study also emphasizes on the improvement in the cognitive skill provided
by games to such user groups. One of the major reasons pointed out with respect to poor
linguistic comprehension is the lack of syntactic and semantic knowledge by an individual.
The focal question was on how to design an early intervention to improve the level offormal communication in DHH students by addressing their challenge to frame sentences
involving conjugating verbs. Studies have recommended the usage of gamification approachin activities involving repetition to keep up the motivation of the user (Faiella and Ricciardi,
2015). Little evidence is available on whether the development of speech or sign language or
a combination of both would be beneficial for DHH children. Hence, making an educational
programme that defies homogeneity and caters to the unique educational needs of eachcategory (age group/ grade) of students with hearing loss may hold the key to finding thesolution. In this paper, we attempt to give an overview of the design and usability of a learningintervention focused on teaching elementary English sentence construction among DHH
students which can be further extended to address the challenge involving sentence
construction using conjugating verbs.

3. Design and Development of the intervention
The broad objective was to design an intervention that supports special needs education(formal/ informal) of DHH users. The design and development of this game-based learningintervention was carried out in three phases by a group of four researchers as part of the
Human Computer Interaction course. The phases included:1) Needs Analysis, 2) Ideation andConceptualization, and 3) Prototyping. These are elaborated as follows:

3.1 Needs Analysis
The researchers interacted with DHH students at a school with the help of the teachers andconducted a semi-structured interview with the teachers. Further, three therapists who provide
early intervention to DHH students for their learning needs were also interviewed through asemi-structured interview. The discussions were capitulated in the form of a concept map
providing us a peek at the interconnections in various aspects of their life and guided us
towards identifying a problem, through a 5 Why analysis, that appeared important andaddressable given our expertise. The key issues identified include difficulty in constructing
formal sentences in any language other than sign language, difficulty in engaging in in-person
conversations, difficulty in memorizing long sentences, for example, while learning topics inhistory as part of school curriculum, where formal long and complex sentences in English areessential. Further exploration through fishbone diagrams and empathy maps was done to get
deeper insights into the problem. Additionally, a user persona was also prepared to guide thedevelopment of the intervention. Grounded on these analyses, the core problem statement
was refined as “Design of an early intervention to improve the level of formal communicationin DHH children by addressing their challenges to frame sentences involving conjugating
verbs”.

826

3.2 Ideation and Conceptualization
The four group members discussed the concept map and interview data thoroughly and thenutilised brainwriting technique to generate 48 ideas. These ideas were then grouped based on
the major tools/ technologies they utilised. Further development of ideas and the addition of
new insights and inspirations also occurred. Following this process, a Decision Matrix analysis
was performed to obtain the optimal solution. Each of the four members conducted theanalysis individually, based on the collectively decided factors, which were later incorporated
into a single Decision Matrix via averaging. The outcome of this process was the finalisationof the idea brief – “Develop puzzles/ games (tangible/ digital) that teach English grammar andsentence construction”.
Following the ideation process, the paper prototype was created to further understandthe interactions associated with the elements of the puzzles. Along with the playability aspectof the game, the paper prototyping also helped in conceptualising the sequence of events,
which will address the learning objective. The paper prototyping process was informed by thepedagogical references and the gameplay ideation.

3.3 Prototyping
The prototype of the puzzle was further developed using
Scratch, a no-code platform suitable for creating
educational tools and games. This allowed us to add
many game design elements, such as rewards, levels,
goals, rules, and progress. The focus of the team at thisstage was to create a functional prototype. Figure 1
shows the Scratch coding environment and the userinterface for the game prototype, which was then tested
with users. The game's objective was to help users
create sentences using blocks containing words, shaped
like puzzle pieces that best describe a given image on
the screen. The prototype game had two levels, which ismade available on the Scratch project page (Level-1:
https://scratch.mit.edu/projects/743654775/fullscreen/;
Level -2: https://scratch.mit.edu/projects/ 755127867/
fullscreen/ )4. Gameplay and Scaffolding
In the game, the user was presented with puzzle pieces
on which words were written, which were clustered
together into three different bins based on the parts ofspeech they belonged to. In this iteration, these bins
were subject, verb, and object. The content in thestandard textbooks for grades 1-2 also supported sucha presentation. Upon initial instruction, the user is
expected to assemble the puzzle pieces to form asentence of subject-verb-object (SVO) type (see Figure.1).

827

Figure 1: Screenshots of thegame screen developed on
ScratchUpon successful completion of a sentence, feedback is generated by the system (see Figure
1 (4)), and the screen is refreshed for the next sentence activity. The scaffolding in the games
was provided using visual elements, such as shapes
and colours. Figure 2 indicates the scaffolds at
different levels, i.e. for level 1, the scaffolds were
provided in terms of shape and colour of the puzzle
pieces, so that words corresponding to a specific partof speech will have different colour and shape.
These elements will gradually fade away (colour
followed by shape) as indicated in Figure 2. Aprogress bar indicates the remaining number ofpuzzles to be solved in each level.
Figure 2: Scaffolds used at different
levels of the game

5. Methodology (User testing)
The usability evaluation of the prototype was conducted with five users in a laboratoryenvironment. Out of these five people, two were children (male) between the ages of 6 - 6.5
years without any hearing impairment, whereas the other three belong to the age group of 18
years and above, with severe hearing impairment to complete deafness. The variation in age
group of the participants is due to our limited access to young DHH children. Users 1 and 2
helped us understand the usability challenges faced by young children while playing the game.
Users 3, 4 and 5 helped us understand the challenges faced by DHH users while playing our
game. We acknowledge this as a limitation to be addressed in future studies. Despite thislimitation, since this was a preliminary study, data from these users helped us refine our gamefor young DHH users.
The two users of 6 and 6.5 years of age received instructions on how to play the puzzle
in both English and Tamil. User 1 had some initial difficulty in positioning the pieces andnavigating the information on the screen, but was able to complete Level 1 successfully. User2, who had prior knowledge of puzzles, was able to quickly solve the first puzzle andsuccessfully completed both the levels with the help of the progress bar. User 2 was able tonavigate between the two levels and figured out the changes in the puzzle shape in Level 2.
User 1 showed interest in moving to Level 2, but was not willing to play with the pieces due tothe puzzle appearance.
The remaining three users with severe
hearing impairment to complete deafness were
given instructions to play the game in written
format, as indicated in Figure 3. They
completed both the game levels with minimal
difficulty; post this study their feedback on thegame was obtained by communicating withgestures and writing.
Figure 3: Interaction with DHH people withsevere hearing impairment to complete
deafness

6. Results and Discussion
Educational games, as visual instruments, play a crucial role in facilitating knowledge
acquisition among DHH students and leveraging a strong visual approach in the creation andutilization of materials can greatly enhance the learning journey of such users. According to
Bouzid et al., 2015, many of the applications developed for DHH students do not possessessential gameplay elements, such as well-defined objectives and scoring systems, which arenecessary for them to be accurately classified as complete games.
The researchers tried to incorporate several gamification elements in this learningintervention and in the present study, the usability of the intervention was examined that828

produced several key takeaways regarding the gamification aspects. A thematic analysis(Braun & Clarke, 2006) of the study revealed the main four themes regarding the perceptionof the usability of the intervention: (1) consistency of images (2) navigation within the game(3) feedback, and (4) question format. The user responses varied for themes (1) and (4) but
shared more similar aspects for themes (2) and (3).
Theme 1: Consistency of images:- Users 1, 4 & 5 were able to map the given image
to corresponding words (visual consistency) but user 2 suggested ensuring consistency in the
mapping of pictures to different forms of verbs. User 3 expressed the need to increase theimage size and use of animated images. User 1 expressed a confused tone while saying ‘Is
the boy eating a cake ?’ User 2 asked for clarification: ‘Is it Rahul? A person is playing chess.
Is it she ?’ The results show that more consistency in the images are required to improve theuser experience of the intervention.
Theme 2: Navigation within the game:- The users could easily navigate between boththe levels of the game. Users also found the progress bar and option to skip the questions
useful. Simplicity in navigation is crucial in any online game and the results showed users
appreciating the basic features incorporated in the intervention.
“The game is interesting” (User 1)“I am going to level 2. The last one is this. I see the dot.” (User 1)“In both the games, I completed all the dots.”(User 2)
Theme 3: Feedback :- Most of the users emphasised the need to improve the reward
system, for example, incorporating animated images as suggested by user 3. Also, most users
suggested incorporating better feedback techniques. The users also noted that the progress
bar was helpful while playing with the intervention. Providing timely and informative feedbackis crucial to keep the motivation high in users and contribute towards learning.
“What is this tick ? Did I make everything correct ? Did I win ?” (User 2)
Theme 4: Question format :- Users 4 & 5 suggested adding a picture and sign for eachletter on the puzzle in addition to exploring other formats including fill-in-the-blanks and matchthe-following (to make the intervention similar to applications like Duolingo). User 3 observedthat the game was appropriate for classes 1-2. Users 4 & 5 suggested that the game wasappropriate for classes up to grade 4. Further, user 1 & 2 could identify the difference inscaffolds used in questions in both levels.
“Oh, what to do ? You have to arrange it (puzzles in level 2). I don’t want to do it.” (User 1)“This is box-box now.” (User 2)

7. Conclusion and Future work
The aim of this study was to understand the usability of a learning intervention designed toaddress the difficulty in sentence construction among the DHH students. The users playedwith the digital prototype and the results of the study provided primary insights on the usability
and design of this game-based early intervention. Though the content delivery of theintervention was found to be appropriate, more elements of gamification including clear andconcise instructions, help functions, and control over gaming options such as speed, anddifficulty need to be incorporated. This is crucial in bringing in engagement, interactivity, andactive participation among the users though there still exists ambiguity in identifying whatfeatures would a learning game need to be both entertaining and pedagogically useful. Further,
the intervention could be used by any novice learner with a motivation to learn the English
language. Future research is proposed to focus on incorporating more features of gamification,and conjugating verbs and study the effectiveness of the intervention in addressing thechallenge to frame sentences involving such forms of verbs.

Acknowledgment
We acknowledge the guidance given by Prof. Chandan Dasgupta and contribution of Shirish
Chinchanikar, Nisumba Soodhani K and Meera Daulatrao Pawar for their inputs during the829

conducted study. We also thank the users who participated in the user-testing on theintervention.

References
Mohanty, E., & Mishra, A. J. (2020). Teachers’ perspectives on the education of deaf and hard ofhearing
students
inIndia:A
study
of
Anushruti.
Alter,
14(2),
85–98.
https://doi.org/10.1016/j.alter.2020.02.002
Mayer, C., & Trezek, B. (2019). Writing and Deafness: State of the Evidence and Implications for
Research and Practice. Education Sciences, 9(3), 185. https://doi.org/10.3390/educsci9030185
Cannon, J. E., & Kirby, S. (2013). Grammar Structures and Deaf and Hard of Hearing Students: A
Review of Past Performance and a Report of New Findings. American Annals of the Deaf, 158(3),292–310. https://doi.org/10.1353/aad.2013.0027
Marschark, M., Spencer, P. E., Adams, J., & Sapere, P. (2011). Evidence-based practice in educating
deaf and hard-of-hearing children: teaching to their cognitive strengths and needs. European
Journal of Special Needs Education, 26(1), 3–16. https://doi.org/10.1080/08856257.2011.543540
Adnyani, N. L. P. S., Wisudariani, N. M. R., Pradnyana, G. A., Pradnyana, I. M. A., & Suwastini, N. K.
A. (2021). Multimedia English Learning Materials for Deaf or Hard of Hearing (DHH) Children.
Journal of Education Technology, 5(4), 571. https://doi.org/10.23887/jet.v5i4.38829
Chan, G. L., Santally, M. I., & Whitehead, J. (2022). Gamification as technology enabler in SEN andDHH
education.
Education
andInformation
Technologies,
27(7),
9031–9064.
https://doi.org/10.1007/s10639-022-10984-y
Faiella, F., & Ricciardi, M.E. (2015). Gamification and learning: a review of issues and research. Journalof e-learning and knowledge society, 11, 13-21.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Bouzid, Y., Khenissi M. A.. and Jemni M (2015), "Designing a game generator as an educational
technology for the deaf learners," 5th International Conference on Information & Communication
Technology and Accessibility (ICTA), Marrakech, Morocco, 2015, pp. 1-6, doi:
10.1109/ICTA.2015.7426914.

830

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Using T-Robot Board Games toEnhance Learning Gains of Rural
Elementary School Students
Yu-Wei WU a*, Wen-Yu YE b*, Wen-Chi Vivian WU c*& Rong-Jyue WANG d*
Department of Foreign Languages and Literature, Asia University, Taiwan
yuweii530@gmail.comb
Department of Foreign Languages and Literature, Asia University, Taiwan
wunyu89@gmail.com
cDepartment of Foreign Languages and Literature, Asia University, Taiwan
vivwu123@asia.edu.twdDepartment of Electronic Engineering, National Formosa University, Taiwan
orffwang@nfu.edu.twaAbstract: This study used T-Robot board games to create an interactive and enjoyable
English language learning environment for the purposes of reducing learners’
anxiety while also enhancing their speaking skills. The participants of this study were
28 fourth-grade students with limited English proficiency from a rural area of
Taichung City, Taiwan, and multiple data sources using mixed methods were also
adopted for data analysis, including pre-and post-tests, questionnaires, andinterviews. Quantitative results showed significant improvements in both speaking
ability and reduced anxiety, and positive qualitative feedback was also displayed
from the interviews. The integration of robotic board games provided a promisingalternative to traditional teaching methods, particularly for lower-proficiency students.
Keywords: Robot board game, Speaking anxiety, Enjoyment

1. Introduction
In today's globalized world, English proficiency is crucial for success. However, the ruralarea of Taichung City faces a challenge with low English proficiency among elementary
students, particularly in speaking, leading to heightened anxiety in communication. Ho(2010) highlighted how traditional teaching methods and reliance on commercial materials
hinder cognitive development and critical thinking skills. To overcome this, an innovative
approach incorporating board games and robots into the curriculum has been explored.
Notably, Huang Qianyi's 2018 study introduced board games tailored for elementary
students to enhance English conversation skills. The combination of these games with theengaging T-Robot aimed to improve learning enjoyment and reduce anxiety. By offering aninteractive and appealing English learning experience, the goal was to elevate proficiency
and foster a greater sense of enjoyment in the learning process. To investigate theeffectiveness of the proposed teaching method, the following three research questions
guided this study:
1. To what degree did robot board games alleviate elementary school students' oral
communication anxiety in English language learning?
2. To what degree did the integration of Robot board games improve students' Englishspeaking proficiency?
3. How did students perceive the use of Robot board games in enhancing their enjoymentof speaking English?

831

2. Research methods
The study utilized a robot board game as a teaching tool to integrate English language
learning for 29 fourth-grade students in the rural area of Taichung City. The research
aimedto evaluate the learning effect and feedback of students after a 10-weekexperiment, consisting of two 40-minute classes per week. In the first week, students
were administereda pre-test of English-speaking proficiency and English ability, alongwith a pre-questionnaireon anxiety. Throughout the teaching process, illustrated in Fig. 1,
the teacher began by teaching the necessary vocabulary for the students to apply in theirwork. Next, students were instructed to work collaboratively in groups to create a school
map, incorporating places, treasures, monsters, and a route, while simultaneously
practicing the oral speaking part for their final presentation, which would take place
during weeks 2-7. During the final week, students completed a post-test of Englishspeaking proficiency and ability, as well as a post-questionnaire. The study used pretest, post-test, and questionnaires as assessmenttools, including the modified test andquestionnaire which was adapted from Luo, K. M. (2019). The speaking test comprised
three parts: cloze, multiple choice, and oral tests, and the modified questionnaires
included 10 questions items. In addition to quantitative analysis,the study also conducted
qualitative one on one interviews with 12 students and classroom observations to gain
an in-depth understanding of this innovative instruction. The interview protocol asked twoquestions about students' enjoyment of the course and their feelings about the robot.
The study sought to answer three research questions related to the improvement ofEnglish-speaking anxiety, enjoyment, and English-speaking proficiency score.
Fig. 1 Students collaborating as a team to use coding cards and make T-Robot move

2.1 T-Robot (Assisted by Reading & Rhythm Co., Ltd.)T-Robot, a robotic board game, utilized coding cards for directing the robot's movement. The
game involved students interpreting and executing commands from these cards on a map.
With dimensions of 106x71x71 mm, the robot was designed for easy handling, featuring
wheels for smooth movement. Command cards provided visual symbols and English
explanations for user-friendly interaction.

3. Findings
The study found that incorporating the T-Robot board game into English teachingeffectively reduced speaking anxiety, improved enjoyment, and enhanced speaking abilities.

3.1 To What degree did robot board games alleviate elementary school students'
oral communication anxiety in English language learning?
The study showed significant reduction in oral communication anxiety using T-Robots. A
five-point Likert scale pre-intervention (M = 3.66, SD = 0.744) and post-intervention (M =4.16, SD = 1.023) demonstrated a -0.500 difference. The 95% confidence interval (-0.851 to-0.149) supports this change. A t-value of -2.923, 27 degrees of freedom, and p-value of0.007 confirm the anxiety reduction. T-Robots effectively alleviate language-related anxiety,highlighting technology's role in addressing student challenges in language learning.

832

Table 1Pre- and Post-English Oral Communication Anxiety Questionnaire

3.2 To what degree did the integration of Robot board games improve students'
English-speaking proficiency?
In RQ2, Table 2's analysis for the English-Speaking Proficiency Test revealed a significant
reduction in oral communication anxiety (mean difference: -12.857, pre-test: M = 31.43,
post-test: M = 44.29, SDs provided). The highly significant t-value of -7.586 (27 degrees of
freedom, p = 0.000) reinforces this finding. Improved English-speaking proficiency is
indicated by increased post-test scores and reduced variability. These results guideeducators towards effective strategies for enhancing learners' oral communication skills.
Table 2Pre-test and Post-test Results for English Speaking Proficiency Test

3.3 How did students perceive the use of Robot board games in enhancing theirenjoyment of speaking English?
RQ3 interviews revealed students' enjoyment of robot board games due to their engagement
and interactivity. They valued the challenges and teamwork, with one student expressing
increased enthusiasm for English class. These findings highlight the games' potential toenhance English learning enjoyment, encourage participation, and improve proficiency,
particularly for students with lower language skills.

4. Conclusion
Incorporating innovative teaching methods like using games in the English curriculum has
the potential to improve learning outcomes, especially for less proficient students. This
study's results supported the effectiveness of the Robot board game method in enhancing
English proficiency, reducing anxiety, and increasing learning enjoyment. Therefore,
integrating this method can create a more engaging and effective language learning
experience, ultimately benefiting students' academic and professional achievements.

References
Huang, C. Y. (2018). A Discussion on Incorporating Games into English Festival Activities.
Taiwan Education Review Monthly, 7(7), 87-89.
Lu, K. M. (2019). Exploring the factors causing oral English anxiety: A case study of college
students majoring in Applied English in Taiwan. Institute of Applied English Studies,
Kainan University.
Ho, C. C. (2010). Review of teaching and learning of science and mathematics in schools.
Academy of Sciences Malaysia.

833

PTP

834

How teachers’ conceptions of student
engagement influenced their actual
strategy implementation, and student
online engagementa, bGulipari MAIMAITIa & Khe Foon HEWb*
Faculty of Education, The University of Hong Kong
Hong Kong SAR, China* kfhew@hku.hk

Abstract: In this study, we explored the interplay between teachers' conceptions ofstudent engagement, their strategy implementation, and student engagement in avideoconferencing-based fully online learning environment. The mixed-method multicase study involved two postgraduate courses, with data collected via surveys andsemi-structured interviews. Findings illustrated diverse conceptions of student
engagement among instructors, influencing their teaching strategies and ultimately
affecting student engagement. Instructors with a comprehensive, multidimensional
understanding of engagement encompassing behavioral, emotional, and cognitive
aspects were more likely to employ effective strategies, resulting in improved student
engagement.
Keywords: Student engagement, teacher’s conception, teacher’s knowledge,
engagement strategy, synchronous online learning

1. Introduction
The COVID-19 pandemic triggered a transition to online learning, utilizing
videoconferencing for synchronous learning. However, research points to low student
engagement in this setting (Maimaiti et al., 2021). Engagement, a key factor in learning
outcomes and positive behaviors (Martin & Bolliger, 2018), is greatly influenced by the
instructor (Farrell & Brunton, 2020). Yet, discrepancies often exist between teachers'
conceptions and practices (Ndeke & Keraro, 2021). This study investigates how teachers'
engagement conceptions influence their strategies and its subsequent effect on student
online engagement. It seeks to answer: (1) How do instructors conceptualize student
engagement? (2) How do their conceptions affect their implementation of engagement
strategies? (3) How does their strategy implementation affect student engagement?

2. Method
This study, using a multi-case approach, quantitative and qualitative methods,
explored two postgraduate courses at a Hong Kong university that had moved to online
learning due to the pandemic. Participants, selected through convenience sampling,
provided data via interviews and a survey. We conducted online interviews with 24 students(12 students from each course). Each interview took about 35 minutes. Pedler et al.'s (2020)
engagement strategies framework (as shown in Figure 1) is used to guide question design.
This framework offers instructors guidance for fostering behavioral, emotional, and cognitive
engagement. Our research focused on these strategies within teaching practices,
addressing gaps in previous studies that either ignored engagement's multidimensional
aspects (e.g.,Nafukho & Chakraborty, 2014) or concentrated on student strategy use (e.g.,
Redmond et al., 2018). Along with the interview, we also used a self-reported engagement835

questionnaire adapted from Skinner et al. (2008) and Rotgans and Schmidt (2011). This 14item questionnaire, validated with Cronbach's alpha values over 0.8 (Lo, 2017), gathereddata on behavioral, emotional, and cognitive engagement. We received 47 valid responses,
with internal consistency of 0.89.
The interview data was analyzed using deductive content analysis with Pedler et al.'s
framework as the coding matrix, and the survey data was descriptively analyzed and tested
for significant differences between the two courses.

Figure 1. A framework of guidelines for teachers to promote student engagement(Pedler et al., 2020, p. 55)

3. Findings
Instructors from two courses held distinct conceptions of student engagement.
Course One's instructor focused on learning experience design and alignment of activities
with learning outcomes, viewing engagement as students mentally navigating through thelearning process. Conversely, Course Two's instructors conceptualized engagement more
specifically in terms of behavioral, emotional, and cognitive aspects, underlining theimportance of active participation, enjoyment, and content understanding.
Course Two implemented all engagement strategies outlined by Pedler et al. (2020),
while Course One's implementation was more selective. Course One's instructor used a rigid
course outline, limiting opportunities for behavioral engagement. Students desired moreinteractive activities. Similarly, emotional engagement strategies were deficient in Course
One due to the instructor's lack of emphasis on interaction, making students feel distant anddisengaged. In contrast, Course Two fostered positive teacher-student relationships bycreating interaction opportunities. Cognitive engagement strategies were partially
implemented in Course One, with a focus on self-directed learning but limited measures toensure student understanding and application of the content and promote student interest.
Course Two, on the other hand, utilized all cognitive engagement strategies, employing
diverse resources and activities to make lessons interesting, interactive, and practical.
Quantitative survey data of a T-test as showed in Table 1, indicated that students in
Course Two exhibited significantly higher levels of behavioral, emotional, and cognitive
engagement compared to students in Course One, which aligned with the interview findings.
Table 1. T-test result of students’ behavioural, emotional, and cognitive engagement836

Engagement
dimension
Behavioral
engagement
Emotional
engagement
Cognitive
engagement

CourseN

Mean

Course One
Course Two
Course One
Course Two
Course One
Course Two

25
22
25
22
25
22

3.592
4.264
3.39204.03643.04003.6136t-testp-value-4.323<.001-4.211<.001-3.832<.001

4. Accessibility
This research investigated how instructors' engagement conception affect theirstrategy implementation and consequently, online student engagement. Instructors' uniqueconception, perhaps influenced by their research interests and other elements, significantly
affected strategy effectiveness. The instructor of Course One focused on behavioral
engagement strategies, leading to lower student involvement due to limited interaction anddisregard for student interests. However, Course Two's instructors, who employed a broader
engagement conception, utilized diverse strategies, enhancing student engagement. They
fostered behavioral, emotional, and cognitive engagement via group activities, personalized
communication, and practical application. The study highlights that instructors' engagement
understanding informs their teaching strategies and affects student engagement online. It
implies a comprehensive engagement conception can direct effective strategy execution.
Future research could delve deeper into factors influencing instructors' conceptions ofstudent engagement and involve larger participant sample size.

References
Farrell, O., & Brunton, J. (2020). A balancing act: A window into online student engagement
experiences. International Journal of Educational Technology in Higher Education, 17(1), 25.
https://doi.org/10.1186/s41239-020-00199-x

Lo, C. K. (2017). Examining the flipped classroom through action research. Mathematics
Teacher, 110(8), 624–627. https://doi.org/10.5951/mathteacher.110.8.0624
Maimaiti, G., Jia, C., & Hew, K. F. (2021). Student disengagement in web-based videoconferencing
supported online learning: An activity theory perspective. Interactive Learning Environments, 1–
20.https://doi.org/10.1080/10494820.2021.1984949
Martin, F. & Bolliger, D.U. (2018). Engagement matters: Student perceptions on the importance ofengagement strategies in the online learning environment. Online Learning 22(1), 205–222.
https://doi.org/10.24059/olj.v22i1.1092
Nafukho, F., & Chakraborty, M. (2014). Strengthening student engagement: What do students want inonline courses? European Journal of Training and Development, 38, 782–802.
https://doi.org/10.1108/EJTD-11-2013-0123
Ndeke, G. C., & Keraro, F. N. (2021). Teachers’ perceptions of classroom practices that inculcatescientific creativity. Asian Journal of Social Sciences & Humanities, 9(2), 27–42.
Pedler, M., Yeigh, T., & Hudson, S. (2020). The Teachers’ Role in Student Engagement: A Review.
Australian Journal of Teacher Education, 45(3), 48–62.
https://doi.org/10.14221/ajte.2020v45n3.4
Skinner, E., Furrer, C., Marchand, G., & Kindermann, T. (2008). Engagement and disaffection in theclassroom: Part of a larger motivational dynamic? Journal of Educational Psychology, 100(4),765–781.
Redmond, P., Heffernan, A., Abawi, L., Brown, A., & Henderson, R. (2018). An Online Engagement
FrameworkforHigher
Education.
OnlineLearning,
22(1),
Article1.
https://doi.org/10.24059/olj.v22i1.1175
Rotgans, J. I. & Schmidt, H. G. (2011). Cognitive engagement in the problem-based learningclassroom. Advances in health sciences education, 16(4), 465–479.
https://doi.org/10.1007/s10459-011-9272-9

837" !   !" " "   #" !
#" !"$  #" ! #"  "  
!   
   #
 "#     %!  '#$%*! ! !  ! !   %!   '#$%*! ! !  ! !   %!   ! ! %#!"!%  '#$%* ! !   FHAALM;BA:6BAA86G ;>H ;>  CD85EC5?681D'*9>54E31D9?>253?=5C=?B5@B5F1<5>D9D9C9=@?BD1>DD? 5H1=9>5 D85 @?D5>D91< 9=@13D ?> D51385BC B?<5C 1>4 945>D9DI *89C CDE4I 19=54 D?
9>F5CD971D5 8?G 81D'* =1I C81@5 D51385BC B?<5C 1>4 945>D9DI 2I 3?>4E3D9>7 1
D85=1D931>1<IC9C?63?>D5>DC6B?=-581D@?CDCD81D49C3ECCD85EC5?681D'*2I
D51385BC*851>1<IC9C6?3EC54?>945>D96I9>7D85=5CB5<1D54D?D859=@13D?681D'*?> D51385BC @5B359F54 C5<6 @5417?7931< B?<5C 1>4 B5<1D9?>C89@C G9D8 CDE45>DC *8569>49>7C CE775CD D81D 81D'* =1I 81F5 2?D8 @?C9D9F5 1>4 >571D9F5 9=@13DC ?>
D51385BCB?<5C1>4945>D9DI&>D85?>581>481D'*31>2B9>7>5G@?CC929<9D95CD?
D51385BC1<<?G9>7D85=D?6?3EC=?B5?>CDE45>DCAE1<9D95C1>4C?6DC;9<<C45F5<?@=5>D
1>4 @5BC?>1<9C5 6554213; 1>4 1CC5CC=5>D 6?B CDE45>DC G8938 3?E<4 B54E35 D85
G?B;<?14?6D51385BC1>45>81>35D859B56653D9F5>5CC&>D85?D85B81>4D85EC5?6
81D'* =1I D8B51D5> D85 @B?65CC9?>1< 945>D9DI ?6 D51385BC <5149>7 D? 655<9>7C ?6
9>145AE13I ?B 1 <?CC ?6 3?>DB?< ?F5B D85 D51389>7 @B?35CC EBD85B=?B5 D85 EC5 ?6
81D'*=1I79F5B9C5D?5D8931<9CCE5C9>D51389>71>4<514D?1DBECD3B9C9C25DG55>
D51385BC1>4CDE45>DC*89CCDE4I8978<978DCD85>5546?B6EBD85BB5C51B38?>D859=@13D?681D'*1>4@B?F945C9>C978DC9>D?D859=@<5=5>D1D9?>?681D'*9>54E31D9?>  81D'* BD969391< 9>D5<<975>35   C?391< =5491 D51385B 945>D9DI
D51389>7B?<5CD85=1D931>1<IC9C 

,8F84E6;BA4EG<9<6<4?<AG8??<:8A68#<FABGA8J5HG586B@8FA8J?L4A7HACE86878AG87?L
94F;<BA45?87H8GBG;88@8E:8A68B9;4G!*.!8A8E4G<I8*E8 GE4<A87.E4AF9BE@8E 4
CBJ8E9H?A4GHE4??4A:H4:8CEB68FF<A:@B78?78I8?BC875L)C8A#G;4G64A:8A8E4G8
;H@4A ?<>8G8KG54F87BA4:<I8ACEB@CG4A764A58GE4<A874F46BAI8EF4G<BA4?4:8AG
E8FCBA7<A:GB4J<78I4E<8GLB96B@@4A7F9EB@8A7HF8EF($4?<?8G4?   #A4E8@4E>45?LF;BEGG<@89B??BJ<A:<GFE8?84F8 ;4G!*.;4FFCHEE87J<78FCE8477<F6BHEF8F46EBFF4
7<I8EF8E4A:8B9FB6<8G4?FC;8E8F -;4=<!8BE:88G4?  <FJ4F  -HE4@88EL
-;4>BE   4@BA:J;<6;G;887H64G<BA4?F86GBE<FBA84E84J;8E8;4G!*.;4FG;8
CBG8AG<4?GB@4>84F<:A<9<64AG<@C46G '4AL87H64GBEF;4I86BAI8L878AG;HF<4F@45BHGG;8
CBG8AG<4?9BEG;8F8GBB?FGB<AGEB7H68<AABI4G<I8C874:B:<64?BCCBEGHA<G<8F(J4?<?8G4?  ).
)G;8EF ;BJ8I8E 4E86BA68EA8745BHGG;8CBG8AG<4?B9G86;AB?B:LGB78;H@4A<F8G;8
?84EA<A:8KC8E<8A687E<I8A5L984EB96B@@8E6<4?<F4G<BABEA8B ?<58E4?<78B?B:<8FJE4CC87HC
<AG86;AB?B:LBK   
1<G;4EG<9<6<4?<AG8??<:8A684A7EB5BGFCEBI<7<A:6HFGB@<F87<AFGEH6G<BAGBFGH78AGF 
G846;8EFREB?8F@4LF;<9GGBJ4E7BI8EF88EFJ;B78F<:A4A7F8?86G@46;<A8 ?87<AFGEH6G<BA @BA<GBEFGH78AGCEB:E8FF 4A7CEBI<78FHCCBEG7J4E7F8G4?   .846;8EFC?4L4
6E<G<64?EB?8<AF;4C<A:G;8?84EA<A:8KC8E<8A68FB9FGH78AGF 4A7G;8<ECEB98FF<BA4?<78AG<GL<F
4A<@CBEG4AG4FC86GB9G;8<EJBE>4FFH6; E8F84E6;BAG846;8EFRCEB98FF<BA4?<78AG<GL
9BE@4G<BA6BAGE<5HG8FGBBHEHA78EFG4A7<A:B9J;4G<G988?F?<>8GB584G846;8E<AG;89468B9
E4C<7?L47I4A6<A:G86;AB?B:<8F4A7;BJG846;8EFE8FCBA7GBG;8F86;4A:8F8<=44E78G4?    EB@G;<FCB<AGB9I<8J <G<F6EH6<4?GBHA78EFG4A7;BJG846;8EF6BAF<78EG;8<E<78AG<GL
<AG;8G846;<A:CEB98FF<BA 1;4G<F9BHA7E8?8I4AGGBG;8CEB98FF<BA 8FC86<4??LJ<G;G;8

838

<A6E84F<A:<AGEB7H6G<BAB94EG<9<6<4?<AG8??<:8A68 <AGB87H64G<BA @4L6?4F;J<G;4G846;8EF
I4?H8F4A758?<89F45BHGJ;4G6BAFG<GHG8F89986G<I8G846;<A:CE46G<68F 8<=44E78G4?     
  
.846;8E<78AG<GL ;4F588AI4E<BHF?L789<A87<AG;8?<G8E4GHE8 8<=44E78G4?   !88
789<A87 G846;8E<78AG<GL4F4 68EG4<AO><A7B9C8EFBAPBEOF8I8E4?7<998E8AG><A7FP46G<A:
4A7<AG8E46G<A: <A4:<I8A6BAG8KGR !88 
C  "BJ8I8E "B99@4A %<CC  58?<8I8
G;4G G846;8E<78AG<GL<F ABG49<K87CB<AGG;4G<A7<I<7H4?F<AG8E46GJ<G;G;8JBE?7 5HG QG;8<AG8EF86G<BAB9C8EFBA4? C874:B:<64? 4A7CB?<G<64?C4EG<6<C4G<BA4A7E89?86G<BAJ<G;<A4?4E:8E
FB6<BCB?<G<64?6BAG8KGRC  8<=44E78G4?   1;<?8G;8E8<FAB6BAF8AFHFE8:4E7<A:
J;4G6BAFG<GHG8FG846;8E<78AG<GL G;8E8<F4:8A8E4?46>ABJ?87:8@8AGG;4GG846;8E<78AG<GLC?4LF4>8LEB?8<A786<F<BAFG846;8EF@4>845BHGG;8<EG846;<A:CE46G<68F G;86BAG8AGG;8L
G846; G;8><A7B9E8?4G<BAF;<CFG;8L@4<AG4<AJ<G;G;8<EFGH78AGF8<=44E78G4?   4FJ8??4FQJ;8E8G;8LC?468G;8<E899BEG 4A7J;8G;8E4A7;BJG;8LF88>BHGCEB98FF<BA4?
78I8?BC@8AGBCCBEGHA<G<8FR"4@@8EA8FF8G4?   CC N #AG;<F FGH7L G846;8E <78AG<GL<FHA78EFGBB79EB@4FB6<B 6H?GHE4?CB<AGB9I<8J J;<6;
;B?7FG;4GG846;8EF7BABG78I8?BCG;8<E<78AG<GL<A4I46HH@ 5HGE4G;8E<A46BAG8KGG;4G
5E<A:FFB6<4?4A76H?GHE4?9BE68FGB584EHCBAG;4G78I8?BC@8AG"B??4A7&46;<6BGG8  .;86BA68CGB9G846;8E<78AG<GL8A6B@C4FF8FG;E88>8L7<@8AF<BAF C8E68<I87F8?9E898EF
GBG;8C8EFBA4?4A7FH5=86G<I8F8AF8B9F8?9G;4GG846;8EF78I8?BC4FG;8LA4I<:4G8G;8<EEB?8
4F87H64GBEF J;<6;<A6?H78F4G846;8EF58?<89F 4GG<GH78F I4?H8F 4A78@BG<BAF C874:B:<64?EB?8FE8?4G87GBG846;<A:4A7?84EA<A: G;4G<AIB?I8FG;8>ABJ?87:8 F><??F 
4GG<GH78F 4A7I4?H8FG;4G789<A84C4EG<6H?4ECEB98FF<BA4A77<FG<A:H<F;<G9EB@BG;8EF 4A7
G;8<EE8?4G<BAF;<CFJ<G;FGH78AGF 8<=44E78G4?   

'5B395F54C5<6?BB?<51C1
D51385B

 (5<1D9?>C89@C
G9D8CDE45>DC'5417?7931<
945>D9DI&# BA68CGH4?<F<A: G846;8E<78AG<GL  

! 

.;8HF8B9G86;AB?B:L<A87H64G<BA;4FE4<F876BA68EAF45BHG<GFCBG8AG<4?<@C46GBA
G846;8EFEB?8F4A7<78AG<GL  BE8K4@C?8 FB@8FGH7<8F;4I8FH::8FG87G;4GG;8HF8B9
G86;AB?B:L<AG;86?4FFEBB@64A8A;4A68G846;8EFCEB98FF<BA4?<78AG<GL5L8A45?<A:G;8@GB
58@BE8<AABI4G<I8 6E84G<I8 4A789986G<I8<AG;8<EG846;<A:CE46G<68F BA4:;H8   
%B8;?8E'<F;E4  3;BA:8G4?   )G;8EFGH7<8F;4I8;<:;?<:;G87G;8CBG8AG<4?
A8:4G<I8<@C46GB9G86;AB?B:LBAG846;8E<78AG<GL FH6;4F988?<A:FB9<A478DH46LBE4?BFF
B96BAGEB?BI8EG;8?84EA<A:CEB68FF <!E8:BE<B&<FGBA  "<??/E<58 ?BE8M    

839

1;<?8;4G!*.<F4E8?4G<I8?LA8JG86;AB?B:L G;8E8;4I8588AF8I8E4?FGH7<8F8KC?BE<A:<GF
CBG8AG<4?<@C46GBAG846;8E<78AG<GL J<G;@<K87E8FH?GF ;4G!*.64A4FF<FGG846;8EF<A
6B@C?8G<A:G;8<EEB?8F<AG846;<A:CE46G<68F  BE<AFG4A68 G;8# CBJ8E87FLFG8@64A;8?C
G846;8EFCEBI<786HFGB@<M879887546>GB<A7<I<7H4?FGH78AGF F4I<A:G;8@G<@84A7899BEG
&HA714A:   #G64A58HF87GB6E84G8<AG8??<:8AGGHGBEFG;4GCEBI<78C8EFBA4?<F87
:H<74A684A79887546>GBFGH78AGF4FG;8LCEB:E8FFJ<G;G;8<EFGH7<8F%4??4-@<G;  '4E>8?8G4?   8FC<G8G;858A89<GF -;<7<D <A7<64G87G;4GG;8EB?8B9;H@4A
G846;8EF64AABG58E8C?46875L;4G!*.5864HF887H64G<BAE8DH<E8F7<E86G<AG8E46G<BA 
8@BG<BA4?6BAA86G<BA 6E84G<I<GL 4A7G;845<?<GLGB64CGHE8AH4A68F4A7?84EA<A:FGL?8FB9
<A7<I<7H4?FGH78AGF 77<G<BA4??L G;8C874:B:<64?7<@8AF<BAB9G846;8E<78AG<GL@4L58<@C46G875LG;8
CBG8AG<4?9BE;4G!*.GB4HGB@4G868EG4<A4FC86GFB9:E47<A:4A74FF8FF@8AG%4??4
-@<G;    BE<AFG4A68 4FGH7LHF<A:DH8FG<BAF9EB@4FB9GJ4E8G8FG<A:5BB>5L
@@4AA4A7)99HGG<A7<64G87G;4G;4G!*.64ACEBI<786BEE86GBEC4EG<4??L6BEE86G4AFJ8EF
<A B964F8F CEBI<786BEE86GBEC4EG<4??L6BEE86G8KC?4A4G<BAFB94AFJ8EF<A B9
64F8F 4A7G;4GCEB@CG<A:G;8GBB?<A4F;4E87DH8FG<BA6BAG8KG?847FGB4@4E:<A4??L;<:;8E
E4G8B96BEE86GE8FCBAF8F$4?<?8G4?  . "BJ8I8E .46>4A7*<86; ;4I8
78I8?BC874FB?HG<BA9BEG8FG<A:G;845<?<G<8FB94A#G846;8E5LEHAA<A:6BAI8EF4G<BA4?
4:8AGF<AC4E4??8?GB;H@4AG846;8EF4A76B@C4E<A:G;8<EE8FCBAF8F<AG8E@FB9FC84><A: 
HA78EFG4A7<A: 4A7;8?C<A:FGH78AGF .;89<A7<A:F<A7<64G87G;4GJ;<?8#C8E9BE@FJ8??BA
6BAI8EF4G<BA4?HCG4>8 G;8L4E8DH4AG<9<45?LJBEF8G;4AE84?G846;8EFBAF8I8E4?C874:B:<64?
7<@8AF<BAF 8FC86<4??LJ<G;E8:4E7GB;8?C9H?A8FF.46>*<86;   *<6>8??B4> ;4I8;<:;?<:;G87G;4GJ;<?8<AG8:E4G<A:;4G!*.<AGBG846;<A: <AFGEH6GBEFF;BH?7
9B6HFBA6E84G<A:58GG8E4FF<:A@8AGFG;4GE8DH<E86E<G<64?G;<A><A:4A74A4?LF<FE4G;8EG;4A4
@8E8E86B??86G<BAB9946GF4A7G;4GG;8LF;BH?7E8@4<AI<:<?4AG<A78G86G<A:C?4:<4E<F@4A7
6;84G<A: #GJ4F4?FBABG87G;4GG;8<AGEB7H6G<BAB9;4G!*.@4L49986GG;8G846;8EF
E8?4G<BAF;<CJ<G;FGH78AGF ;4G!*.64A58HF87GB78I8?BC<AABI4G<I8CEB=86GF4A7
E8FBHE68F4A76E84G8<AG8E46G<I8:4@8F4A746G<I<G<8FG;4G8A:4:8FGH78AGF@BE8
@84A<A:9H??L%4??4-@<G;   FGH7L5L'4E>8?8G4?  9BHA7G;4G;4G!*.
CEBI<78FG;8BCCBEGHA<GL9BEG846;8EFGB:8GI4?H45?8G846;<A:CE46G<68J<G;BHGG;8CE8FFHE8F
B949986G<A:FGH78AGF 4??BJ<A:G;8@GB<G8E4G8G;8<EE8FCBAF8F5BG;7HE<A:846;F8FF<BA4A7
46EBFFF8FF<BAF ;4G!*.64AF<@H?4G8FB6<4?7LA4@<6F<AFGH78AG G846;8EG846;<A:
F8GG<A:F :<I<A:G846;8EFG;86;4A68GBCE46G<68@87<4G<A:C88E C88E?84EA<A:<AG8E46G<BAF<A
4F498F8GG<A:   

!<I8AG;8A4F68AGA4GHE8B9;4G!*. G;8E8<F6HEE8AG?L4?46>B9FB?<78@C<E<64?FGH7<8F
E8?4G87GB;BJG;8HF8B9;4G!*.@4LF;4C8G846;8EF<78AG<GL .;<FC4C8E4<@FGB9<??G;<F:4C5L6BA7H6G<A:46BAG8AG4A4?LF<FB9J85F<G8FGB<AI8FG<:4G8;BJ;4G!*.@4LF;4C8
G846;8EF<78AG<GL -C86<9<64??L G;8FGH7L8K4@<A87;BJ;4G!*.@4L<@C46GG846;8EF
58?<89F 4GG<GH78F 4A7I4?H8FE8?4G87GBG;8<EEB?84F87H64GBEF 4A7;BJ<G@4L7LA4@<64??L
F;4C8G846;8EFO6B??86G<I8P<78AG<GL .;89B??BJ<A:E8F84E6;DH8FG<BAF:H<787G;8<ADH<EL,+ #AJ;4GJ4LF7B8F;4G!*.<@C46GG;8EB?8FB9G846;8EF,+ "BJ7BG846;8EF<AG8:E4G8;4G!*.<AGBG846;<A:C874:B:<8F,+ "BJ7B8F;4G!*.6;4A:8G846;8EFRE8?4G<BAF;<CFJ<G;FGH78AGF 
 #AG;<FFGH7L J88@C?BL87G;8@4G<64A4?LF<FGB4A4?LF8G;86BAG8AGFB9CBFGFBE4EG<6?8F
CH5?<F;87BAG;818;4GC?4G9BE@ 4;<:;?L<A9?H8AG<4?FB6<4?@87<4C?4G9BE@<A@4<A?4A7

840

;<A4 .;8CBFGFBE4EG<6?8FJ8E868AGE874EBHA7G;8CBG8AG<4?<@C46GB9HF<A:;4G!*.BA
G;8EB?8F4A7<78AG<GLB9G846;8EF .;8@4G<64A4?LF<F<F4DH4?<G4G<I8E8F84E6;@8G;B7G;4G<F
C4EG<6H?4E?LHF89H?9BE4A4?LM<A:G;86BAG8AGB9G8KGH4?74G404<F@BE47< 8G4?   #G<F4
9?8K<5?84A7FLFG8@4G<64CCEB46;G;4G64A58HF87GBCEBI<7846B@CE8;8AF<I84A7AH4A687
HA78EFG4A7<A:B9G;8HA78E?L<A:@84A<A:F4A7C8EFC86G<I8F<AG;874G4E4HA?4E>8   
 

.B6B??86G74G49BEG;<FFGH7L J86BA7H6G874F84E6;B9E8?8I4AGCBFGF4A74EG<6?8FHF<A:4
6B@5<A4G<BAB9>8LJBE7F <A6?H7<A:;4G!*. G846;8E 4A787H64G<BAG;EBH:;BA8
F84E6;8A:<A818;4GC?4G9BE@GB<78AG<9L4EG<6?8FG;4G4E8E8?4G87GBG;8HF8B9;4G!*.5L
G846;8EF E4G;8EG;4A6E4J?<A:4??CH5?<6CBFGFBAG;8HF8B9;4G!*.<A87H64G<BA -C86<9<64??L J8HF87G;89B??BJ<A:F8?86G<BA6E<G8E<4GB@4AH4??L<78AG<9LJ85F<G8F9BE
<A6?HF<BA<AG;8FGH7LCBFGF4A74EG<6?8F7<F6HFFG;8HF8B9;4G!*.<A87H64G<BA4A7<GF
CBG8AG<4?<@C46GBAG846;8EF<78AG<GL CBFGF4A74EG<6?8FCH5?<F;875LE8CHG45?8FBHE68F 
FH6;4F46478@<6<AFG<GHG<BAF 87H64G<BA4?BE:4A<F4G<BAF BE87H64G<BA4?5?B::8EF CBFGF
4A74EG<6?8FG;4G7<F6HFFG;8HF8B9;4G!*.<A87H64G<BA 4A7G;4GCEBI<78<A9BE@4G<BABAG;8CBG8AG<4?<@C46GB9G;<FG86;AB?B:LBAG846;8EF<78AG<GL FBHE68FCH5?<F;87J<G;<AGJB
@BAG;F9EB@ 85EH4EL  GB'4E6;   <ABE78EGB8AFHE8G;4GG;874G4<FHC GB
74G8 4A7G;84EG<6?8FCH5?<F;87BAG;8C?4G9BE@;4I8@BE8G;4A
E847FGB8AFHE8
G;4GG;84EG<6?8FF8?86G879BE4A4?LF<F;4I846;<8I87468EG4<A?8I8?B9CBCH?4E<GLBE
E8478EF;<CBAG;8C?4G9BE@4A74E8E8CE8F8AG4G<I8B9G;8@BFGJ<78?LE8474A7<A9?H8AG<4?
4EG<6?8FBAG;8C?4G9BE@ F4E8FH?G 4EG<6?8F4A7CBFGFJ8E8E8GE<8I87 189HEG;8E
8K6?H7874EG<6?8FG;4G4E8<78AG<64?4A7GBBF;BEG #78AG<64?4EG<6?8F@4L<A7<64G84?46>B9
7<I8EF<GLBEBE<:<A4?<GL<AG;86BAG8AG J;<?8I8ELF;BEG4EG<6?8F@4LABGCEBI<788ABH:;
6BAG8KGBE78G4<?FGB9H??LHA78EFG4A7G;8GBC<64G;4A7 .;818;4GC?4G9BE@FLFG8@
E8DH<E8FBE<:<A4?4EG<6?8FGB;4I84JBE76BHAG:E84G8EG;4A ,8:4E7<A:G;8A4GHE8B9G;8
C?4G9BE@ 4EG<6?8FG;4G7<7ABG@88GG;8JBE7E8DH<E8@8AGJ8E88K6?H787  <A4??L 4EG<6?8F
@8GG;8<A6?HF<BA6E<G8E<449G8E8?<@<A4G<A:7HC?<64G8F GG;8G<@8B9G;89<A4?4A4?LF<F F<K
F<G8FJ8E8789HA6G ?84I<A:4EG<6?8F4A7CBFGF<AG;89<A4?F4@C?8   
#AG;<FFGH7L G;8@4G<64A4?LF<F4?BA:J<G;4787H6G<I84CCEB46;J4FHF879BE4A4?LF<A:G;8
6BAG8AGFB9G;84EG<6?8F .;8787H6G<I84CCEB46;<F4GBC 7BJAG86;A<DH8G;4G<AIB?I8F
FG4EG<A:J<G;46?84E9E4@8JBE>4A7HF<A:<GGB:H<78G;84A4?LF<FB9G;874G4E4HA?4E>8   .;<FFGH7LJ4F:H<7875LG;86BA68CGH4?9E4@8JBE>BAG846;8E<78AG<GL <:HE8 
J;<6;CEBI<78F4AHA78EFG4A7<A:BAG;8<@C46GB9;4G!*.BAG;E887<@8AF<BAFB9G846;8E
<78AG<GL <A6?H7<A:C8E68<I87EB?8F4F4G846;8E G846;8EFRC874:B:<64?<78AG<GL 4FJ8??4F
G846;8EFRE8?4G<BAF;<CFJ<G;FGH78AGF FF;BJA<ACC8A7<K G;84EG<6?8F<A6?H787<AG;8
4A4?LF<F6BI8E87BA8BE@BE87<@8AF<BAFB9G846;8E<78AG<GL .;89<EFG4A7F86BA74HG;BEFB9G;8FGH7L8K4@<A87<784FG;4G4E8E8?8I4AGGBG;8
9E4@8JBE>4A7G;8A<78AG<9<878I<78A68B9G;8F8<784F9EB@G;874G4 1858:4A5L
6BA7H6G<A:4CE8?<@<A4EL4A4?LF<FB9G;874G4GB<78AG<9L6B@@BAG;8@8F4A7C4GG8EAFHA78E
846;7<@8AF<BAB9G846;8E<78AG<GL 18G;8A78I8?BC8746B7<A:F6;8@84A7HF8<GGBFLFG8@4G<64??L6B78G;86BAG8AGFCC8A7<K .;86B7<A:F6;8@8J4F78F<:A87GB64CGHE8
74G4E8?4G87GB;BJ;4G!*.@4L<@C46GG846;8EFEB?8F4A7<78AG<GL9EB@G;E887<@8AF<BAF
< 8 6;4A:8F<AG846;8EFEB?8F 6;4A:8F<AC874:B:L 4A76;4A:8F<AG;8G846;8E FGH78AG
E8?4G<BAF;<C 184A4?LF87G;86B78774G4GB<78AG<9LC4GG8EAFE8?4G87GB;BJ;4G!*.@4L
<@C46GG846;8EF<78AG<GL .B8AFHE8G;8I4?<7<GL4A7E8?<45<?<GLB9G;8FGH7L G;8GJB4HG;BEF
HF874FLFG8@4G<64A7GE4AFC4E8AG4CCEB46;GB74G44A4?LF<F .;84HG;BEF@4<AG4<A874
78G4<?874H7<GGE4<?B9G;88AG<E8E8F84E6;CEB68FF <A6?H7<A:786<F<BAF@4784A76;4A:8F
<@C?8@8AG87 .;874G44A4?LF<F<AIB?I87F8I8E4?EBHA7F #AG;89<EFGEBHA7 G;89<EFG4HG;BE
4A7F86BA74HG;BEE847G;84EG<6?8FGB:4<A46B@CE8;8AF<I8HA78EFG4A7<A:B9G;8@8FBA

841

846;7<@8AF<BAB9G846;8E<78AG<GL #AG;8F86BA7EBHA7 G;8GJB4HG;BEF<A78C8A78AG?L
6B787G;874G44A7>8CG7<4E<8FGBE86BE7G;BH:;GF4A7DH8FG<BAFG;4G8@8E:877HE<A:G;8
6B7<A:CEB68FF .;8GJB4HG;BEFG;8AE8I<8J874A76B@C4E87G;8<EE8FH?GF4A7:EBHC87
G;8@8F .;8<E<AG8E 6B78E4:E88@8AGJ4F  <A7<64G<A:4A8K68??8AG<AG8E 6B78EE8?<45<?<GL
'<?8F"H58E@4A  <F4:E88@8AGFBA6BAG8FG876BAG8AGFJ8E8477E8FF87G;EBH:;
7<F6HFF<BAF58GJ88AGJB4HG;BEFHAG<?4 6BAF8AFHFJ4FE846;87  
 !(* % "%%#!$!%#$.;8DH4?<G4G<I84A4?LF<FB9E8?8I4AGCBFGFBAG;818;4GC?4G9BE@E8I84?87F8I8E4?
FH5G;8@8FE8?4G87GBG;8<@C46GB9;4G!*.BAG846;8EFRC8E68<I87F8?94F4G846;8E )A8
B9G;8@4=BE6BA68EAFJ4FE8?4G87GBJ;8G;8E;4G!*.JBH?7E8C?468G;8EB?8B9G846;8E '4AL58?<8I87G;4GG;878I8?BC@8AGB9;4G!*.J<??CBF84G;E84GGBG;8G846;<A:
B66HC4G<BA-BHE68 H8GBG;845<?<GLB9;4G!*.GB4HGB@4G<64??LC8E9BE@E8C8G<G<I8
G4F>FFH6;4F:E47<A: CEBI<7<A:9887546> 4A76E84G<A:87H64G<BA4?6BAG8AG <G@4L?847GB
FB@8G846;8EF586B@<A:HA8@C?BL87-BHE68 )G;8EFCB<AG87BHGG;4GJ;<?8G846;8EF
64AABG58E8C?46875L#9BEG;8G<@858<A:7H8GBG;87<99<6H?GL9BE#G86;AB?B:LGBE8C?468
;H@4A786<F<BA @4><A:<A6B@C?8KF<GH4G<BAF 8FC86<4??LG;BF8G;4GE8DH<E8G;8<AG8:E4G<BAB9
8@BG<BAF 8KC8E<8A68 4A7F6<8AG<9<6>ABJ?87:8-BHE68   87H64GBEF ;BJ8I8E 
F;BH?758:<AGB6BAF<78E6;4A:<A:G;8<EG846;<A:@8G;B7FJ;8A#<FCEBI<7<A:4A
<A6E84F<A:?LC8EFBA4?<F8787H64G<BA9BEFGH78AGF F#J<??6BAG<AH8GB8IB?I8 G846;8EF4E8
6B@C8??87GBG;<A>B9G;8<EQ<EE8C?46845?8I4?H8R<AG;89HGHE8-BHE68  
77<G<BA4??L G846;8EFREB?84FGE47<G<BA4?>ABJ?87:8GE4AF@<GG8EFJ4FDH8FG<BA87 #A
G;8GE47<G<BA4?G846;<A:@B78? FGH78AGF@4<A?L46DH<E8>ABJ?87:8G;EBH:;G8KG5BB>F4A7
G846;8E?86GHE8F J<G;?<@<G87?84EA<A:@4G8E<4?F4A7<A478DH4G86BAA86G<BA58GJ88A?84EA<A:
46G<I<G<8F4A7E84? ?<98F<GH4G<BAF "BJ8I8E <AG;89HGHE8B987H64G<BA FGH78AGF64AB5G4<A4@BE87<I8EF8E4A:8B9>ABJ?87:8G;EBH:;4EG<9<6<4?<AG8??<:8A68G86;AB?B:<8FFH6;4F;4G!*.-BHE68 .;8FB?<7@4FG8ELB9946GH4?>ABJ?87:8G;EBH:;EBG8@8@BE<M4G<BA<F
6BAF<78E87BA8B9G;847I4AG4:8FB94FGF<4A87H64G<BA "BJ8I8E J<G;G;88@8E:8A68B9
;4G!*. G;8F847I4AG4:8FGHEA87<AGB7<F47I4AG4:8F-BHE68       F4E8FH?G G846;8EFF;BH?7:B58LBA7>ABJ?87:8GE4AF@<FF<BA4A7C4L:E84G8E4GG8AG<BAGBG;8DH4?<GLB9?84EA<A:-BHE68  <EFG?L G846;8EFA887GB8KC?BE84A7GE4AF9BE@G;8
87H64G<BA4?I4?H8B9;H@4A @46;<A87<4?B:H8 6E84G8@BE87<I8EF8?84EA<A:@8G;B7F 
E8G;<A>J;4GGBG846;4A7J;4GGB?84EA 4A7H?G<@4G8?LHF8<A9BE@4G<BAG86;AB?B:LGB
6B@CE8;8AF<I8?L<@CEBI8G;8?84EA<A:B9FGH78AGF-BHE68 -86BA7?L G846;8EFF;BH?7
8@C;4F<M8@BE8BADH4?<GL6H?G<I4G<BA4A7FB9GF><??F78I8?BC@8AG-BHE68  F<GJ4F
<A7<64G87<ABA84EG<6?8 J;4GGEH?L?847FGB4C8EFBAFFH668FF<FG;845<?<GLGB9B6HFBA
?84EA<A: GB8?8I4G89EB@<AG8E8FGFGB4FC<E4G<BAF 4A7GB586B@84HA<DH8<A7<I<7H4?J<G;
7<FG<A6G<I8C8EFBA4?DH4?<G<8F #G<F4?FBE8?<4AGBAE8FCBAF<5<?<GL4A7E8F<?<8A68 4FJ8??4FG;8
45<?<GLGBC8EF8I8E84A7C8EF<FG<AG;89468B97<99<6H?G<8F 77<G<BA4??L <G<F78C8A78AGBA:BB7
<AG8EC8EFBA4?E8?4G<BAF;<CF4A74FC<E<GB96B??45BE4G<BA-BHE68 
HEG;8E@BE8 G;8E8?4G<BAF;<CB9G846;8EFJ<G;#G86;AB?B:<8FJ4F4?FBG;89B6HFB9
@4AL4EG<6?8F4A7CBFGF )A8B9G;8@BFG7<F6HFF87GBC<6F<F<AJ;4GJ4LFJ<??;4G!*.
;8?CE86BAFGEH6GG;8E8?4G<BAF;<C58GJ88AG846;8EF4A7@46;<A8F -B@8I<8J87<G4F4A
BCCBEGHA<GLGBE88I4?H4G8G;8I4?H8B9;H@4AG846;8EF4A7E8F;4C8G846;8E64C45<?<G<8F
-BHE68 )G;8EF58?<8I87G;4GO<AG;8CEB68FFB9G846;<A: ;H@4AG846;8EF4A7<AG8??<:8AG
GBB?FF;BH?7;4I846B??45BE4G<I8E8?4G<BAF;<C A4@8?L;H@4A @46;<A86B??45BE4G<BA "H@4A
G846;8EF4A7<AG8??<:8AG4FF<FG4AGFF;BH?77<I<78G;8<EJBE>E84FBA45?L 6BBC8E4G8 4A7=B<AG?L
6B@C?8G8G846;<A:G4F>F-BHE68  !((!&%#$ %#% %  %!% "!$F<78AG<9<87<AG;8F8?86G874EG<6?8F4A7CBFGF G;8E8J8E8F8I8E4?J4LFB9<AG8:E4G<A:
;4G!*.<AGBG846;<A:C874:B:<8F  <EFG G846;8EF64AHF8;4G!*.GBCEB@BG8

842

C8EFBA4?<F87?84EA<A: ;4G!*.64A;8?CG846;8EF58GG8EHA78EFG4A7FGH78AGF?84EA<A:
F<GH4G<BAF CEBI<78G4E:8G87:H<74A68466BE7<A:GBG;8<EA887F 4A7<@CEBI8G846;<A:DH4?<GL
-BHE68  ;4G!*.64AHA78EFG4A74A7:8A8E4G8A4GHE4??4A:H4:8 CBFF8FFFGEBA:
6BAI8EF4G<BA4?<AG8E46G<BA64C45<?<G<8F 4A7CEBI<78FGH78AGFJ<G;;<:; DH4?<GL4A7
C8EFBA4?<F87?84EA<A:6BAG8AG4A7F8EI<68F :E84G?LGE4AF9BE@<A:G;8?84EA<A:CEB68FF4A7
@8G;B7F  BE8K4@C?8 ;4G!*.64A4A4?LF8FGH78AGF?84EA<A:<AG8E8FGF4A7A887F54F87
BAG;8<E?4A:H4:8<ACHG4A758;4I<BHE :8A8E4G8G846;<A:E8FBHE68F4A7FGE4G8:<8FG;4G
@4G6;G;8<EC8EFBA4?<F87?84EA<A:FGL?8F @4>8?84EA<A:@BE8G4<?BE87GBFGH78AGF 4A7
<@CEBI8FGH78AGF?84EA<A:8AG;HF<4F@4A7<A<G<4G<I8-BHE68 -86BA7 G846;8EF64A
<AG8:E4G8;4G!*.J<G;8K<FG<A:GBB?FGBCEB@BG8@BE8<AG8E46G<I8G846;<A: FG;89HA6G<BAF
B9;4G!*.6BAG<AH8GB8IB?I8 G846;8EF64AHG<?<M8;4G!*.GB:8A8E4G84J<78E4A:8B9
?84EA<A:E8FBHE68F FH6;4FBA?<A8I<78BF :4@8F I<EGH4?E84?<GL 4A7BG;8E9BE@F G;8E85L
<A6E84F<A:?84EA<A:<AG8E8FG4A7@BG<I4G<BA-BHE68  .;<E7 G;8J4LB9CEBI<7<A:
9887546>GBFGH78AGFJBH?758@BE87<I8EF<9<874A7C8EFBA4?<F87-BHE68   .846;8EF64AHF8:8A8E4G<I84A7C8EFBA4?<F879887546> ;4G!*.64A4A4?LF8FGH78AGF
?4A:H4:8<ACHG4A758;4I<BHE 4A7:8A8E4G89887546>G;4G<FG4<?BE87GBG;8<E?84EA<A:FGL?8F
4A7A887F .;<F64A;8?CG846;8EFGBCEBI<78@BE889986G<I89887546>G;4G<FG4E:8G87
FC86<9<64??LGB846;FGH78AGFFGE8A:G;F4A7J84>A8FF8F 4A764AG;8E89BE8<@CEBI8G;8
BI8E4??DH4?<GLB9G846;<A:4A7?84EA<A:-BHE68   HEG;8E@BE8 ;4G!*.;4FG;8
CBG8AG<4?GB8A;4A68BE6;4A:8G;88I4?H4G<BACEB68FF9BEG846;8EF-BHE68     
.E47<G<BA4?87H64G<BA4FF8FF@8AGCE<@4E<?LE8?<8FBA8K4@F6BE8F4A7G846;8E8I4?H4G<BAF 
J;<6;B9G8ABI8E?BB>FGH78AGF46GH4??84EA<A:F<GH4G<BAF4A7C8EFBA4?<F87A887F  HGHE8
87H64G<BA4FF8FF@8AGFJ<??58@BE87<I8EF<9<87 HG<?<F<A:4EG<9<6<4?<AG8??<:8A68G86;AB?B:<8F
FH6;4F;4G!*. GBCEBI<78FGH78AGFJ<G;@BE8B5=86G<I84A76B@CE8;8AF<I88I4?H4G<BAF -GH78AGFJ<??5845?8GB8A:4:8<AG4E:8G87?84EA<A:54F87BAG;84FF8FF@8AGE8FH?GF
-BHE68  BE<A7<I<7H4??84EA8EF BA68G;8LCBFF8FF4Q6B@CBF<G85E4<AR6BAF<FG<A:B9G;8
<AA8E5E4<A4A7QBHG8E5E4<ARB9;4G!*. ;BJ7BG;8GJB5E4<AF6B??45BE4G84A7JBE>
GB:8G;8EB8FG;8FGE8A:G;B9G;8BHG8E5E4<A@4>8G;8<AA8E5E4<AFGEBA:8E BE7B8F<G
F?BJ?L4GEBC;L.;8E89BE8 G846;8EF@HFGE8G;<A>6HEE8AG8I4?H4G<BAF4A778I8?BCA8J
G;<A><A:9BE?84EA<A:8I4?H4G<BAFG;4G4E8<A?<A8J<G;G;8FLA8E:L4A7FL@5<BF<FB9G;8GJB
5E4<AF-BHE68  
8FC<G8G;86E84G<I8<AG8:E4G<BAB9;4G!*.<AGBG846;<A: 8G;<64?<FFH8FB9HF<A:G;8
G86;AB?B:L<AG;8G846;<A:CEB68FFJ8E8@BFG8@C;4F<F87<AG;8F8?86G874EG<6?8F ?G;BH:;
;4G!*.64A5E<A:A8JGBB?F4A7G86;AB?B:<8FGBG846;8EF<AG;89<8?7B987H64G<BA <G<F
8FF8AG<4?GB64E89H??LJ8<:;<GFCEBF4A76BAF4A7G4>84CCEBCE<4G8@84FHE8FGB@<G<:4G84AL
47I8EF889986GF H8GBG;8ABG45?8789<6<8A6<8FB9;4G!*. @4AL4EG<6?8F<A7<64G87G;4GG;8
E<F8B9:8A8E4??4E:8 F64?8@B78?F4A7G;8<EE4C<7C8A8GE4G<BA<AGB87H64G<BA@4L4?FB
8K468E54G88G;<64?4A7F86HE<GLE<F>F-BHE68    #GJ4FFH::8FG87G;4GG;8
<AG8:E4G<BA4A778I8?BC@8AGB94EG<9<6<4?<AG8??<:8A684A787H64G<BA@HFG47;8E8GB4C8BC?8
BE<8AG874CCEB46; 9B6HFBA6H?G<I4G<A:FGH78AGF;<:; ?8I8?G;<A><A: 4A78FG45?<F;4FBHA7
FLFG8@B96BEE8FCBA7<A:8G;<64?ABE@F-BHE68  -B@88G;<64?4A7@BE4?<FFH8FA887GB
58G4>8AF8E<BHF?L4A7477E8FF87<AG;84CC?<64G<BAB9;4G!*.<AG;86?4FFEBB@ #GJ4F
788@87A868FF4ELGBE8:H?4G84A76BAFGE4<AG;8G86;A<64?78F<:A4A74CC?<64G<BA 4FJ8??4F
8A;4A6887H64G<BA4A74J4E8A8FFB9HF8EF498GL4A774G4CE<I46L-BHE68   !(* %  %#$+#%! $"$(%$%& %$.;8HF8B9;4G!*.@4L6;4A:8G;8J4LG846;8EF<AG8E46GJ<G;FGH78AGF-BHE68            @BA:G;8F84EG<6?8F8K4@<A87 FB@84EG<6?8F@8AG<BA87G;4G
4FG846;8EFJ<G;7E4J9EB@G;87B@<A4AGEB?8B9?84EA<A:4A7G846;<A: <GJBH?7E87H68G;8
;<8E4E6;<64?E8?4G<BAF;<C58GJ88AG846;8EF4A7FGH78AGF -B@8CB<AG87BHGG;4GJ;8A
>ABJ?87:8586B@8F@BE884F<?L4668FF<5?8 FB9GF><??F4A7DH4?<G<8F78I8?BC@8AG586B@8
<A6E84F<A:?L<@CBEG4AG 4A7F><??FFH6;4F6B??45BE4G<BA4A76E84G<I<GLJ<??58<A6E84F<A:?L
FGE8FF87 66BE7<A:?L G846;8EFJ<??78F<:A@BE86B??45BE4G<I846G<I<G<8FJ<G;FGH78AGF "BJ8I8E <GJ4FCB<AG87BHGG;4GG;8HF8B9;4G!*.@4L4?FB:<I8E<F8GB4GEHFG
6E<F<F58GJ88AG846;8EF4A7FGH78AGF )A68;4G!*.<F<AGEB7H687<AGBF6;BB?F4A7

843

FGH78AGF58:<AGBHF8<G9BEG;8<EBJACHECBF8F 46B@CBF<G85E4<A<F6E84G87 .;88K<FG8A68
B946B@CBF<G85E4<A64ACBF8468EG4<A46478@<6<AG8:E<GL6E<F<F9BEG846;8EFJ;8A
8I4?H4G<A:FGH78AGF?84EA<A:FG4GHF-BHE68  #ABA8B94EG<6?8F <G78ABG87G;4G6;84G<A:
HF<A:;4G!*.ABGBA?LHA78E@<A8F46478@<6<AG8:E<GL5HG4?FB@4>8F<G7<99<6H?G9BE
G846;8EFGB4FF8FFFGH78AGFGEH8CEB9<6<8A6L J;<6;<FB9AB58A89<GGBG;8FGH78AGF
G;8@F8?I8F #A477<G<BAGB<A6E84F<A:G;8CBFF<5<?<GLB9C?4:<4E<F@ G;86BAG8AG:8A8E4G875L;4G!*.@4L4?FB6BAG4<A5<4F8F BA8 F<787A8FF 4A7<A466HE46<8F .;<FABGBA?L49986GF
?84EA<A:BHG6B@8F5HG4?FBC8EC8GH4G8FFG8E8BGLC8F4A7A8:4G<I8I<8JF 8K468E54G<A:G;8
FCE847B9@<F<A9BE@4G<BA )I8EE8?<4A68BA4EG<9<6<4?<AG8??<:8A68@4L4?FB64HF8FGH78AGFGB
@<FFBCCBEGHA<G<8FGB78I8?BC6E<G<64?G;<A><A:4A78A;4A68B99?<A8<AG8EC8EFBA4?
6B@@HA<64G<BAF><??F .;8E89BE8 <G<FABG8AG<E8?LHAE84FBA45?89BEFB@846478@<6<AFG<GHG<BAFGB<@CBF8454ABA;4G!*.-BHE68   

.;89<A7<A:FB9G;<FFGH7LCEBI<78I4?H45?8<AF<:;GF<AGBG;8CBG8AG<4?<@C46GB9;4G!*.BA
G;8EB?8F4A7<78AG<G<8FB9G846;8EF .;8HF8B9;4G!*.<A87H64G<BA6E84G8FBCCBEGHA<G<8F
9BEG846;8EFGB6;4A:8G;8<EGE47<G<BA4?G846;<A:C4E47<:@ 8KC?BE8<AABI4G<I8G846;<A:
C874:B:<8F 4A79BFG8E@BE86B??45BE4G<I84A7<AG8E46G<I8E8?4G<BAF;<CFJ<G;FGH78AGF L4HGB@4G<A:68EG4<A4FC86GFB9G846;<A: ;4G!*.64A;8?CG846;8EFGB9B6HFBA;<:;8E BE78E
G;<A><A:F><??F FH6;4F6E<G<64?G;<A><A: CEB5?8@ FB?I<A: 4A76E84G<I<GL 4A7GBC8EFBA4?<F8
?84EA<A:8KC8E<8A68F9BE<A7<I<7H4?FGH78AGF?:4E9*8G8EF   )AG;8BG;8E;4A7 G;8
HF8B9;4G!*.@4L4?FBE8DH<E8G846;8EFGBF;<9GG;8<EG846;<A:E8FCBAF<5<?<G<8F ?847<A:GB
988?<A:FB9<A478DH46LBE4?BFFB96BAGEB?BI8EG;8G846;<A:CEB68FF .846;8EF@4L988?G;4G
G;8L4E8AB?BA:8EG;8CE<@4ELFBHE68B9>ABJ?87:8BE8KC8EG<F8<AG;86?4FFEBB@ BEG;4G
G;8L4E858<A:E8C?46875L4@46;<A87J4E7F;8B>  -8?JLA  77<G<BA4??L 
G;8<AG8:E4G<BAB9;4G!*.<A87H64G<BAE4<F8F<@CBEG4AG8G;<64?<FFH8FG;4G;4I8ABGL8G
588AF4G<F946GBE<?L477E8FF87 .;8F8<FFH8F<A6?H786BA68EAFBI8E74G4CE<I46L4A7F86HE<GL8EG<AB8G4?   G;8CBG8AG<4?9BE4?:BE<G;@<65<4F4A77<F6E<@<A4G<BA,L4A    4A7
G;8E<F>B9C?4:<4E<F@4A746478@<67<F;BA8FGLBGGBA8G4?   #9G;8F88G;<64?
6BA68EAF4E8ABG477E8FF87 G;8L@4LHA78E@<A8G;8<AG8:E<GL4A789986G<I8A8FFB9
87H64G<BA4?CE46G<68FG;4GE8?LBA;4G!*. 4A75E8874GEHFG6E<F<F58GJ88AFGH78AGF4A7
G846;8EF 1;<?8;4G!*.;4FG;8CBG8AG<4?GBGE4AF9BE@4A7<@CEBI8G846;<A:4A7?84EA<A: <G<F<@CBEG4AGGB477E8FFG;8CBG8AG<4?E<F>F4FFB6<4G87J<G;<GFHF84A7GBCEBI<78G846;8EF
J<G;G;8A868FF4ELFHCCBEG4A7GE4<A<A:GB474CGGBG;86;4A:<A:?4A7F64C8B987H64G<BA !<I8AG;86B@C?8K4A7@H?G<9468G87<@C46GB9;4G!*.BAG;8EB?8F4A7<78AG<G<8FB9
G846;8EF 9HEG;8EE8F84E6;<FA88787GB58GG8EHA78EFG4A7G;8<@C46GB9;4G!*.BAG846;8EF )A87<E86G<BA9BE9HGHE8E8F84E6;<FGB<AI8FG<:4G8G;88KG8AGGBJ;<6;;4G!*.<F
6BAGE<5HG<A:GBG846;8EF<78AG<GL6E<F<F 4A7;BJG;<F6E<F<F64A58@<G<:4G87G;EBH:;GE4<A<A:
4A7FHCCBEG ABG;8E7<E86G<BA<FGB8KC?BE8J4LFGBE85H<?7GEHFG58GJ88AG846;8EF4A7
FGH78AGF<AG;86BAG8KGB9# 4FF<FG87G846;<A: FH6;4FCEBI<7<A::E84G8EGE4AFC4E8A6L<AGB
G;8786<F<BA @4><A:CEB68FFB9;4G!*.4A7CE<BE<G<F<A:;H@4A<AG8E46G<BA4A79887546><A
# 4FF<FG87G846;<A:  <A4??L @BE8E8F84E6;<FA88787GB78I8?BC8G;<64?:H<78?<A8F9BEG;8
HF8B9#<AG;8CEB68FFB9G846;<A: <A6?H7<A:<FFH8FE8?4G87GB74G4CE<I46L 4?:BE<G;@<6
5<4F 4A7G;8CBG8AG<4?9BE#GBC8EC8GH4G8BE8K468E54G88K<FG<A:<A8DH<G<8F<A87H64G<BA .;<FFGH7LJBH?74?FB<A9BE@9HGHE887H64G<BA4?CE46G<68F F<G<F<A8I<G45?89BE;H@4A
58<A:FGB8@5E468A8JG86;AB?B:L G;8FGH7LJBH?7<A9BE@G;878I8?BC@8AGB989986G<I8
FGE4G8:<8F9BEG846;8EFGB<AG8:E4G8G;<FG86;AB?B:L<AGBG;8<EG846;<A: GG;8@46EB?8I8? :BI8EA@8AGF4A746478@<64FFB6<4G<BAFF;BH?7HE:8AG?L78I8?BCE8?8I4AGCB?<6<8F 
E8:H?4G<BAF 4A78G;<64?ABE@F9BE87H64G<BA4?4CC?<64G<BAFB9:8A8E4G<I84EG<9<6<4?<AG8??<:8A68
G86;AB?B:<8F?<>8;4G!*. 4A79BE@H?4G8FC86<9<6 6?84E 4A7984F<5?8CB?<6L
E86B@@8A74G<BAF-6;<99   GG;8@8FB?8I8? F6;BB?FBEHA<I8EF<G<8FA887GB584J4E8
B9G;8<78AG<GL6E<F<F4@BA:G846;8EFJ<G;;4G!*.G4><A:C4EG<AEB?8FB9G846;8EF4A7G;8
GEHFG6E<F<F58GJ88AG846;8EF4A7FGH78AGF7H8GBG;8789<6<8A6<8FB9;4G!*.<A<GF
GE4AFC4E8A6L4A7E8?<45<?<GLIBAF6;8A546;   -C86<9<64??L F6;BB?FBEHA<I8EF<G<8F
F;BH?7<AGEB7H68#78G86G<BAGBB?F9BEG846;8EFGB<78AG<9L4A77<998E8AG<4G858GJ88AFGH78AG

844

JBE>F4A7E8?4G87@4G8E<4?F:8A8E4G875L#4A7G;BF86E84G875LG;8FGH78AGFG;8@F8?I8F
GBCE8I8AG4GEHFG6E<F<F58GJ88AG846;8EF4A7FGH78AGF '84AJ;<?8 G;8LF;BH?7CEBI<78
E8?8I4AGCEB98FF<BA4?78I8?BC@8AGCEB:E4@FBAG;88G;<64?HF8B9;4G!*.GBCE8C4E8
G846;8EFG86;AB?B:<64??L6B@C8G8AG9BE9HGHE8?84EA<A:4A7G846;<A: GG;8@<6EB?8I8? 
G846;8EFA887GB47=HFGG;8<E@<A7F8G FG4L<A:BC8AL8G6E<G<64?GBJ4E7FG;8A8JG86;AB?B:<8F
J;<?8<AG8:E4G<A:G;8@<AGBG;8<E?84EA<A:4A7G846;<A: 8FC<G8G;8I4?H45?8<AF<:;GF9EB@G;<FFGH7L G;89<A7<A:FF;BH?758I<8J87<A?<:;GB9
<GF?<@<G4G<BAF 1;<?8;4G!*.8@8E:8F4F4@<?8FGBA8<AG86;AB?B:<64?78I8?BC@8AG FB@8
6BHAGE<8F FH6;4F;<A4 ;4I854AA87G;8HF8B9;4G!*.<A87H64G<BA .;HFG;874G4
6B??86G879EB@4EG<6?8F4A7CBFGFCH5?<F;87BA4F<A:?8FB6<4?C?4G9BE@@4LABG8AG<E8?L
64CGHE8G;8<@C46GB9;4G!*.BAG846;8EF<ABG;8E6BAG8KGF 'BE8BI8E G;89<A7<A:F54F87
BA74G49EB@BA8FB6<4?@87<4C?4G9BE@@4LABG:8A8E4?<F8GBG;85EB478ECBCH?4G<BAB9
G846;8EF5864HF87<998E8AGC?4G9BE@F@4L;4I87<998E8AG984GHE8F4A7ABE@FG;4G64A49986G
G;86BAG8AG4A7A4GHE8B9G;87<F6BHEF8E8CE8F8AG4G<I8 .;818;4GC?4G9BE@<F?<>8?LGB
4GGE46GFC86<9<6GLC8FB9HF8EFJ<G;C4EG<6H?4E<AG8E8FGF C8EFC86G<I8F 4A758;4I<BHEF .;8E89BE8 9HGHE8FGH7<8F64A<A6?H7874G49EB@4J<78EE4A:8B9FB6<4?@87<4C?4G9BE@F9BE
@BE86B@CE8;8AF<I84A4?LF<F #G<F4?FB47I<F45?8GB<A6?H78CE<@4EL74G49EB@I4E<BHF
6BAG8KGFGB8K4@<A8;BJ;4G!*.JBH?7<@C46GG846;8EF;B?<FG<64??L 
)81:95?B75)?F1>5?B75)12B9?$1BD9>
&/>3/?802+<$(B;69+-<
87'/>/:+5=;37/;;'/-<8:;8DD@C 4?9?B7   0%&& 
59:11B4$59:5B',5B<??@%
(53?>C945B9>7B5C51B38?>D51385BCL@B?65CC9?>1<945>D9DI(/+-2371+7.(/+-2/:.=-+<387   K 
8DD@C 4?9?B7  :D1D5
 5BD9>?"1>D1B39?7<E$;3?B1)1=D1>9)$9DD1<)E@D1$@B9< 6?B
)53EB9DI1>4)53EB9DI6?B  >$:8-//.371;80<2/5/>/7<2!870/:/7-/87+<++7.
9953-+<387'/-=:3<A+7.$:3>+-A@@ 
9CG1C))
'?D5>D91<+C5?681D'*9><?21<-1B=9>777+5;80386/.3-+5
7137//:3718DD@C 4?9?B7  C   
B1E>,<1B;5,+C9>7D85=1D931>1<IC9C9>@CI38?<?7I%=+53<+<3>/:/;/+:-237
9;A-28581A  

B1E>,<1B;5,(2/6+<3-+7+5A;3;=5B931>'CI38?<?7931<CC?391D9?>
?DD?>(?DD?>')89@G1I!(81DD9>71>43851D9>7>CEB9>713145=93
9>D57B9DI9>D855B1?681D'*778>+<387;37.=-+<387+7.(/+-23717</:7+<387+5  
8DD@C 4?9?B7     
?H$H@<?B9>7D859=@13D?6BD969391< >D5<<975>351>4B?2?DC?>89785B54E31D9?>
D8B?E78<9D5B1DEB521C5445C97>693D9?>C7</:7+<387+58=:7+580.=-+<387+5(/-278581A37312/:.=-+<387   8DD@C 4?9?B7   C



1>95<C?<5$-5BDC38!,(2/+6,:3.1/869+7387<8)A18<;4A1=2B9475
+>9F5BC9DI'B5CC
1B<9>71==?>4#B1>C6?B4!
1$:/9+:371</+-2/:;08:+-2+71371?8:5.*2+<</+-2/:;;28=5.5/+:7+7.,/+,5/<8.8!?8>-9<5I)?>C1B<9>71==?>4#B1>C6?B4!
2$:/9+:371</+-2/:;08:+-2+71371?8:5.*2+<</+-2/:;;28=5.5/+:7+7.,/+,5/<8.8!?8>-9<5I)?>C9B57?B9?%#9CD?>
H@5B95>39>7D538>931<496693E<D95C*51385BC5<65669313I1>4
9>CDBE3D9?>1<D538>?<?7I'/50003-+-A377;<:=-<387+5(/-278581A87</@<;
K 
?>178E5
554213;*1<;1C1$51>C?6B51D9>7(1D96I9>71>4%?B=1<9C9>71>
>CD9DED9?>1<<I,1<E54*51385B 45>D9DI8=:7+580 +71=+1/./7<3<A.=-+<387  K
 8DD@C 4?9?B7      
4G1B4C 85?;
-8I>?DB?2?DD51385BC1BD969391<9>D5<<975>356?B144B5CC9>7
D51385BC8?BD1759953/.:<303-3+57</5531/7-/    
8DD@C 4?9?B7       
4G1B4C4G1B4C)@5>35'(#9>.
 D51385B+C9>71BD969391<9>D5<<975>35 
1>4C?391<B?2?DC9>3?==E>931D9?>1>49>CDBE3D9?>866=73-+<387.=-+<387 K
 8DD@C 4?9?B7      
<71B6$'5D5BC
&3D?25BB51D9F5?D1B51D9F5)D?BID5<<5B75>D5F5<?@542I

845

#5F5B179>7'B5DB19>54#1>7E175$?45<C >
&'7</:7+<387+5870/:/7-/87
7</5531/7<&8,8<;+7.'A;</6;&#'@@    8DD@C 4?9?B7  (&) 


55!' 45>D9DI1C1>>1<ID93#5>C6?B(5C51B389>4E31D9?>&/>3/?80&/;/+:-237
.=-+<387 K 8DD@C 4?9?B7  9<<!+B925<?B5J#
+>45BCD1>49>7)53?>41BI)38??<*51385BCL*'"1>4
*538>?<?7I =@<5=5>D1D9?>9>$1D85=1D93C<1CCB??=C7</:7+<387+58=:7+580(/-278581A
37.=-+<387   K 8DD@C 5B93547?F 94!  
1==5B>5CC"1B<9>71==?>4#B1>C6?B4!
?GD51385BC<51B>1>445F5<?@9>#1B<9>71==?>4!B1>C6?B44C$:/9+:371</+-2/:;08:+-2+71371?8:5.?2+<</+-2/:;
;28=5.5/+:7+7.,/+,5/<8.8)1>B1>39C3?!?CC5I1CC K 
?66=1>"9@@'3DE1<9J9>75=?3B13I*85'B1H9C?6*51385B 45>D9DI?>CDBE3D9?>
(/+-2/:.=-+<387%=+:</:5A    K 8DD@C GGG:CD?B?B7 CD12<5 
?<<1>4#13893?DD5!B-,I7?DC;I$5141>4D85>5GC?39?3E<DEB1<CDE495C?6945>D9DI>1>95<C$?<5!,-5BDC384C(2/+6,:3.1/-869+7387<8)A18<;4A@@
K 1=2B94751=2B9475+>9F5BC9DI'B5CC
!1<9<)(169)#1*?J1*$?B1>"#1=-2+<$(+7.'80<?+:/(/;<371
.=-+<387$:863;/;+69$/:35;8DD@C 4?9?B7  (. ,
"1<<1)=9D8%
'<=.A+7.7+5A;3;802+<$(+7.3<;69+-<87300/:/7<3/5.;80
'<=.A
"?58<5B$$9C8B1'
-81D9CD538>?<?7931<@5417?7931<3?>D5>D;>?G<5475*'"87</698:+:A;;=/;37(/-278581A+7.(/+-2/:.=-+<387  K #E>4-1>7*81DD9>712?ED81D'*?G=1I 1>4'*9=@13D13145=91
1>4<92B1B95C 3,:+:A3(/-2"/?;+2/+.809:37<18514?6@B9>D
8DD@C 4?9?B7  #*% $1B;5<!$&@65B=1>)#1>41I!'9538
$(/+-27</:+-<3>/((:+37371
?3<2$(+;/.'<=./7<;
$9<5C$E25B=1>$ %=+53<+<3>/.+<++7+5A;3;7/@9+7./.;8=:-/,884C175
'93;5<<*(?1;(3>/./+;08:8?$:80/;;8:;+7/+5?3<2$( 8:"8?
(I1>$ > G5DBECD5D893C1BD969391<9>D5<<975>351>4B5<9129<9DI'-3/7-/+7.7137//:371
<23-;   8DD@C 4?9?B7  C 

I
)38966
4E31D9?>6?B >?D 6?B4E31D9?>D85B?<5?654E31D9?>1>45D893C9>>1D9?>1< @?<93ICDB1D5795C7</:7+<387+58=:7+580:<303-3+57</5531/7-/37.=-+<387    
8DD@C 4?9?B7  C    
)5<GI>%
'28=5.:8,8<;:/95+-/</+-2/:;+7.<2/0=<=:/80/.=-+<387!?8>-9<5I
)?>C
)8949A$*85EC5?61BD969391<9>D5<<975>3521C5481D'*1>49DC381<<5>75C6?BD85G?B<4?6
54E31D9?>6B?=D85F95G@?9>D?645F5<?@=5>D?63B51D9F5GB9D9>7C;9<<C$:8-//.37180
7</:7+<387+5870/:/7-/87.=-+<387'8-3/<A+7.=6+73<A    K )EB1=55BI%$))81;?B$/+C581D'*D?)?<F5'B?7B1==9>7E7C
7</:7+<387+58=:7+580708:6+<387(/-278581A+7.869=</:7137//:371  K 
8DD@C 4?9?B7   9:9D3   *13;'9538
(2/(/+-2/:(/;<!/+;=:371<2/$/.+1813-+5,353<A805/7./:+7.
$( 37.=-+<387+53+581=/;8DD@C 4?9?B7  (. ,  *?=<9>C?>$!13;C?>
'B?65CC9?>1<945>D9DI6?B=1D9?>9>3?>D5=@?B1BI89785B
54E31D9?>CDE45>DC'<=.3/;37312/:.=-+<387 K 
8DD@C 4?9?B7     ,19C=?B149$!?>5C!*EBE>5>)>5<7B?F5)
*85=545F5<?@=5>D9>AE1<9D1D9F5
3?>D5>D1>1<IC9C1>4D85=1D931>1<IC9C
F?>C385>2138-!
*B1>C@1B5>3I1>4D852<13;2?H@B?2<5=-8IG54?>?DDBECD $2358;892A(/-278581A     8DD@C 4?9?B7 C  
08?>7/E?")E!8E)"-*859=@13D?65C@?BDC@1BD939@1D9?>?>D85
45F5<?@=5>D?6 CD35>DEBIC;9<<C9>I?ED8CICD5=1D93B5F95G869=</:;.=-+<387   8DD@C 4?9?B7  :3?=@54E
  









846

! %$!&#$%% !"$$%# $! $!%# %%*
(B 
*BFGF EG<6?8F
*8E68<I87F8?94F4 *874:B:<64?
,8?4G<BAF;<C
G846;8E
<78AG<GL
J<G;FGH78AGF
-BHE68
2-BHE68
2-BHE68
2
2


-BHE68
2
2
2

-BHE68

2
2

-BHE68
2
2
2

-BHE68
2
2
2

-BHE682

-BHE68

2
2
 
-BHE68 
2
2
2

-BHE68
2
2
2

-BHE68
2
2


-BHE68
2
2


-BHE68
2
2


-BHE682
2
-BHE682
2
-BHE68

2


-BHE68
2
2


-BHE68
2
2
2
 
-BHE68 
2
2
2

-BHE68
2
2
2

-BHE68
2
2


-BHE68
2
2
2

-BHE68
2
2


-BHE68
2
2


-BHE68
2
2
2





























847

!
! $!#)  %"%! % ! %# %%*

<@8AF<BAFB9<78AG<GL -H5G;8@8F
*8E68<I87F8?9BEEB?8
4F4G846;8E





*874:B:<64?<78AG<GL








,8?4G<BAF;<CFJ<G;
FGH78AGF

BAG8AGFBEDHBG8F9EB@FBHEF8F8K4@C?8F
8 : 1;<?8G846;8EF64AABG58
GG<GH78FGBJ4E7F
E8C?46875L#9BEG;8G<@858<A:
G846;<A:B66HC4G<BA
7H8GBG;87<99<6H?GL9BE#.E47<G<BA4?>ABJ?87:8
G86;AB?B:LGBE8C?468;H@4A
GE4AF@<GG8E
786<F<BA @4><A:<A6B@C?8K
.4><A:BAA8JEB?8F
F<GH4G<BAF-BHE68  
"H@4AG846;8E
%%%&$%!(#$% 64C45<?<G<8F!&"%! 
*8EFBA4?<F87?84EA<A:

#AG8E46G<I8?84EA<A:
8 : FG;89HA6G<BAFB9;4G!*. 887546>
6BAG<AH8FGB8IB?I8 G846;8EF64A
I4?H4G<BA4A7
HG<?<F8;4G!*.GB:8A8E4G84J<78
4FF8FF@8AG
E4A:8B9?84EA<A:E8FBHE68F FH6;
G;<6F<AG86;AB?B:<64?
4FBA?<A8I<78BF :4@8F I<EGH4?
C874:B:L
E84?<GL 4A7BG;8E9BE@F G;8E85L
<A6E84F<A:?84EA<A:<AG8E8FG4A7
@BG<I4G<BA -BHE68  %#%'#   ?4GBE78@B6E4G<6
E8?4G<BAF;<CF
8 : )A68;4G!*.<F<AGEB7H687
.EHFG6E<F<F
,87H6876B@@HA<64G<BA <AGBF6;BB?F4A7FGH78AGF58:<AGB
HF8<G9BEG;8<EBJACHECBF8F 4
6B@CBF<G85E4<A<F6E84G87 .;8
8K<FG8A68B946B@CBF<G85E4<A64A
CBF8468EG4<A46478@<6<AG8:E<GL
6E<F<F9BEG846;8EFJ;8A8I4?H4G<A:
FGH78AGF?84EA<A:FG4GHF -BHE68 #&$%#$$




848

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Designing Faculty Development
Programs by a Team fromDifferent Majors
Yukari KATOa*, Yukihiko YAMASHITAa, Hisashi HATAKEYAMAa & Toshiya
OISHIb aCenter for Innovative Teaching and Learning, Tokyo Institute ofTechnology, Japan bScience Academy, Chiba University of Commerce, Japan*kathy@citl.titech.ac.jp
Abstract: This study investigated how three faculty members and an external lecturer
selected trainingcontent and implemented teaching methods for faculty development(FD) to ensure learning opportunities for faculty members at science universities.
Qualitative analysis revealed the following three points: (1) the lack of a clear imageof the target participants causes discrepancies between participant needs and the
FD seminar planners; (2) the faculty developer (FDer) is apt to “chase after two
hares,” when seeking student-centered teaching style and instructors’ language
skills improvement; and (3) FD seminars should be focused on a specific topic to be
more attractive to the target participants.
Keywords: Professional Development, English Medium Instruction, Faculty
Developer, Steps for Coding and Theorization

1. Introduction
Internationalization has been vigorously promoted in Japanese higher education throughgovernment-led policies such as the Global 30 Project and the Top Global University
Project (MEXT, 2014). Japan’s Ministry of Education, Culture, Sports, Science andTechnology (MEXT) has been providing support to improve the international
competitiveness of Japanese higher education through its “Super Global University(SGU)” program, which focuses on 13 top universities (Type A) that conduct world-classeducation and research, and 24 globalization-driving universities (Type B) have beenleading the globalization of Japanese society since 2014. The selected universities haveprovided their students with globalization-oriented education and research by fostering
students’ global responsiveness and further accelerating exchanges and collaborations
with world-class universities. SGU-adopted schools; however, they must meet the goals
established in “common outcome indicators,” such as international students as apercentage of all students, the rate of courses taught in foreign languages, a flexible
academic calendar (introduction of the quarter system), and various organizational andcurricular reforms. To achieve these indicators and demonstrate their internationalization
initiatives worldwide, top universities held workshops in English by invited lecturers fromleading overseas universities and launched joint research projects in cooperation withoverseas partner universities. Furthermore, English-medium instruction (EMI) is
encouraged to attract talented international students and researchers, and both existingand newly hired faculty members are increasingly required to “teach in English.” At theTokyo Institute of Technology, more than 90% of graduate school courses are taught inEnglish to comply with the SGU outcome indicators. The Center for Innovation in849

Teaching Language (CITL), to which the authors belong, has invited lecturers fromoverseas partner universities (University of Queensland) and English-language institutes
in Japan to hold training sessions to promote the use of English in class.

2. FD Seminar with Flipped Learning Material
Preparation for FD Seminar
The theme of the FD training in FY2022 was “Acquisition of knowledge and skills forpractical graduate school lectures in English (EMI).” As shown in Table 1, preparation for
the FD training started in April 2022. Faculty member A was in charge of the FD training
content, creating teaching materials, and conducting the training with an external
Instructor B.
Table 1. FD Training Project Members and Roles
Faculty A
Lecturer B
Faculty C
Faculty D

Role
Planning
FD seminar
LectureSupporting
FD seminar
Supporting
FD seminar

Specialty
Educational
Technology
EnglishTeaching
Informatics

Age, Years at CITL
50s, less than one year

Science
Education

40s, about one year

40s, not applicable60s, more than 30 years

2.1 Team for FD Seminar
Initially, only Faculty Member A and external Lecturer B, who were in charge of the
training, consulted with each other and decided on the FD training content. For example,
the range of sessions on (a) the concept of lesson planning, (b) assignments andhomework, and (c) how to start a class were planned as English lecture basics.
Additionally, two lectures on strategies to promote active learning among students were
given at different language levels to accommodate various learning styles as interactive
sessions. They will also touch on a class environment in which students can easily ask
questions to stimulate interaction among teachers and students and methods for verifying
results (Kato & Oishi, 2022 a; 2022b). To design an effective FD training program topromote specialized education in English, internal meetings were held not only including
Faculty A and external Lecturer B, who are foreign language education specialists but also
other full-time CITL faculty members (C and D) who have experience in specialized
higher education in science. They discussed “what kind of training would be acceptable toscience faculty” and “what kind of content would interest science faculty who are busy
with research, enabling them to participate without creating a burden” with Faculty C, who
has experience teaching specialized fields in English, and Faculty D, who had attended
classes in his university’s doctoral program. Based on the results of this discussion, theperson in charge of planning (Faculty A) developed the FD courses and publicized themwithin the university. Despite multiple publicity campaigns, however, fewer than 10
participants attended. Unfortunately, the project was not of interest to science faculty
members.

2.2 Research Question
This study aims to clarify the decision-making process of planning and operating FD
training programs for EMI courses and develop knowledge for effective planning andimplementation of FD training at science universities with a strong research orientation.
In particular, the authors sought to elucidate how a newly appointed Faculty Developer

850

(FDer, Faculty A) searched for clues to establish FD training tailored to universities withsolid research activities through discussions with other members (Faculty C and D). The
question we have to ask here is what kind of FD training content and implementation
methods are acceptable for faculty at science universities. In addition, this issue
examined building collaborative relationships among faculty members and problemsolving teams, which were discussed in previous studies (Kato et al., 2023a; 2023b).

3. Research Method
3.1 Participants
The subjects of this study were three faculty members at a science university and one
external lecturer. As shown in Table 1, Faculty members A and D were newly transferredfrom other universities. Hence, they needed more information on the roles of faculty
members in research-oriented science education at their home universities and how tomanage graduate classes using English. Faculty A was in charge of this training course,and Faculty C and D belonged to the same organization. Faculty A had approximately ten
years of experience as an FDer, and Faculty D had previous experience as a professor at
the information processing center of another university and has been involved in
educational improvement and evaluation at the Center for educational technology since lastyear. Alternatively, Faculty C is an experienced researcher who transferred from a
specialized department to an education-related center within the university. He has
teaching experience at the undergraduate and graduate levels in his area of expertise andis familiar with the university’s situation.

3.2 Qualitative Analysis Method (Steps for Coding and Theorization)
In this study, the authors attempt to explore and structure the issue of FD training inscience universities using Steps for Coding and Theorization (SCAT), a qualitative
research method, to explore how the “desirable training style” was determined in smallgroup discussions. The authors then conducted a qualitative analysis using SCAT
proposed by Otani (2011) to clarify the process of discussion among the trainers on how
to proceed with “FD training for promoting English language teaching at science
universities.” SCAT is an analytical method providing a roadmap for discovering potential
meanings in language records collected through observation and interviews and fordeveloping new concepts. In addition, qualitative research methods aimed at hypothesis
generation are considered suitable for exploratory investigations to obtain helpful
information to overcome the current situation (Fukushi & Nago, 2011; Kato et al., 2023a;
2023b).

3.3 Data Analysis
The first author transcribed the audio recordings. The transcripts were analyzed using
SCAT, a sequential, thematic qualitative data-analysis technique (Otani, 2008; 2011). It
includes coding steps from open to selective, storyline creation using final selective
codes, and creating theories from the storyline. Accordingly, each utterance of faculty
members was considered to be a single recording unit. These recording units were thenclassified into categories and subcategories based on the similarity of their semantic
content, with different themes extracted following the exercise. This approach wasselected for its explicit analysis process, which integrates qualitative data analysis withtheoretical coding, and for its efficiency and validity of theorization using relatively smallscale data (Otani, 2008). This analysis covered the language records of a December 19,
2022, meeting. The recording time was 53 minutes and 12 seconds. Faculty A was themeeting facilitator, and three faculty members expressed their opinions on “In what typeof FD training course content and implementation would science faculty members be851

interested in participating?” according to the interview guide in Table 2.The verbal
recordings of the meetings were divided into segments based on the SCAT analysis
procedure, resulting in 301 segments. Some segments (8 segments, 28 seconds) were
excluded from the analysis because they included reports of FD training at otherinstitutions attended by Faculty D. There were 293 total segments.
Table 2. Interview Guide
1
2
3
4

How to teach a specialized class in English at a science university.
How to promote active learning and other student-centered classes.
What kind of FD training content and implementation methods are
acceptable for faculty at science universities?
What is the role of an FDer? Did she promote participants' reflections?

4. Results
Qualitative analysis identified the main categories and subcategories in Tables 3 and 4.
Examples of each aspect of the nine main themes are provided below, with thesubcategories and main categories denoted by angle brackets (<…>) and square
brackets ([…]), respectively. In the following paragraphs, double quotation marks (“…”)
denote representative descriptions, and the numbers inside brackets denote unit
numbers for each participant in Table 1. At the preparatory meeting for the FD training on
December 19, 2022, faculty members at CITL expressed their opinions on the “ideal FD
training program” based on their own experiences and the situation of their organizations.
The research question “In what type of FD training course content andimplementation would science faculty members be interested in participating?” prompted
us to aggregate five itemized subcategories into two main categories: [FDer Losing theWay] and [Clear and Simple Course]. As shown in Table 3, the first main category
included the following three subcategories:<Targets are blurred>, <Pursuing two hares>,and <Discrepancies in training content>. The second main category [Clear and Simple
Course] clarified that FD training should be <clear vision of training courses> or deal with
<Simple Skill Training> that focuses on language skills, as shown in Table 4.
Table 3. Subcategories of Problems with FD Training Courses[ FDer Losing the Way]
<Targets are
I had the idea that I should be teaching more about teaching strategies,
blurred>but Instructor B was more interested in discussions, how to teach(language activity), icebreaking, holding a clinic, and so on. B was more
focused on activities, such as how to teach discussion, icebreaking, etc.
(186: A).
If you put two things in, you end up with a different target audience, andIt’s a bit of a blur, and I think it is more for people who are already using
English or are thinking about using it, or know how to use it. (234:D).
(omission) It might be an overstatement to say that this is a bit of a niche,
but we are aiming at that audience, so it’s not a bad direction to take. It’s
not a bad direction to take, and it’s excellent. It's not a bad direction to
take. (270:D).
<Pursuing(They may not go to FD training.) They may only be able to make it to
two hares>
this level if they learn how to communicate well, speak, explain, andgive presentations in English, which may be an overstatement. You
Can’t reach this level unless the participants clear the primary level.
(218:D)

852

(omission) My impression is that it’s trying to have both, and that is why
the ARCS model that I mentioned earlier when it tries to have both,
becomes too much like this, and that is fine if time permits. It would bebetter if I narrowed down the target of the learners and the goal ofwhere the learners will reach so that the number of participants is also
narrowed down, but it would be more apparent. (236:D)<Discrepancies I wonder how it would work in this situation. So, we should have somein training
specific specialized lectures. I do not know. I do not care if it is a video
or something, but he (Lecturer B) does not specialize in that (science)content>
field. (188:C)
I don't think it would be a good fit (to bring lesson plans) to have them
do a mock class and hold a clinic. It might be possible to do what I
mentioned earlier. It will still be okay if it is the kind of group-work
lecture I mentioned earlier. (192:C)
Table 4. Subcategories of Ideal FD Training Courses for a Science University[Clear and Simple Course]<Clear Vision Yes, it would be nice to have a demonstration. So, it would be nice toof Training
have a demonstration. Like in the sciences, we are doing that kind of
Courses>
interaction with math or something like that. (23:C).
One example of a non-native but foreign-style class based on current
trends is a class in which students participate. If we connect this to a
discussion of how this type of class style could be introduced, it would
clarify what the teacher learns. Since it is pretty clear what the teachers
will gain from this FD training, even considering the time constraints, itwould be better to focus on such a specific topic and not make it too
much. (121:D)<Simple andFocusing
Training>

That is not the point; that is not possible. I am sorry. Let us give up on that,but we do have group work, so I think we should try to improve
the English group works. I am sorry, but I cannot do that. (229:C)
I cannot say. Many things can be said in Japanese but not in English.
(57:C)
I have been working on this since April or May, and recently I have beenreflecting on whether I overdid it a bit. I could have done it sooner if I had
used a better lecture method like that of English-language institutes.
(209: A).
That should not be possible. I am sorry. I have given up on that, but wehave group work to improve the English group work. I am sorry, but I
am afraid that is not possible. (229:C)

1. Conclusion
This qualitative and exploratory study intended to define how faculty developers at a
CITL design practical training courses for professors and how they would adjust the plan
to meet participants’ needs and the situation at science universities.
First, in preparation for this FD training course, the FDer led the FD training topromote EMI in specialized classes at science universities and asked an external
lecturer, Lecturer B, to prepare teaching materials and conduct the training. However,Faculty A and Lecturer B, who specialize in foreign language education, did not fully
understand the specialized education at science universities. Therefore, a further
discussion involving Faculty C and D regarding suitable FD training was conducted,
which organized a team from different major fields.
Second, Faculty A reflected on her own struggle to design and implement
successful FD training courses at a science university. Although she was eager toimplement good FD courses at a science university, she did not find the solutions853

convincing. For example, Faculty A stated, “Instructor B was more interested in
discussions, how to teach (language activity), icebreaking, holding a clinic, and so on”(unit 186). In the meeting, Faculty C was hesitant to set the hard-burden courses under
the strong recommendation and pressure to eager in high-level research, saying, “I don't
think it would be a good fit (to bring lesson plans) to have them do a mock class and holda clinic” (unit 192).
Finally, the research question prompted us to aggregate five items into two maincategories: [FDer Losing the Way] and [Clear and Simple Course]. The first main category
included the following three subcategories: <Targets are blurred>, <Pursuing two hares>,and <Discrepancies in training content>. As shown in Table 3, for newcomer Faculty A(FDer) at CITL, the target audience and content for FD training were unclear at a science
university. She then planned the “dual pursuit” FD seminar to improve both educational
methodology and English language skills. However, this has created a mismatch with the"training needs" of science faculty. These three subcategories are summarized as "FDer
Losing the Way.” It was suggested that training be reconstructed as a [Clear and Simple
Course]. It was also clarified that FD training should have a <clear vision of training
courses> or deal with <Simple Skill Training> that focuses on only language andclassroom communication skills. These two subcategories were conceptualized as tips for
FD training course improvement.
There were limitations to conducting an analysis of only one discussion, the findingsof which might be helpful for future qualitative studies that would require clarity andrequirements for successful FD courses at science universities.

Acknowledgments
JSPS KAKENHI Grant Number JP23K02682 has funded this project. Information about
this project can be found online at http://www.katoyukati.net/

References
Fukushi, M. & Nago, N. (2011). Supervisors do not accept residents without a sense of belonging
to the clinical training system for physicians: From a survey of the needs of supervisors at a
training session for supervisors, Medical Education, 42(2), 65–73.
Ministry of Education, Culture, Sports, Science, and Technology, "Status of Reform of
Educational Contents, etc. at Universities in the 2023 Academic Year (Summary),"
https://www.mext.go.jp/content/20211104-mxt_ daigakuc03-000018152_1.pdf (referenced
on 2023.2.25)
Ministry of Education, Culture, Sports, Science and Technology, Super Global University Creation
Support Project: https://tgu.mext.go.jp/about/index.html (Reference date: 2023.2.25)
Kato, Y., & Oishi, T. (2022a). Planning and implementing FD seminars incorporating flipped
learning in engineering universities, Proceedings of National Conference of the JapaneseSociety for Information and Systems in Education (47th), 213–214.
Kato, Y., & Oishi, T. (2022b). E-Content development for English lecture method seminar,
Proceedings of the 28th Annual Conference of the Japan Association of Colleges ofTechnology, A4-1, 42–43.
Kato, Y., Yamashita, Y., Hatakeyama, H., & Oishi, T. (2023a). Implementation of FD training on
English medium instruction at a science university- selection and delivery of training
contents-, The Study Meeting Reports of The Japan Association for Educational Media Study,
No.54, 7–14.
Kato, Y., Yamashita, Y., Hatakeyama, H., & Oishi, T. (2023b). Building a cross-disciplinary facultyteam for improvement of FD seminars at science university, JSiSE Research Report 38(1),
11–16.
Otani, H. (2008). Proposal of SCAT, a qualitative data analysis method with 4 step coding: A
theorizing procedure easy to start and applicable to small data", Graduate School of
Educational Developmental Sciences, Nagoya University Bulletin (Educational Science),
54(2), 27–44.
Otani, H. (2011). SCAT: Steps for Coding and Theorization, Kansei Engineering, 10(3), 155–160.

854

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

What digital tools teachers are ready touse in kindergarten – international
comparative study with early childhood
pre-service teachers
Nataliia DEMESHKANTa*, Siri Sollied MADSENb, Aleksander JANEŠc, AndrejaKLANČARd, Rita BRITOe, Ahmet Sami KONCAf, Sergey KRASINg, Heidi Iren SAUREh,
Jane O’CONNORi, Mustafa JWAIFELLj, Steinar THORVALDSENk & Sławomir TRUSZlaPedagogical University of Krakow, Polandb
UiT the Arctic University of Norway; Norwayc
University of Primorska; SloveniadUniversity of Primorska; Sloveniae
ISEC Lisboa; PortugalfErciyes University; Turkeyg
H.S. Skovoroda Kharkiv National Pedagogical University; Ukraineh
NLA University College; Norwayi
Birmingham City University; United KingdomjAl-Hussein Bin Talal University; Jordank
UiT the Arctic University of Norway; Norwayl
WSB University, Dąbrowa Górnicza; Poland*demesznat@gmail.com
Abstract: This paper presents study results regarding digital tools and work methods
which early childhood pre-service teachers consider using in their future pedagogicalactivities with children. These results are part of wider international research regarding
the digital competencies of the students-future early childhood teachers. The purposeof the presented results is to compare students' attitudes towards digital tools forworking with kids from the perspective of 8 countries (Norway, The UK, Portugal,
Slovenia, Poland, Ukraine, Turkey, and Jordan). There were 772 respondents –students of the Early childhood pedagogical specialization. The study follows thedesign of a survey based on the Theory of Action. The results of the study revealed
that the pre-service early childhood teachers of all nations participating in the study
presented positive approach towards the use of IT tools in kindergarten. The digital
tools which were indicated by students as those that will be most often used in future
work are the Internet as a source of knowledge, digital programs for communication
with parents, video, pedagogical apps, and smart boards. Differences in answers
between representatives of different nations can be explained by the specificity of thefunctioning of educational systems in different countries as well as differences in Early
Childhood teachers' curriculums.
Keywords: Digital tools, kindergarten, international study, early childhood pre-service
teachers

1. Introduction
Professional digital competence for early childhood educators (ECE) is important because
digital devices for young children are evolving at a rapid rate (Arnott et al., 2018).
Nevertheless, researchers and society, in general, seem to be more conflicted and critical
towards the use of technology in ECE compared with other educational levels. Some
researchers highlight that a ‘moral panic’ surrounding screen time appears to havepositioned digital technology as an enemy of early childhood practice - one that could hinder855

the implementation and development of digital practices in early education (Fleer, 2020). On
the whole, early childhood teachers’ perceptions of the benefits and disadvantages of
educational technology integration have been the subject of a moderate amount of empiricalresearch (Hatzigianni & Kalaitzidis, 2018; Magen-Nagar l & Firstater, 2019). Therefore, thecurrent study investigates pre-service early childhood teachers’ attitudes towards the use oftechnology in the educational setting. The goal is to gain a deeper understanding of thisissue within an international educational context.

2. Materials and Methods
2.1 Study sample
This paper presents an overview of the perspectives of early childhood pre-service teachers(n=772) regarding the use of different digital tools and educational methods within theirprofessional practices. The survey was conducted at educational institutions in eight different
countries during the academic year 2021-2022. The universities which were included in the
study: UiT the Arctic University and NLA University from Norway College (n=191); Universityof Primorska, Slovenia (n=177); ISEC Lisboa, Portugal (n=93); Pedagogical University of
Krakow, Poland (n=83); Erciyes University, Turkey (n=82); H.S. Skovoroda Kharkiv National
Pedagogical University, Ukraine (n=55); Birmingham City University, UK (n=46) and AlHussein Bin Talal University, Jordan (n=45). Data were collected from institutions providing
teacher education in each country, applying the same survey structure for all the institutions.
The overall average response rate was 78.38 %.

2.2 Research questions
1. What are the pre-service early childhood teachers' attitudes towards the use of IT
tools in kindergarten differed in every country?
2. What kind of digital tools are ECE students in different countries more likely to be
expected to use in their future practices?

2.3 Study Tools and Procedure
The results presented in this paper are a part of wider international research regarding
digital competencies of the students-future early childhood teachers.
The survey includes pre-service ECE teachers’ attitudes towards the future use ofdigital technology in kindergarten. The following items of the research tool were extracted forthis variable: ECE students’ self-reflection on the use of digital tools in future pedagogicalwork in kindergartens; availability of digital tools in early childhood education; difficulties with
providing digital tools in kindergarten for economic reasons; the use of digital technology
because of professional responsibility; frustration at the need to use IT tools in kindergarten.
This part of the survey consists of optional questions comprising a range of 5-point Likertscale statements ranging from 1 (Strongly disagree) to 5 (Strongly agree). The item
regarding the digital tools which students will use for educational activities with children
contains a list of various digital tools and closed answers to indicate the frequency of theiruse (1 - never, 2 - rarely, 3 - occasionally, 4 - often and 5 - extensively).
An English version of the survey was used as a template for translating the survey into
the different languages needed. Translations were done by researchers from each nation,
thus ensuring a high level of accuracy. Data were collected through the use of online survey
tools, mainly through Nettskjema, a Norwegian tool for the design and implementation ofsurveys and secure online data collection, made for the university and college sector.
Considering the ordinal character of the measurement of variables, the between-groupcomparisons were conducted using the Mann-Whitney U and Wilcoxon W nonparametrictests.

856

4. Results
4.1 Pre-service ECE teachers’ attitudes towards future use of digital technology inkindergarten
The results of the single items regarding the pre-service ECE teachers’ attitudes towards
future use of digital technology in kindergarten revealed differences among the different
nations. These differences are presented in Table 1, using mean scores and standard
deviations.
Table 1. Mean Scores for Single items regarding the pre-service ECE teachers’ attitudes
towards future use of digital technology in kindergarten

Norwayself-reflection
on the use ofdigital tools in
futurepedagogicalwork inkindergartens
4.12 (0.99)

UK

4.26 (0.58)

Portugal

4.27 (0.51)

Slovenia

3.56 (0.82)

Poland

4.13 (1.11)

Ukraine4.00 (0.64)

Turkey4.32 (0.58)

Jordan3.69 (0.85)

average

4.04 (0.76)

availabilityof digital
tools inearly
childhoodeducation
3.96(1.02)
3.76(0.85)
4.12(0.69)
3.33(1.05)
3.65(1.08)
3.67(1.07)
3.95(0.87)
3.53(1.14)
3.74(0.97)
4680.00

Mean Score (SD)difficulties with use of digitalprovidingtechnology
digital tools in askindergarten
professional
for economic responsibility
reasons

frustration at
the need touse IT toolsinkindergarten

4.16 (0.91)

2.16 (1.02)

2.30 (1.11)

4.04 (0.76)

2.48 (0.84)

2.63 (0.74)

4.31 (0.59)

2.21 (1.30)

2.45 (1.23)

3.69 (0.83)

2.42 (0.95)

2.84 (0.96)

3.77 (1.03)

2.83 (1.24)

2.34 (1.11)

4.04 (0.86)

2.42 (0.98)

2.75 (1.02)

4.05 (0.72)

2.00 (0.96)

2.52 (0.92)

4.07 (0.99)

3.58 (0.72)

3.20 (1.10)

4.02 (0.84)

2.51 (1.00)

2.63 (1.02)

Mann4204.50
4792.50
430.002485.50
Whitney U
Wilcoxon W 19957.50
20433.00
20545.50
3833.00
20821.50
Z statistic
-6.09
-6.12
-6.11
-7.45
-4,54
Asym. sig. < 0.001< 0.001< 0.001< 0.001< 0.001(2-tailed)
rg
0.42
0.43
0.42
0.77
0.42
Note: rg - Effect size between the mean ranks in the two compared groups;
The bolded values concern the groups in which the lowest and highest scores relate to a
specific attitude component
When looking at nations’ mean scores independently, the results tend to vary. The effect sizes
between the lowest and highest scoring nations are significant for all the items.
The biggest difference between the highest and lowest score relates to the use of digitaltechnology due to professional responsibility (rg=0.77). Teachers from most countries disagree857

and are neutral regarding the statement of the need to use IT as part of professional duties.
Teachers from Jordan, however, are generally in favor of such a justification for the use of IT.
For the other components of the attitude the effect sizes were similar (about rg=0.40). The
highest level of self-reflection on the use of digital tools in future pedagogical work inkindergartens were recorded among students from Turkey (4.32), whereas the lowest one
were observed among students from Slovenia (3.56). The results for the other attitude
components should be interpreted similarly (see Table 1).

4.2 Students’ future Early Childhood teachers’ opinions about the digital tools for use
during ECE from the perspective of different countries
This variable is a good indicator of the respondents’ reported future use. Obtained results are
based on the mean scores for 17 different tools and work methods (see Table 2).
Table 2. Mean Scores and standard deviation (SD) for Single Variable regarding digital tools
and work methods which will be used by pre-service early childhood teachers in their future
pedagogical activities with children

Digital tools
and methods
Quiz

Norway2,66(1,01)
Digital tools for 3,06
presentation(1,02)
Programs to
3,56
create texts(1,21)
Video use
3,49
(1,07)
Production of 3,05
film/video/(1,10)
animation
The Internet as 4,17
a source of(0,91)
knowledge
iPad
3,78(1,03)
Pedagogical
3,79
apps
(0,96)
Toys and
2,73
technology for (1,17)
codingDigital camera 3,64
(1,07)
Digital2,49
microscope(1,18)
Digital3,33storytelling(1,07)
Digital music 3,30
and sounds (as (1,3)
a producer)
Digital music 3,79
and sounds (as (1,14)
a consumer)

Mean Score (SD)UK Portugal Slovenia Poland Ukraine Turkey Jordan average
3,04 3,27(0,89) (0,63)
3,78 3,01(0,89) (1,06)
3,70 3,15(0,99) (0,94)
4,07 4,04(0,65) (0,55)
3,72 3,67(0,94) (0,86)

2,91(0,90)
3,38(0,96)
3,84(1,19)
3,69(0,82)
3,27(0,99)

3,05(1,23)
3,94(1,06)
4,00(1,20)
2,20(1,12)
2,22(1,21)

3,64(0,80)
3,60(0,89)
4,07(0,81)
3,76(0,84)
3,24(1,02)

2,79(1,23)
3,37(0,85)
2,82(1,22)
4,02(0,74)
3,89(0,80)

3,64(0,88)
3,38(1,05)
2,98(1,03)
3,69(1,10)
3,53(1,18)

3,00(1,02)
3,35(1,02)
3,55(1,19)
3,59(1,04)
3,26(1,12)

4,24 3,95(0,79) (0,80)

3,81(1,07)

4,69(0,66)

4,31(0,81)

4,16(0,78)

3,98(0,92)

4,12(0,92)

3,70 2,92(0,73) (1,05)
3,70 3,68(0,73) (0,77)
3,37 3,05(0,88) (0,95)

2,10(1,03)
3,30(0,99)
2,45(0,99)

2,07(1,30)
3,53(1,17)
1,60(1,01)

2,35(1,24)
3,93(0,86)
2,89(1,05)

3,20(1,16)
4,33(0,77)
3,70(1,05)

2,58(0,89)
4,00(0,93)
2,62(1,05)

2,87(1,27)
3,71(0,98)
2,73(1,17)

3,46 3,41(0,94) (0,78)
2,89 2,77(1,04) (0,98)
3,48 2,84(0,98) (0,80)
3,20 3,29(1,00) (1,02)

3,58(1,00)
2,44(0,94)
3,10(1,05)
3,22(1,11)

2,71(1,43)
1,39(0,93)
1,99(1,21)
1,55(0,91)

3,15(0,93)
2,27(0,99)
3,00(1,02)
3,25(1,14)

3,39(1,02)
3,24(1,02)
3,49(0,91)
3,66(0,84)

2,56(1,06)
2,51(1,01)
3,07(1,29)
3,00(1,23)

3,36(1,10)
2,48(1,13)
3,06(1,12)
3,10(1,22)

3,63 3,65(0,77) (0,80)

3,87(0,98)

2,72(1,49)

3,58(0,99)

3,83(0,81)

3,36(1,28)

3,63(1,11)

858

Art or drawing 3,00
using digital(1,12)
technology
Smart board
2,91(1,21)
Digital
4,15
programs for (1,06)
communication
with parents

3,61 3,03(0,95) (0,98)

2,79(1,05)

1,92(1,21)

3,00(0,82)

3,43(0,85)

2,82(1,17)

2,91(1,12)

4,30 3,31(0,87) (0,80)
4,11 3,85(0,92) (1,00)

2,60(1,07)
3,98(0,96)

2,65(1,44)
1,46
(1,56
)

3,05
(1,10)
3,69(0,90)

3,70(0,91)
4,13(0,77)

3,31(1,35)
4,02(1,08)

3,06(1,20)
3,85(1,16)The obtained results show that the Internet as a source of knowledge was indicated byearly childhood education students from all the nations. Students from Norway reported thatthey intend to use digital programs for communication with parents most often, similarly
indicated by students from Turkey and Jordan. Video, as a tool for frequent uses inkindergarten, was picked up by students from the UK, Portugal and Turkey. Students fromTurkey and Jordan also indicated their intention to use Pedagogical apps frequently in their
future professional work. Students from the UK were the only ones who indicated that they
intended to use the Smartboard often in their work with children in kindergarten.
Summarizing, there were a few kinds of digital tools indicated by students as thosethat will be most often used in future work, including the Internet as a source of knowledge,
digital programs for communication with parents, video, pedagogical apps and smart boards.
Differences in answers between representatives of different nations can be explained by the
specificity of the functioning of educational systems in different countries as well as
differences in Early Childhood teachers' curriculums.

5. Discussion
The obtained results indicate a tendency towards positive attitudes regarding the useof IT tools in kindergarten demonstrated by pre-service early childhood teachers of all thecountries participating in the study. This result indicates the high credibility of theimplementation of innovative technologies in the educational process of kindergarten
students after starting their professional career in kindergarten. Our results are in line withsome previous ones regarding mobile learning as a kind of education in kindergarten(Kalogiannakis and Papadakis, 2019). The authors stated that the pre-service teachers’
attitudes towards the usefulness of mobile learning in the teaching process has the strongest
influence on their intention to adopt mobile learning followed equally by perceived ease ofuse. According to the recent studies (Zilka, 2021) the attitudes of preservice teachers toward
integrating computers into kindergartens are mainly positive, but their perception of thesituation as stressful and threatening is higher than among in-service teachers.
Our results regarding the kinds of digital tools which the pre-service early childhood
teachers are going to use in their future teacher activities in kindergarten are quite diverse
across nations. All the respondents pointed to the Internet as a source of knowledge. Digitalprograms for communication with parents, video, pedagogical apps and smart boards werealso mentioned as students' intentions to use during future teaching with kids. This resultshows ECE students' readiness for the integration of some digital tools into the educational
process in kindergarten. Similar results were reported by Chinese authors, who highlighted
that, preservice teachers conceptualize ICT as screen-based technologies such as
interactive whiteboards, desktop and laptop computers, e-projectors and smartphones, aswell as educational software and applications that can be used on these devices (Dong &Mertala, 2021).

6. Conclusion
Overall, the dynamics of digital practices across the countries reveal that pre-service
early childhood teachers of all the nations participating in the study presented positive859

intentions towards the use of IT tools in kindergarten. Respondents from of all the countries
are aware of different digital tools, hardware, and software for use in ECE. The digital tools
which were indicated by the students as those that will be most often used in future work are
the Internet as a source of knowledge, digital programs for communication with parents,
video, pedagogical apps and smart boards. This outcome calls for a closer examination todetermine whether certain factors can explain the different nations’ strong explanatory powerand statistical significance in terms of attitudes and skills as predictors. Related to this,qualitative studies of different educational contexts are needed to understand the complexityof our results. Finally, further research into the emerging digital pedagogies in ECE isneeded so that there can be more discussions and development of education built on
knowledge and research.
The scope of this study was to present how pre-service early childhood teachers ofdifferent countries consider using digital tools and work methods in their future pedagogicalactivities with children. The findings of the previous studies we examined, showed thatkindergarteners can benefit from the use of new technology and can gain many advantages
and profits using the appropriate educational tools that offer the use of ICT.
However, the teachers’ intentions for using different digital tools are thought to besignificant and demand elaboration and clarification, especially from the perspective ofcomparative international studies. There are lot of learnings that come with the execution ofan multi-institutional and multi-national study like addressing cultural differences, ethicalreview constraints, technology policy differences etc. Address these challenges will offer
valuable insights to researchers in the Asia-Pacific region.

7. LimitationsFirst of all, there was a small number of respondents and the study was conducted only once
and it was cross-sectional. Thus, more research is needed, carried out in more advanced
designs. Finally, a significant challenge was the investigators' cultural differences and theneed to adapt the research tools to the participants' national languages with maintaining
satisfactory psychometric characteristics.

References
Arnott, L., Palaiologou, I., & Gray, C. (2018). Digital devices, internet-enabled toys and digital games:
The changing nature of young children’s learning ecologies, experiences and pedagogies. British
Journal of Educational Technology, 49(5), 803-806.
Dong, C., & Mertala, P. (2021). It is a tool, but not a ‘must’: early childhood preservice teachers’
perceptions of ICT and its affordances. Early Years, 41(5), 540-555.
Fleer, M. (2020). Digital pop-ups: Studying digital pop-ups and theorizing digital pop-up pedagogies
for preschools. European Early Childhood Education Research Journal, 28(2), 214-230.
Hatzigianni, M., & Kalaitzidis, I. (2018). Early childhood educators’ attitudes and beliefs around theuse of touchscreen technologies by children under three years of age. British Journal of
Educational Technology, 49(5), 883–895.
Kalogiannakis, M. & Papadakis, S. (2019). Evaluating pre-service kindergarten teachers’ intention to
adopt and use tablets into teaching practice for natural sciences. International Journal of Mobile
Learning and Organisation, 13(1), 113–127.
Magen-Nagar, N., & Firstater, E. (2019). The obstacles to ICT implementation in the kindergarten
environment: Kindergarten teachers’ beliefs. Journal of Research in Childhood Education, 33(2),165-179.
Zilka, G. C. (2021). Attitudes of preservice kindergarten teachers toward the integration of computers
and the reduction of the digital divide in kindergartens. Educational Technology Research andDevelopment, 69(2), 711-731.

860

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Democratising AI education: Teachingautoencoders to out-of-school children
from low-income backgrounds
Saumay GARGa*
Department of Computer Science and Information Technology, Jaypee Institute ofInformation Technology, India*saumaygarg@gmail.com

aAbstract: Artificial Intelligence (AI) has become an integral part of personal computing
devices and is gaining importance in school curriculums. Several studies and tools
have been developed for teaching AI to school-aged children and it continues to be a
growing area of research in computer science education. However, most of theseinterventions are targeted towards the more privileged and therefore cannot be directly
used with children from socio-economically backward families. Children from suchbackgrounds are often unable to complete even basic elementary education due toseveral social, economic and systemic challenges. This work aims to help extend thereach of AI education to these less privileged children. A learning activity based on thefive AI4K12 big ideas is presented, the activity follows a series of sessions aided by aneasy to create tool for supporting tinkerability. The activity attempts to give a high-level
overview of the functioning of autoencoders. The feasibility of this approach is verified
with a case study involving out-of-school children from low-income families. The
findings show that all of the AI4K12 five big ideas can be addressed and effectively
taught to non-English speaking children with low numeric literacy. The proposed
approach was also shown to increase children’s curiosity and sense of agency whilelearning about AI.
Keywords: Computational Thinking, Inclusive Education, Artificial Intelligence

1. Introduction
Artificial Intelligence (AI) has been accepted to be an integral part of computational thinking(Denning and Tedre, 2019). This is primarily because AI has become omnipresent with mostcomputing devices relying on its applications in one way or another. However, AI can also
have inherent biases that lead to discrimination towards marginalized groups (Angwin et al,
2016; Buolamwini et al, 2018). This omnipresence and potential for bias requires that children
are taught about AI in schools in both a technical and sociological context.
Artificial Intelligence for K-12 (AI4K12) is a globally recognized initiative that supplies
practitioners with standardized guidelines for AI education in schools. This initiative wasstarted as a joint project by the Computer Science Teachers Association (CSTA) and the
Association for the Advancement of Artificial Intelligence (AAAI). The AI4K12 guidelines aredeveloped around the five big ideas for AI education. These big ideas broadly describe thegoals for K-12 AI education while also serving as guiding principles for developing relatedresources and best practices (Touretzky et al, 2022). The 5 big ideas help ensure that the
curriculum and resources focus on teaching all significant areas encompassing the AI fieldand not just a few superficial concepts. These five ideas are represented in Figure 1.
Smart phones have become extremely common in recent years, so much so that evenmarginalized and low-income families own smartphones now. According to a recent report, up
to 88% of Indian households own a smartphone (Bhattacharya, 2022). And surprisingly, evenchildren from these socio-economically backward backgrounds are extremely proficient in theusage of these tiny mobile computers. But these kids are not educated in the more technicalfunctioning of computers and AI. Such children are very likely to be out of school, have never861

attended school or have subpar resources in their poorly funded schools. To put the magnitudeof this problem in context, around 59.1 million primary school-aged children don’t attend
schools (UNESCO, 2019), primarily due to financial and social constraints. Consequently,
these families continue to be stuck in an endless cycle of poverty and are unfortunately devoidof the opportunities that education and computer skills can bring. Lack of research into
teaching practices for such populations makes the educational disparity even moreconcerning.

Figure 1. The five Big Ideas in AI4K12.
Research into teaching and learning of AI for K-12 has examined various best practices
and studied the efficacy of teaching resources for AI. Some of these studies include works by
Druga et al (2019), Long et al (2020) and Payne et al (2019). Though the progress in researchfor supporting AI literacy is promising, there are several unexplored yet significant researchareas. The primary research gaps in AI education that this work addresses are as follows:
1) Though language as a barrier to computing education has been considered by
Banerjee et al (2018), most of the research fails to take into account other socioeconomic conditions relevant to marginalized children with little or no formal schooling.
2) Studies have shown that children develop a more nuanced understanding of AI when
involved with actual code for AI agents and that scaffolding can lead to optimal learning(Druga et al, 2021; Hitron et al, 2019). However, what we don’t know is how children
from low-income backgrounds and lack of access to schools react to this exposure tocode and what scaffolding might be required in such demographics.
3) In an in-depth review of resources for AI education by Druga et al (2022), it was foundthat most of these resources cover only a few of the 5 big ideas of AI4K12 and no
existing resource covers all five.

2. Method
2.1 Participants
The participants for this study were out-of-school children in the age group of 9-13. These arechildren who have once attended school but have since left and don’t plan to rejoin due tofinancial or social constraints. Though these children had attended schools till different grades,
their numerical aptitudes were more or less similar. For example, an 11-year-old boy claimed
to have attended school till fifth grade but was unable to perform single digit additions. Thiscan be attributed to poor instruction in previous schools and frequent absenteeism on thestudent’s part. Therefore, the children in this study were essentially at similar educational
levels. The study participants were the children of migrant workers working low-paying jobs
like domestic help and manual labour in India. They were selected primarily on the basis of
their socioeconomic background. The total number of children who participated in the study
were ten and all except two showed basic numeric literacy (the ability to count and recognize
numbers from zero to ten) or had some understanding of smartphones (experience using
YouTube and Google assistant features). None had any English proficiency and all had Hindi
as their native language. Six of the ten children identified as male and the remaining as female.
The proposed activity was conducted in a single sitting with all sessions taking place one after
another, taking about 50 minutes to complete on average.

862

2.2 Supporting Learning Resource
The teaching resource used in this study is very simple to develop and can easily be recreatedeven without much technical proficiency. The autoencoder and the associated interface wasdeveloped in the Google Colab, a cloud hosted development environment for python andmachine learning. An Autoencoder (Hinton et al, 1989) trained on the MNIST dataset togenerate hand-written digits was programmed using TensorFlow. This code, however, isunlikely to be understood by the child participants because 1) it requires proficiency in English
and the target demographic for this study is non-English speaking and 2) codes for neuralnetworks are not trivial.
The exact architecture of the autoencoder is not introduced to the participants for thesake of simplicity. This study's focus is on teaching the broad concept of encoders, decoders
and latent space representation. To allow the children to interact and tinker with the code,
variables for the latent feature vector (सीखने क मता) and number of epochs (इतनी बार दोहराए)in the code are parameterized with the forms feature in Colab. The value of the latent featureparameter can be adjusted by choosing a value from a dropdown list with three values: 0, 3,
6. Setting it to zero would render the model useless, this can be understood as the neuralnetwork not creating any representation of the input and thereby not learning anything. Higher
dimensions of this vector means that it is retaining more information to reconstruct images
from. Similarly, the parameter for the number of epochs is also presented as a dropdown menu
and the training process is visualized with a progress bar. Finally, an option to supply thetrained neural network with user input is provided. These elements are illustrated in Figure 2.

Figure 2. Users can run a cell, tinker with parameters and view the corresponding codea) dropdown menu for parameterized variables in the code b) progress bar to visualize thetraining process of the neural network c) drawing board for user input.

2.3 Study Procedure
The study introduced autoencoders to the children in a series of logically segregated sessionsaligned with the AI4K12 big ideas (Figure 3). These sessions help structure the learning
process, thereby simplifying both the learning and instruction. Also, they can be held over a863

course of multiple days which can be used as a scaffolding technique. However, in this study,
these sessions were held consecutively on the same day. The 3 sessions are described below:
Session 1 (Develop analogies): The session starts with a general introduction ofcomputing devices using the example of smartphones as these children are most likely tohave access to these devices. This discussion serves two purposes, it helps the instructor
gauge the aptitude of children with respect to computers and clarify any related
misconceptions. Children are then introduced to Big Ideas 1 (perception), 2 (representation)and 3 (learning) using analogies like the ones described in Figure 3 (a). The experience oflearning in a classroom setting was used because the participants found the scenario
relatable.
Session 2 (Introducing code and interface): The personal analogies form the basis
for developing an intuitive understanding of the functioning of an autoencoder. The first 3 big
ideas are sequentially introduced to the children with support from the analogies establishedin session 1. This progression is apparent in Figure 3 (b). The idea is to help children see theneural network as a learning entity like themselves; this is pretty much like applying the theoryof mind (Ensink et al, 2010) for the computer. This process takes place while discussing the
Colab notebook containing the code and interface elements. Showing code helps distinguish
the AI model as a distinctly human made entity, similar yet very different to humans. Exposure
to code also aids the development of a more nuanced understanding of AI (Druga et al, 2021).
The general flow of interaction with the interface is illustrated in Figure 2. Initially thechild uses the run button to run a cell containing a piece of code, completion of its execution
is reflected by a green tick mark. Then, the child is encouraged to experiment with the modelby altering the given parameters in the interface. Once a familiarity is established, the “showcode” option is used to reveal the corresponding code to the child. This procedure can repeat
itself multiple times depending upon the number of cells and exposed parameters in thenotebook. In addition to this, the children are allowed to test the performance of the model on
custom inputs for different latent space sizes.
Session 3 (Usability and impact): AI4K12’s fourth (natural interaction) and fifth(societal impact) big ideas are discussed in the final session. First goal of this session is toelicit a discussion on how the deep learning model is intelligent but unusable for any practicalapplication involving other people. Second, a more open-ended discussion on the potential
usage of such models for social good caters to the requirements of the fifth big idea. This
discussion naturally segues into potential biases of such models.

3. Analysis and Findings
The evaluation is done based on perceptions observed during one-one sessions andinterviews. Efficacy of instruction was gauged based on the child’s ability to understand andrelate to the presented analogies, their understanding of the presented neural network and its
visible parameters, comprehend the associated social implications, and suggest novel use
cases for the AI model. A qualitative analysis led to the following key findings.
A simple form-like interface can improve tinkerability and inquiry of neuralnetworks: Since the children recruited for this study were not fluent in English, they found it
difficult to tinker with the program (like by changing the number of epochs) as they couldn’t
read through the code to find the variable to alter. The supporting interface proposed above
removes this need to understand the code and reliance on English for tinkering with the neuralnetwork. Children in this study could easily find the relevant parameters when provided asinterface elements in their native language. Since the options for such parameters were limited
by a dropdown menu, children found it easier to draw meaningful insights from their inquiry
into the functioning of neural networks. This also prevented them from feeling overwhelmed
as the number of options presented was carefully limited such that only valid permutations
could be formed.

864

Figure 3. The overall progression of the proposed activity a) sample analogies developedaround the AI4K12 big ideas b) equivalent explanations relating the analogies to the actual
functioning of autoencoders c) segregation of the five big ideas into a series of sessions.
Showing code can elicit a sense of agency: Children could easily interact with thenotebook using the interface elements but this alone was not sufficient to elicit ownership anda thorough understanding. The children were unable to think of the AI agent as aprogrammable entity when no code was shown. They simply saw it as another black box
application which they merely use and not actively contribute to (despite them tinkering withthe code parameters). However, when code was shown and it was pointed out how the codechanges when they choose values from the dropdown menu, the kids felt more in control. Thisencouraged them to try new combinations of values and feel more confident in dealing withthe deep learning notebook.
Analogies and discussion can help accommodate most AI education goals:
Though the learning resource presented here did not support all five big ideas, the three
sessions proved to be efficacious in incorporating all the five goals. When analogies related
to learning were established before introducing the code and the interface, the participants’
understanding of autoencoders seemed robust and well-informed. The clear progression of
the three proposed sessions allowed for a smooth transition from one idea to another.
Instruction would have to take place individually: During the group instruction,
significant challenges were faced that prevented the sessions from being successful. It wasdifficult to engage all the kids in even smaller groups as they tended to get distracted by theirenvironments. For example, in many instances, the child was called by their parents to help
with chores like fetching water or collecting donations. Also, unlike a regular classroom, thechildren in the group had significantly different aptitudes due to differences in prior experience.
These differences made it difficult for some children to focus or effectively participate. Somesuch children also acted out and prevented others from engaging. However, they were able
to comprehend and contribute effectively in individual instruction. Individual instruction also
helped deal with the unique needs of these children.
Basic numerical literacy and experience with modern computers is thebottleneck: Children who did not possess basic numeracy skills were unable to make senseof the second session. Basic numeric literacy is required to tinker with parameters and makesense of the task performed by the neural network. These kids seemed disengaged even in
individual interactions, but were able to understand and meaningfully engage in the first andthird sessions. A similar response was observed with children who had no prior experience
with computers, they couldn’t grasp the idea of AI as they had no prior experience. Such865

children couldn’t proceed past the first session. The participants who met these two criteria
sufficiently understood the high-level functioning of autoencoders and were able to generalize
their functionality to other generative tasks like reconstructing alphabets or sentences.

4. Conclusions, Limitations and Future Work
After a case study with ten out-of-school children with minimal numeric literacy and no English
language proficiency, it was found that AI concepts can be meaningfully understood by suchchildren irrespective of other educational differences. Use of analogy driven discussions wasshown to effectively cover all five big ideas described by AI4K12 when instruction took place
individually. Despite being unable to understand code, exposure to code made children feel
more in control when this exposure was mediated with a simple form driven interface. These
interfaces are extremely scalable and can be easily created with little technical expertise, thus
allowing teachers or volunteers lacking in appropriate training to effectively be able to use
such tools in their instruction. Nonetheless, this study was limited by its short duration andsmall sample size. Future research could include longer interventions and comparative studies
with control groups to strengthen the generalisability of these findings. Methods for effectively
discussing the AI4K12 big ideas in greater detail should be pursued while also adapting andrefining the existing progression charts and policies to cater to out-of-school children.
Perceptions and challenges in translating the proposed approach for teaching neural networks
in greater depth and breadth also remain to be explored.

References
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2022). Machine bias. In Ethics of data and analytics(pp. 254-264). Auerbach Publications.
Banerjee, R., Liu, L., Sobel, K., Pitt, C., Lee, K. J., Wang, M., ... & Popovic, Z. (2018). Empoweringfamilies facing english literacy challenges to jointly engage in computer programming. In
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
Bhattacharya, Pramit. 88% of households in India have a mobile phone (2016).
https://tinyurl.com/3988y394. Accessed July 14, 2022.
Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial
gender classification. In Conference on fairness, accountability and transparency (pp. 77-91).
Denning, P. J., & Tedre, M. (2019). Computational thinking. Mit Press.
Druga, S., & Ko, A. J. (2021). How do children’s perceptions of machine intelligence change whentraining and coding smart programs?. In Interaction design and children (pp. 49-61).
Druga, S., Otero, N., & Ko, A. J. (2022, July). The landscape of teaching resources for ai education. In
Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science
Education Vol. 1 (pp. 96-102).
Druga, S., Vu, S. T., Likhith, E., & Qiu, T. (2019). Inclusive AI literacy for kids around the world. In
Proceedings of FabLearn 2019 (pp. 104-111).
Ensink, K., & Mayes, L. C. (2010). The development of mentalisation in children from a theory of mind
perspective. Psychoanalytic Inquiry, 30(4), 301-337.
Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. Parallel
distributed processing: Explorations in the microstructure of cognition, 1(282-317), 2.
Hitron, T., Orlev, Y., Wald, I., Shamir, A., Erel, H., & Zuckerman, O. (2019, May). Can children
understand machine learning concepts? The effect of uncovering black boxes. In Proceedings ofthe 2019 CHI conference on human factors in computing systems (pp. 1-11).
Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In
Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1-16).
Payne, B. H. (2019). An ethics of artificial intelligence curriculum for middle school students. MIT MediaLab Personal Robots Group. Retrieved Oct, 10, 2019.
Touretzky, D., Gardner-McCune, C., & Seehorn, D. (2022). Machine Learning and the Five Big Ideas
in AI. International Journal of Artificial Intelligence in Education, 1-34.
UNESCO, New Methodology Shows that 258 Million Children, Adolescents and Youth Are Out of
School (2019). UNESCO Institute for Statistics Database.

866

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Informatics Education for University
Students based on Text Input Time
Yuko MURAKAMIa* & Tomohiro INAGAKIa
Information Media Center, Hiroshima University, Japan*yuk0mura@hiroshima-u.ac.jpaAbstract: Japan is rapidly changing the ICT environment for students due to the Global
and Innovation Gateway for All (GIGA School) since 2019. Students educated invarious ICT environments will soon enter universities, and higher education will need
to be optimized in these environments. For current university students in the transitional
stage of ICT education, we measured the time required to input text using multiple
methods, including laptops and smartphones, and the age when they started using thedevices. For university students, smartphone input time is shorter than laptop input
time, and individual differences are insignificant, suggesting that smartphone use may
be effective in higher education. We also observed that the students who use devices
such as smartphones and laptops from earlier educational stages have better typing
skills. It will be necessary to design classes assuming a variety of students.
Keywords: ICT device, Informatics Education, Touch-typing, Smartphone, Laptop

1. Introduction
Information literacy is one of the essential liberal arts required today. To acquire information
literacy, using ICT in learning activities is recommended; PISA reported that among OECDmember countries, Japan spent the least amount of time using digital devices in schoolclasses (NIER, 2019). The Ministry of Education, Culture, Sports, Science and Technology
announced the Global and Innovation Gateway for All (GIGA School) to expand theeducational ICT environment, making one terminal per student available nationwide inelementary and junior high schools (MEXT, 2019). This terminal includes not only laptop
computers but also tablet devices. They also have established an intra-schoolcommunication network for elementary, junior high, and high schools.
With realizing the GIGA School, universities will enroll students who have studied invarious ICT environments. One can imagine that new students are particularly susceptible tothe informatics education curriculum they have received through high school. Most current
informatics education assumes the use of laptops, and only individual practice reports havebeen published on teaching methods that use other types of terminals. It is consideredinappropriate for students to take classes based solely on the laptop, as in the past.
Examining how ICT can provide better education to university students is necessary.
However, the ability to use information devices is not apparent regarding current
university students, for whom the environment for information education was not developed.
Our research question is to what extent they can use information devices. We surveyed theage at which students started using laptops and smartphones, the time required for typing asan ICT skill, and students' willingness to use devices. We also discuss effective informatics
education for students who will be educated under the GIGA School in the future.

2. Related research
Japanese university students are known to be familiar with the operation of mobile phones.
Teaching materials are also being developed using tablets and cellphones. Delcker, Honal,867

et al. (2018) suggested from a survey of U.S. university students that tablet devices may
provide new university students with the ability to find effective learning strategies and easy
access to course materials. The effectiveness of regular delivery of English teachingmaterials via cell phones was reported to improve reading and grammar skills at a Japanese
university (Smith & Wang, 2013). However, the effectiveness of ICT education in Japan does
not appear uniform. Gougeon & Cross (2021) suggested differences among Japanese
universities in assessing ICT skills.

3. Method
This study measured the time spent touch-typing on a laptop, typing on a smartphone,reading aloud sentences and writing on paper among first-year university students at
University H. There are 1345 eligible students, including 187 in the School of Medicine, 410
in the School of Engineering, 75 in the School of Dentistry, 81 in the School of Information
Sciences, 80 in the School of Biotechnology and Production, 127 in the School of Integrated
Arts and Sciences, 130 in the School of Law, 54 in the School of Pharmacy, and 201 in the
School of Science.
Two types of texts were prepared for input exercises. The texts are shown in romaji
input, which is commonly used to enter Japanese on a keyboard,
1. Wagahai ha neko dearu. Namae ha mada nai. Dokode umareta ka tonto kentou motsukanu. Nandemo jimejime sita tokoro de nya-nya- naiteita koto dake ha kioku siteiru.
2. AI toha Artificial intelligence no ryaku desu. Artificial toha ‘jinkou teki na’, intelligence ha‘tisei’ to iu imi de, jinkou tinou to yakusaremasu.㻌
Sentence 1 is a passage from ‘I Am a Cat’ by Soseki Natsume. It is assumed that students
who use Japanese daily can read the text easily. Sentence 2 is a brief explanation of
Artificial intelligence. Students are required to input Japanese characters and English words.
The time measurement method was not specified. Each student measured the timein each possible way and entered the assignment form. Some students reported times inunits of time finer than seconds. However, all data were corrected to the nearest second in
the analysis since the accuracy of the reported time varied with this method.
To investigate the relationship between the timing of device use initiation and text
input skills, a questionnaire was sent out on when each device was first used. Five options
were available: before elementary school, elementary school, junior high school, highschool, and after high school. We also surveyed preferences for the terminal used for text
input. The survey target, University H, has adopted a “Bring Your Own Device” policy,
requiring students to provide their own laptops for use in class. Therefore, almost everyone
surveyed should have a laptop and a smartphone and be able to answer these surveys.

4. Result
Figure. 1 lists the time taken by each input method for sentences 1 (left (a)) and 2 (right (b)).
Using Tukey’s outlier removal method, data exceeding 1.5 times the interquartile range areremoved. The figure displays the results of outlier removal. Notches in the figure mean 95%
confidence limits the median. In both cases, sentences 1 and 2, the input time on thesmartphone is faster than the input time on the laptop. The variation in input time is found tobe smaller for laptop, smartphone, hand, and aloud, in that order. This trend is more
pronounced for sentence 2 than for sentence 1.
We compared the average input times for laptops and smartphones. Table 1 shows
the mean input time for sentence 1 and 2. In the case of sentence 1, the t-test results
showed t = 24.912, df = 2194.5, confirming a significant difference at p<0.001. The same
analysis was performed for sentence 2, resulting in t = 13.922, df = 2536.2, also a significant

868

Figure 1. Text input time for laptops, smartphones, handwriting, and speechreading.
Table 1. Mean of Input time for sentences 1 & 2.
MethodLaptopSmartphone

Sentence 1
Num
1303
1275

Mean (S.D.)
73 (28)49 (18)Sentence 2
Num
1312
1288

Mean (S.D.)
94 (34)77 (28)difference at p<0.001. It can be said that students spend more time typing on laptops thanon smartphones.
Figure 2 summarizes the mean input time for sentence 1 by the time when the device
was first used. The links connecting the elements at the top of the figure are represented by‘**’ for significant differences at p<0.01 and ‘*’ for substantial differences at p<0.05 based on
multiple analyses by ANOVA. Figure 2(a) summarizes the age of starting to use laptops andinput time. It can be seen that those who began using laptops after graduating from highschool spent more time (77s, 95%CI is from 74 to 84) inputting data than those who started
using laptops before entering elementary school (61s, 95%CI is from 53 to 68), elementary
school (70s, 95%CI is from 67 to 72), or junior high school (71s, 95%CI is from 67 to 74).
Even if they started using laptops in high school, their input time is found to be longer thanthat of those who started using laptops before junior high school. If students want to improve

Figure 2. Means of input time for sentence 1 by the device start date of use.

869

their touch-typing skills, it is adequate to start using laptops from elementary school.
Figure 2(b) summarizes the text input time by the start of smartphone use. None of
the subjects were students who began using smartphones before elementary school. It was
found that those who started using smartphones in high school (52s, 95%CI is from 42 to 63)
took longer to input text than those who started using smartphones in elementary (45s, 5%CI
is from 42 to 48) or junior high school (49s, 95%CI is from 47 to 50). As with laptops, starting
to use smartphones in elementary and secondary school is more effective in improving
smartphone input skills.
Table 2 summarizes the results of a survey of students on which device was faster to
input devices for sentence 1 and whether they would prefer to input 200 characters on asmartphone or a laptop. 579 students (about 43% of all) preferred the laptop for inputting
200 characters, but were slower to enter sentences and faster on the smartphone. This
indicates that some students' preference for text input devices does not match their choice.
Table 2. Cross table of devices that had the fastest input time for sentence 1 and devices
that took the student less time to input 200 characters.
The device you want to use totype 200 characters.
Devices with short text input time
Total
LaptopSmartphone
Laptop222
26
248
Smartphone
579
492
1071
Same
20
6
26
Total
821
524
1345

5. Discussions
5.1 The Importance of ICT Education in the Elementary and Secondary Curriculum
A comparison of the time spent for each input method showed that keyboard input time tothe laptop varied widely among individuals. Furthermore, there was less variation in the timespent on handwriting and reading aloud. The phenomenal speed with which the Japanesewrite on paper was also noted by Houser & Thornton (2004). This is likely because it isrepeated early and integrated into the primary and secondary education curriculum (Ministryof Education, Science and Culture, 1980). This suggests that teaching methods such as
handwriting and reading aloud may reduce individual differences in device input skills.
A study of the relationship between the timing of device use and input time showedthat students who started using devices by junior high school had a faster input time thanthose who began using devices after high school age. This result suggests that ICT
education would be necessary in primary and secondary education.

5.2 Why Text Input Times Differ Depending on When Device Use Begins
This study found that students using laptops in primary and secondary education had shorter
text input times. This can be attributed to the fact that ICT education, such as touch typing, isbeing provided enthusiastically in some Japanese primary and secondary schools. However,special education was not provided for entering text into smartphones, and this reason
cannot explain the shorter time required to enter text on smartphones. For both devices,
students who had had the devices for longer periods had shorter text entry times, possibly
due to their daily experience with the devices.
Another possible cause is the influence of handwriting skills on device input skills. A
study comparing touch-typing and handwriting speeds among elementary school students in870

the United Kingdom reported that students with no handwriting speed problems were able totype faster on the keyboard than by hand in the upper elementary grades, even without any
education on keyboard input (Connelly, V., Gee, D., & Walsh, E., 2007). Our study results
show that students are faster at typing by hand. Therefore, these handwriting skills may
have influenced their input time to smartphones and laptops. However, this study cannot
reveal the causes of these factors; continued research is needed to clarify the impact ofcontinuing education, especially for students who will enter university in the future as a resultof the GIGA School, since their learning process up to that point was different from that ofprevious students.

5.3 University Information Education: Optimal Content and Environment
Smartphone input is faster than laptops, indicating individual variation. This suggests thatsmartphones save time in situations such as simple communication in class or a quick
exchange of opinions in group work. They are also expected to be more efficient in writing
reports and essays. Hubert (2017) discussed using English smartphone applications forJapanese students learning English. Although laptops can also use English translation
functions, smartphones are faster to type and more suitable for university students.
Some papers question the inclusion of text input methods in informatics education for
university students. Zollman (2012) argued that voice input technology advancements will
make typing skills unnecessary, eliminating the need for text input classes. Voice input
performance has improved significantly in terms of speed and accuracy, as reported by
Ruan (2018). University students were found to read text than they typed it, suggesting that
lectures on using voice input for better outcomes align with informatics education goals.
For students who have low typing speeds when they enter university, it is better toavoid situations where slow typing speeds prevent them from participating in classes rather
than to increase their typing speeds. They are generally given remedial courses, but it isbetter to teach them practical ways to output what they want to write as quickly as possible
rather than just having them do the typing. For example, voice input is suggested forstudents who take a long time to type on smartphones and computers. In this study, as
reading aloud takes the shortest time. Using a voice input system is expected to encourage
students to participate in the class actively.
Around 40% of respondents prefer labor-intensive text input devices. This may a lackof accurate understanding of ICT utilization skills among students. This highlights the needto incorporate opportunities for students to gain correct knowledge of ICT utilization skills
into their learning. A more detailed survey of student attitudes is necessary to clarify the
relationship between text input device choices and self-assessment of ICT utilization skills.
It is increasingly difficult to specify which devices to utilize for university students. ICT
education is now provided from elementary education, and many students will have somedevice before their university course. Thus, it isn’t easy to strongly request university
students to purchase a specific device for their university classes. In particular, classes suchas information literacy are held in the first year. Their content is aimed at learning a wide
range of primary education, so there should be no discrimination in the ease of learning
depending on the terminal students' own. Teachers should develop classes assuming
students can access class materials and practice from various devices.

6. Conclusion
This study aimed to clarify the extent to which current university students have the ability
using information devices and to examine how information education for university students
should be conducted. To this end, we investigated text input time by device and the age at
which students began using the devices. Since individual differences in text input time on871

smartphones are smaller than on laptops, smartphones would be more time-efficient for
simple communication in class and for exchanging opinions in a short period in group work.
The study also showed that students who owned ICT terminals in elementary andjunior high schools spent less time inputting text on laptops and smartphones. Therefore,
when the children and students who are the target of the Global and Innovation Gateway for
All (GIGA School) become university students, their text input skills may have improved
more than those of today’s university students. The devices they use in elementary andjunior high schools and high schools vary, as like laptops or tablets and so on. It is difficult to
identify which devices they have the best input skills. Individual students may have different
input methods in which they excel. It isn't easy to decide which device is more suitable forthem. University classes need to construct course content that does not limit the types of ICT
devices.

Acknowledgements
We would like to thank Naoki Yaegashi and Yusuke Namekawa for the discussions. TI was
supported by JSPS KAKENHI Grant Numbers JP23H01007 and JP23K11349.

References
Connelly, V., Gee, D., & Walsh, E. (2007). A comparison of keyboarded and handwritten compositions
and the relationship with transcription speed. British Journal of Educational Psychology, 77(2),479–492.
Delcker, J., Honal, A., & Ifenthaler, D. (2018). Mobile device usage in higher education. Digital
Technologies: Sustainable Innovations for Improving Teaching and Learning, 45-56.
Gougeon, L., & Cross, Effrey S. (2021). Computational Fluency and the Digital Divide in JapaneseHigher Education. Proceedings of the 29th International Conference on Computers in Education,
1, 672–674.
Hubert, R. P. (2017). The use of smartphones as English dictionaries by university students in Japanand China. Acta humanistica et scientifica Universitatis Sangio Kyotiensis. Humanities series, 50,
367-383.
Huser, C., & Thornton, P. (2004). Japanese college students' typing speed on mobile devices. The
2nd IEEE International Workshop on Wireless and Mobile Technologies in Education, 2004.
Proceedings., 129-133.
MEXT (2019). GIGA School Realization Package [in Japanese]. Retrieved April 28, 2023.
https://www.mext.go.jp/content/20200219-mxt_jogai02-000003278_401.pdf
Ministry of Education, Science and Culture. (1980). Japan's modern educational system: A history of
the first hundred years. Research and Statistics Division, Minister's Secretariat, Ministry of
Education, Science and Culture, Govt. of Japan.
Nagasawa, N. (2017). How Japanese university students type on smartphones and PC [in Japanese].
CICE, 67-72.
NIER (2019). Programme for International Student Assessment - OECD. Key Features of OECD
Programme for International Student Assessment 2018 (PISA 2018). Retrieved April 28, 2023.
https://www.oecd.org/education/school/programmeforinternationalstudentassessmentpisa/35188
351.pdf
Smith, S. & Wang, S. (2013). Reading and grammar learning through mobile phones. Language
Learning & Technology, 17(3), 117–134.
Weigelt Marom, H., & Weintraub, N. (2015). The effect of a touch-typing program on keyboarding
skills of higher education students with and without learning disabilities. Research indevelopmental disabilities, 47, 208–217.
Zollman, A. (2012). Learning for STEM literacy: Stem literacy for learning. School Science andMathematics, 112(1), 12–19.

872

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Investigating Student Teachers’ Learning
Experience and Choice of Participation
Modalities in a HyFlex Course: A Mixed
Methods Approach
Liang Jing TEHa, Su Luan WONGb*, Mohd Zariat ABDUL RANIc, Mas Nida MD
KHAMBARId & Sai Hong TANGeaFaculty of Educational Studies, Universiti Putra Malaysia, MalaysiabFaculty of Educational Studies, Universiti Putra Malaysia, MalaysiacFaculty of Modern Language and Communication, Universiti Putra Malaysia, MalaysiadFaculty of Educational Studies, Universiti Putra Malaysia, MalaysiaeFaculty of Engineering, Universiti Putra Malaysia, Malaysia*suluan@upm.edu.my
Abstract: In this paper, the authors adopted a mixed methods explanatory sequential
approach to examine student teachers’ satisfaction and its relationship withparticipation modalities in a HyFlex course. It also explored the factors behind theirextent of satisfaction and choice of participation modalities. Twenty-six student
teachers completed the satisfaction questionnaire, and four participants were recruited
for a focus group discussion. The research findings indicated that students were verysatisfied with the HyFlex course because the flexibility in attendance and the easy
access to learning materials accommodated their learning needs and personalcommitments; this promoted self-directed learning and reduced their academicpressure. The student teachers were also satisfied with the HyFlex course because
they gained exposure to an innovative learning format. As for the factors affecting
student choice of participation modalities in the HyFlex course, peer influence is one of
the significant factors. Students also reported that they chose the synchronous mode
based on their learning preferences and empathy towards the instructor's commitment,
while they chose the online options due to their energy level, health conditions, andpersonal matters. Overall, this study suggests that HyFlex may be considered as alearning model that could cater to various learning needs among student teachers
majoring in baccalaureate in education (physical education), and potentially enhancetheir learning outcomes.
Keywords: HyFlex, satisfaction, participation choice, student teacher, blended

1. Introduction
Student dropouts have become pervasive in Asia (Latif et al., 2015). Since the coronavirus
disease 2019 (COVID-19) outbreak, Malaysia has also been no stranger to this problem. The
New Straits Times (2022) and FMT Reporters (2022) reported over 17,000 dropouts among
public university students in 2021, which marked an increase of over 4,000 compared to theprevious year. In contrast, more than 20,000 students postponed their studies for private
institutions. During the same period, Malaysia also suffered a woeful learning loss rate of 0.95
years, one of the highest learning losses among Asian developing countries (Ferlito et al.,2021). Although COVID-19 has brought unprecedented obstacles to education, it boosted
students’ adoption of online learning three-fold (Robert, 2022), allowing institutions topropagate innovative and flexible instructional approaches. Hybrid-Flexible (HyFlex) course is
one of the learning models that flourished as a remedy in mitigating the challenges imposed873

by the closure of institutions due to the pandemic (Verrecchia & McGlinchey, 2021; Harris etal., 2020) although the concept of HyFlex has existed for more than a decade (Beatty, 2019).
A HyFlex course “provides the capacity for programs to serve remote students inadditional to providing convenience and alternatives to regional students” (Beatty, 2019, p.
12). It allows student to choose from one of the three participation modalities, namely physical(attend physical classroom), online synchronous (attend via online conference call platform),and online asynchronous (watch the recording of the class session). Students can change
their participation for each class session depending on their personal preferences or needs(Beatty, 2019). The HyFlex learning model offers various benefits such as increasing learning
access and flexibility to the learners (Lakhal et al., 2017), providing enriched learning materials
that cater to diverse learning styles and preferences (Abdelmalak, 2014), enhancing students’learning outcomes and course completion rates (Irvine et al., 2013), as well as enabling
institutions to have higher student enrolment and revenue, reduce unnecessary cost, andcreate crisis-proof courses (Beatty, 2019). However, implementing this model poseschallenges like additional workload for the instructors (Lakhal et al., 2017), managing
engagement among both online and in-person learners (Conklina et al., 2017), andtechnological issues (Bower et al., 2015), which can affect students’ learning experience.
Hence, this study aims to investigate learning satisfaction and choice of participation
modalities as well as the factors affecting both variables among student teachers in Malaysia.

2. Literature Review
There are several studies which evaluated student satisfaction towards HyFlex courses. A
qualitative study was conducted by Abdelmalak (2014) which explored the satisfaction of six
university students who joined a HyFlex course and found that students have more control
over their learning. Their schedule, learning styles, and needs were accommodated by theflexibility of participation offered by the HyFlex course. Lakhal et al. (2014) did a quantitative
research with 439 undergraduate students enrolled in a 10-week HyFlex course, the study
revealed that satisfaction level among synchronous students was significantly higher than thestudents who mostly attended the course asynchronously. A mixed methods study with 311
tertiary level students done by Gobeil-Proulx (2019) found that the flexibility and convenience
provided in a HyFlex course were greatly appreciated by students. Interestingly, about 70% of
the students stick to one type of participation modality throughout the semester, but thereasons behind this phenomenon were not explained. Rhoads (2020) also did a mixed method
study on a HyFlex learning format which comprised 81 undergraduate students; the qualitative
findings showed that student satisfaction had a positive relationship with flexibility inattendance. Similar results regarding satisfaction towards the HyFlex learning format werealso found by Kakeshita (2021), who showed that around 85% of the 135 students reported
high satisfaction with the HyFlex course because their learning needs were met, findings fromthe article also indicated that students revisit the learning content as recordings of the lecture
were given.
Nevertheless, most of the aforementioned studies also revealed a variety ofpredicaments associated with the HyFlex model. Students reported difficulties staying
engaged in the online modalities and were concerned about the quality and consistency of
instruction across different modalities (Gobeil-Proulx, 2019). It was also found that somestudents still prefer the traditional learning mode as it motivates them to complete their learning
tasks, and students felt it easier to engage with their instructor should they have any questions(Rhoads, 2020). Furthermore, Liu and Rodriguez (2019) mentioned that students needed help
understanding the HyFlex learning format, which caused them to underutilize the learning
resources and have less interaction with the instructor.
Compared to the number of studies on online teaching, “research regarding students’
perceptions of HyFlex is limited” (Abdelmalak & Parra 2016, p.20), particularly on the factors
affecting students’ choice of participation modalities in a HyFlex course. Students' attendance874influences learning outcomes, but the impact may vary depending on the teaching modality
and individual learning styles (Green, 2021). Students' decisions on the participation modecan be influenced by logistical factors, health conditions, and personal obligations (Blankson
et al., 2014). The Unified Theory of Acceptance and Use of Technology (UTAUT) model can
explain why students chose a particular participation mode. The UTAUT model is a theoreticalframework developed to understand and predict individuals’ acceptance and use of technology.
According to Venkatesh et al. (2003), four key factors that would influence one’s behavioral
intentions and adoption of technology were identified, namely performance expectancy (users'
belief that using the technology will enhance their performance and productivity), effort
expectancy (users' perception of the ease of use and the effort required to use the technology),social influence (the impact of social factors, such as social norms and peer influence, on
users' acceptance and use of technology), and facilitating conditions (the extent to which users
perceive that the necessary resources and support are available to use the technology
effectively). To put these factors in the context of participation modalities selection among thestudents in a HyFlex course, an introverted student may choose to participate via online whilemore extroverted students may choose the physical mode (performance expectancy), learners
that are staying further away from campus or have a packed schedule might opt for the online
options (effort expectancy), one might follow when their friends decided to participate via a
specific learning mode (social influence), and students that have connectivity issues or areless familiar with online learning tools may lean towards the physical mode of learning(facilitating conditions).
Given the mixed perceptions towards a HyFlex course among the students, as well as
the lack of studies regarding HyFlex learning, there is a need to investigate students’ learningsatisfaction and the reasons behind their decision of participation modalities in a HyFlex
course. As such, the research questions addressed in this study are the following:
1. What is the extent of student teachers’ satisfaction with the HyFlex course?
2. Is there a significant relationship between participation modalities and students’satisfaction in a HyFlex course?
3. What factors affect student teachers' satisfaction with the HyFlex course?
4. What factors affect student teachers' choice of participation modalities in a HyFlex
course?

3. Methodology
3.1 Research Design
This study employed an explanatory sequential mixed methods approach. According to
Creswell and Clark (2018), this research approach begins by conducting the quantitative
phase. A subsequent qualitative phase follows it to dive deeper into specific results from theinitial phase. The main intent of this design is to utilize the qualitative component to explain
the quantitative results.

3.2 Participants and the HyFlex Course
A total of 56 student teachers from the Bachelor of Education (Physical Education) programme
registered for the educational technology course which was taught for 14 weeks at the end of2022. Three participation modalities were provided to all students. For each class session,
they were allowed to choose whether they would like to participate by being physically in the
Putra Future Classroom (PFC) (in-person), via Zoom or Skype (online synchronous), or bywatching the recorded lectures (online asynchronous). PFC is equipped with overheadcameras, high-speed broadband, multiple TV screens, and wireless audio system, allowing

875

face-to-face and online synchronous students to interact. The recordings of the weekly lecture
and the learning materials were uploaded to the learning management system accessible by
the students.
For the quantitative phase of this study, 26 students (14 males, 12 females) completed
the questionnaire, four participants (2 males, 2 females) from the same group were
purposively recruited to participate in the focus group discussion (FGD), two (1 male, 1 female)of the four participants mostly participated the HyFlex course physically while the other two (1male, 1 female) mostly participated the course online synchronously.

3.3 Data Collection and Data Analysis
A satisfaction questionnaire which comprises 11 items were adapted from Shek et al. (2022)
to measure student teachers’ satisfaction towards the HyFlex course. Shek et al. (2022) tested
the reliability of the items in 11 different lectures and obtained an average Cronbach’s alphavalue of more than .97. In this study, the learning satisfaction measured was segregated into
two categories, namely course design (6 items) and instructor attributes (5 items). All the items
used a five-point Likert format, ranging from 1 to 5 (1 = strongly disagree to 5 = strongly agree).
Table 1 illustrates the 2-times-2 contingency table of participation modalities and satisfaction.
Based on the table, the assumption of chi-square test is not met as 50% (more than 20%) of
the cells have expected count less than 5. Thus, the Fisher’s Exact test was used, this test isalso particularly useful for a small sample size and a two-times-two contingency table (Sprent,2011), which fits this study well.
After quantitative data analysis, a FGD was conducted to follow up on quantitative
findings requiring more context. Qualitative data were collected via a combination of structured
and unstructured questions. Before the FGD, prompt questions were pre-prepared according
to the research objectives and throughout the FGD, open-ended questions were also asked
spontaneously so that the participants’ responses are not limited to the researcher’s bias
(Creswell & Creswell, 2018). For qualitative data analysis, content analysis is employed as itallows researchers to systematically identify key themes within the FGD transcript (Erlingsson& Brysiewicz, 2017). All four students’ names were modified to S1 to S4 in the qualitative
findings section to keep their personal information confident.
Table 1. Crosstabulation between participation modalities and satisfaction.
Satisfaction
Participation Mostly at PFC
Modalities
Mostly via Zoom
or Skype
Total

Satisfied

Very satisfied

Total

Count

10

8

18

Expected Count

7.6

10.4

18.0

Count

1

7

8

Expected Count

3.4

4.6

8.0

Count

11

15

26

Expected Count

11.0

15.0

26.0

4. Quantitative Findings
As shown in Table 2, most of the respondents (69.2%) regularly participated the lecture
physically, followed by online synchronous mode (30.8%), and no respondent reported that876

they often participate the lecture online asynchronously. Based on Table 3, more than 80% of
the students strongly agreed or agreed that they were satisfied with both the course design
and instructor attribute domain of the HyFlex course. The overall mean score for satisfaction
towards course design and instructor attribute were 4.52 (SD = .70) and 4.64 (SD = .62)
respectively, showing a high rating of overall satisfaction towards the HyFlex course. The
Fisher Exact significant value (2-sided) obtained is P = .08, as the significant value is morethan the alpha value (ߙ = .05), this indicates that no significant association was found between
participation modalities and satisfaction.
Table 2. Frequency and percentage of student teachers’ participation modalities preference.
Participation modalities

Frequency

Percentage

Mostly at PFC

18

69.2

Mostly via Zoom

8

30.8

Mostly watch recording

0

0

Total

26

100.0

Table 3. Frequency and percentage of student teachers’ satisfaction.

Items

Strongly
Disagreedisagree

NeutralAgree

Strongly
agree

݂

%

݂

%

݂

%

݂

%

݂

%

1. Overall, I like the learning
experience in this HyFlex course.a

0

0

2

7.7

3

11.5

5

19.2

16

61.5

2. The course design is well thought
out. a

0

0

1

3.8

1

3.8

4

15.4

20

76.9

3. It is easy for me to collaborate
with my peers in this course. a

0

0

1

3.8

0

0

5

19.2

20

76.9

4. This HyFlex course allows me tolearn the lecture according to mysituation. a

0

0

0

0

2

7.7

9

34.6

15

57.7

5. This HyFlex course allows me tolearn the lecture efficiently. a

1

3.8

1

3.8

1

3.8

9

34.6

14

53.8

6. This course helped me inunderstanding the concept of
educational technology. a

0

0

1

3.8

0

0

7

26.9

18

69.2

7. The instructor has a good
mastery of the learning materials. b

0

0

1

3.8

0

0

5

19.2

20

76.9

8. The instructor used different
methods to promote learning. b

0

0

1

3.8

0

0

4

15.4

21

80.8

9. The instructor is able to help
students understand the topics
covered in the lecture. b

0

0

1

3.8

0

0

7

26.9

18

69.2

877

10. The instructor can manage both
physical and online students
effectively. b

0

0

1

3.8

0

0

9

34.6

15

57.7

11. Overall, my evaluation for the
instructor is very positive. b

0

0

0

0

1

3.8

6

23.1

19

73.1

Notes: aCourse design domain, binstructor attributes domain.

5. Qualitative Findings
The factors behind students’ satisfaction were explored in the qualitative phase. This section
begins with reasons why students were satisfied with the HyFlex course, which comprises (1)meeting learner’s needs, (2) promoting self-directed learning, and (3) providing affective andprofessional benefits. It is followed by the factors affecting student choice of participation
modalities in the HyFlex course. It was reported that students chose the synchronous mode
based on their personal preference and empathy towards the instructor’s commitment while
they chose the online options due to energy level, together with health and personal matters.
Peer influence plays a significant role in affecting their decision too.

5.1 The Flexibility of Attendance Fulfils Learners’ Needs
When asked about what they liked about the HyFlex course, one of the students stated thatthe flexibility provided by HyFlex is what he desired most, “…we can choose which[participation modalities] we want or the one that is most suitable for us… If we can’t attend
live, we can still watch back the recording” (S1). Another student recalled one scenario duringthat course and shared: “…I remember why I joined via online is because I was tested positive
for COVID, and during that time everyone was doing group discussion…” (S2), that student
continued sharing by holding up her phone with one hand and waved towards her phone withanother hand: “…and they video called me from the physical classroom like ‘hi’” (S2). The
same student shared the struggles faced by the students in attending lectures: “although timeis allocated for each subject, most of us have commitments like training as we are athletes
right, so we would have training outside of campus…” (S2). Other students further added thatlearning still occurs in spite of the participation modalities which made learning moreconvenient for the students, “...like in Zoom, we can be divided into breakout rooms and haveactivities…” (S3). “I agree… the options give us convenience and ease because we canchoose the time we learn” (S4).

5.2 The Accessibility of Learning Resources Empowers Self-directed Learning
Another theme that emerged from the participants’ positive experience with the HyFlex courseis that students take ownership of their learning process. A student mentioned that her selflearning was enriched by the accessible learning materials provided in a HyFlex course: “itcan become a self-learning material too, like we did join the class physically, then after theclass, instead of opening the slides and take notes only, we can actually watch back what has
been said by Prof. via the recording, so from there we can revise” (S2).
Another student resonated with the fact that students have more autonomy on how
they learn in a HyFlex course: “Like for me honestly, I’m the last-minute kind of student, so I
would use all the learning resources provided by Prof. and study before the final exam. So it’svery helpful for me because I just click, and all the lectures and learning resources from week
one to week 13 are there” (S3). The same student also pointed out that the learning resources
provide student with different learning progress the same learning opportunities, making
learning more equitable to all students: “because it’s like unfair to those that couldn’t make it
to class, as they would not learn the knowledge of that session. Like ‘S2’ mentioned just now,878

we still have the learning materials and recordings provided, so there’s no excuse to not study”(S3).

5.3 HyFlex Course Provide Students with Affective and Professional Benefits
When prompted deeper on how the flexibility and autonomy benefits the students, one of theparticipants expressed that HyFlex learning format reduced her psychological distress: “Forme, it reduces my pressure, as in we don’t know right sometimes, we have emergency or othermatters that are critical, so I can’t join the class physically, it’s okay because I can have anotherchoice to gain the knowledge that Prof. taught on that session” (S2). Another student agreed
by saying: “…we also have back-to-back classes, the distance between two classes can bevery far away too, so it will be hard. Like we need to make sure the classes end on time right,
let’s say this lecturer does not finish class on time, it will drag and affect your next class. So I
think this HyFlex can reduce students’ pressure as they can choose” (S1).
Besides reducing academic pressure, the students stated that HyFlex course canincrease their motivation to learn due to the ubiquitous learning resources given: “let’s say thecourse is only physical, once you missed a session, you will need to approach your friends or
the lecturer to catch back on the learning content for that session. When we have HyFlex, likewe said before all the lectures and learning materials are there. So you don’t have to like
WhatsApp your friend, and your friend would need effort to recall and reply, maybe that friend
know the learning content but it’s not 100% accurate like in class” (S3). Intriguingly, aparticipant perceived being in a HyFlex course as a chance to be exposed to different
instructional methods that can potentially enhance his future career, “…we can get a taste ofrunning physical, online synchronous, and online asynchronous class. I can observe how Prof.
prepare and conduct the different medium of learning so that all students are not left out in
learning. So I think experience like this is important for us especially the ones that are majoringin education for our future career” (S1).

5.4 Factors Affecting Student Choice of Participation Modalities in a HyFlex Course
Regarding the reasons behind students’ choice of participation mode for each class session,
the students shared that they chose to participate physically because of their empathy towards
the instructor, “…only a few students in physical class and online synchronous, I feel like it's
not respecting the lecturer…and I empathize with lecturers that have put in the commitment
to teach” (S1). This comment was seconded by another student, saying: “Like what 'S1' said,
we respect Prof. as our lecturer. How would she feel if she sees four physical students andnine online students only. At that moment she did not show her feelings, but we need tounderstand, because we are going to be in her shoe also one day” (S2). Students decide their
participation mode based on their learning style too, one participant shared at length about
this factor: “Like for me, I don’t really like watching recording, some more the videos are always
like long…more than an hour… I need to just sit down, it’s really not my style” (S1).
Moreover, students also shared that they chose the online options when their degreeof fatigue is higher, “Like even at night time we still have classes and it’s all physical classes,
it’s very tiring, and the next morning we also have classes…so it becomes like ahh just watch
the recording would be enough la” (S1), “I also have training every day, so will feel like lazyhaha, so my option is online. But it depends on the student himself, if [I] feel like [I] have theresponsibility to study, then I will go [physical class]. But there are times when the students
don’t join even join through online, didn’t join at all…” (S4). When participants were asked
about why those students did not join, one participant claimed that: “maybe they are notinterested with the subject” (S3), but another participant disagreed and stated that they did notjoin because of the lack of interest in study as a whole instead of that subject only, and further
described the unique learning style of physical education students: “…we are more interested
in workshops, do activities instead of sit and listen, can’t focus on one thing for a long time”
(S2).

879

Peer influence was also one of the driving motives that affected student's decision, one
student explained how this happened: “Like for example our class is until 11pm at night, so
we become really tired right, so when our friends are tired and do not feel like attending, we
would follow, and sometimes we would be the ones initiating and them follow too” (S3). Other
than these factors, the previous section also indicated that students chose the online options
due to health matters like the contraction of COVID-19 as well as personal commitments suchas trainings.

6. Discussion
The primary aim of this study is to examine student teachers’ extent of satisfaction towards a
HyFlex course and the association between participation modalities and satisfaction, as well
as exploring the factors affecting their level of satisfaction and decision on participation
modalities in a HyFlex course. It is important to mention that there is a possibility that thestudents who participated mostly via online asynchronous mode did not respond to the survey
as the participant indicated that they do not have an interest in studying in general.
The high overall mean scores for satisfaction towards both the course design andinstructor attributes of the HyFlex course found in the quantitative phase were validated by thequalitative findings, the student teachers appreciated the flexibility in participating the class foreach week as they were allowed to learn according to their learning preferences and personalschedules. When the student teachers could not attend the class synchronously due topersonal commitments or unforeseen reasons like health issues, HyFlex enabled them to learn
using their most effective ways and keep up with the learning progress. The flexibility offeredby HyFlex was particularly important to the student teachers in this study because the majorityof them were athletes, and they had commitments like trainings and competitions that often
clashed with the timing of the class sessions. This flexibility increased the ease of learning
within the students and helps them balance their academic progress and personal
responsibilities, contributing to their overall satisfaction towards the course. This is in line withthe findings in the studies done by Rhoads (2020), Gobeil-Proulx (2019), and Lakhal et al.
(2017) which concluded that students valued the flexibility and convenience provided in a
HyFlex course.
The statistical findings also showed that the level of satisfaction towards the HyFlex
course among the student teachers is independent of the types of participation mode, and thiscan be explained via the responses from the FGD which revealed that students had equal
learning opportunities despite their choice of participation mode. This is because the enrichedlearning materials such as recordings of lectures, reading materials and discussion forums
were ubiquitous and accessible to all students within the HyFlex course. This prevented thestudent teachers from being left out as they could still participate in the lecture asynchronously
when they were unable to be present during the class sessions due to personal obligations.
This is especially beneficial for the participants who were athletes in this study, as their regularcommitment to trainings and competitions would often leave them feeling exhausted, making
them more prone to being absent or distracted in class. By providing multiple modes of accessto the learning content, all students with different learning needs can have an equal opportunity
to succeed in the course. This finding resonated with the research by Abdelmalak (2014) and
Kakeshita (2021) which stated that HyFlex courses can cater to different learning needs.
Findings from this study also found that with the flexibility, accessibility, andconvenience given in a HyFlex course, learners become not only more satisfied, but also more
motivated to learn, making their learning more self-directed (Kakeshita, 2021). This canempower learners to take ownership of their learning by deciding when and how they want tolearn. For instance, the responses from the FGD in this study showed that before exams,
students would re-experience the lecture via the recordings to review the course materials
multiple times for a better understanding. This can potentially improve students’ learning
outcomes, reduce learning loss and dropout rates, and increase the overall course completion
rate amongst the students (Beatty, 2019; Irvine et al., 2013). This finding is particularly
pertinent to Asian learners as their classrooms tend to be more teacher-centered, but880

classrooms with a more student-centered instruction within East Asian societies have beenfound to have higher achievement (Kim, 2018). To the researchers’ surprise, a HyFlex course
can also develop students’ future career prospects as they are exposed to the innovative
instructional approaches of HyFlex throughout the course. They could learn effective ways of
conducting physical, online synchronous, and online asynchronous instruction, which could
contribute to their professional growth, considering their future roles as educators. This handson experience could also concretize students’ learning about educational technology. Thisfinding is rarely seen in the literature on HyFlex learning, but the researchers acknowledge
that this finding would most likely only ring true in cases where the learners of the HyFlex
course are majoring in education-related programmes of study.
Furthermore, this study also revealed that although HyFlex can cater to various
learning needs, practitioners must be aware of students’ learning styles and tailor theirinstruction accordingly to ensure the quality of the learning experience. As most of the student
teachers in this study were athletes, they admitted that their learning style leans towards thekinesthetics’ side of learning style. Thus, the instructor needs to utilize teaching methods which
are more tactile to keep the learners engaged. However, this study indicates that when it
comes to student choice of participation modalities, there were factors which are out of thelocus of control of the instructor. Student’s decision to participate in a class session can directly
influence their peers’ decisions as learners that are close friends tend to stick together
throughout the instruction. Findings from this study also suggested that students attended theclass synchronously because they felt bad for the instructor if only a few students were present
during the lecture. This shows that Asian students may conform to classroom norms due totheir respect for authority figures such as instructors in a HyFlex learning environment (Li &Campbell, 2008). Additionally, responses from the FGD suggested that when students are notin good health condition, feel exhausted due to a packed academic timetable and personalcommitments like training, or are bound by personal obligations like a family plan whichclashes with the lecture’s timing, they would opt for the online options. This finding parallels
the study by Blankson et al. (2014), which concluded that student choice of participation mode
was influenced by their health, personal commitments, and commuting challenges. In otherwords, different groups of students would have a different pattern of participation modalities
selection as their academic schedule as well as their attitudes towards learning and the
instructor would be different. Also, students’ choice of participation mode would be affectedduring the period of homecoming festivals as students would prioritise travelling plans, or
during the onset of the cold and flu season when more learners are more prone to getting ill.

7. Conclusion
As a whole, this study showed that HyFlex course can potentially be a suitable learning modelfor Asian student teachers who are majoring in bachelor’s degree in education (physicaleducation) mainly because the flexibility in classroom participation and the equity in learning
opportunities. The HyFlex learning model was able to accommodate students’ learning needs
due to unforeseen circumstances and personal commitments, which enabled learners to havethe best of both worlds between their academic progress and personal lives. With theaccessible and ubiquitous learning resources, learning becomes more self-directed and more
equitable among students, this may alleviate their anxiety about falling behind in learning andallow them to feel more motivated to learn. This could potentially enhance their learning
performance as well as reduce dropout rate and learning loss among the students. Other thanthat, student teachers also benefitted from the HyFlex course in terms professional
development as they gained exposure to integrating an innovative learning format. Suffice tosay, policy makers, faculties and instructors should consider incorporating the HyFlex learningmodel into the teaching and learning experience.
Regarding student choice on participation modalities, this study found that students
choose their participation mode depending on their empathy towards the instructor, learning
style, level of fatigue, and influence from their peers. The trend of students’ choice ofparticipation can be predicted based on factors such as the holiday season and times when881

illnesses are more prevalent — leading to a higher number of students choosing the online
options. Future studies can look into learning interest or motivation among students and how
it relates to their preferred participation modalities in a HyFlex course as this study suggeststhat students’ choice of learning mode was affected by their learning desire. While qualitative
findings from this study have shown that a HyFlex course can promote learning equity, futurestudies can provide more empirical results regarding learning equity across all participation
modalities.

8. Limitations of the Study
A few limitations exist within this study. Firstly, the small sample size which did not encompass
students who mostly participated in the lectures asynchronously, and the fact that all
participants in this study were from the physical education programme affect the
representation of this study towards other student groups, a larger number of samples from amore diverse demographics can be recruited to expand the generalizability of the findings
further. Moreover, the data collected were self-reported and were based on the student
teachers’ honesty and perception of their learning experience in the HyFlex course.

References
Abdelmalak, M. (2014, March). Towards flexible learning for adult students: HyFlex design. In Society
for Information Technology & Teacher Education International Conference (pp. 706-712).
Association for the Advancement of Computing in Education (AACE).
Abdelmalak, M. M. M., & Parra, J. L. (2016). Expanding learning opportunities for graduate students
with HyFlex course design. International Journal of Online Pedagogy and Course Design(IJOPCD), 6(4), 19-37.
Beatty, B. (2019). Hybrid-flexible course design. EdTech Books.
Bower, M., Dalgarno, B., Kennedy, G., Lee, M. J., & Kenney, J. (2014). Blended synchronous
learning: A handbook for educators. Sydney, Australia: Office for Learning and Teaching,Department of Education.
Conklina, S., Oyarzun, B., & Barreto, D. (2017). Blended synchronous learning environment: Student
perspectives. Research on Education and Media, 9(1), 17-23.
Creswell, J. W., & Clark, V. L. P. (2018). Designing and Conducting Mixed Methods Research. 780.
Creswell, J. W., & Creswell, J. D. (2018). Research Design: Qualitative, Quantitative, and Mixed
Methods Approaches. 388.
Dziuban, C., Graham, C. R., Moskal, P. D., Norberg, A., & Sicilia, N. (2018). Blended learning: the
new normal and emerging technologies. International journal of educational technology in Higher
education, 15, 1-16.
Erlingsson, C., & Brysiewicz, P. (2017). A hands-on guide to doing content analysis. African Journal of
Emergency Medicine, 7(3), 93–99.
Ferlito, C., Sazuki, F., Lim, A., & Weerasena, B. (2021). The Economic Impact of School Closures in
Malaysia.
FMT Reporters. (2022). Over 17,000 dropped out of Govt varsities last year, says minister. FMTMedia Sdn Bhd. https://www.freemalaysiatoday.com/category/nation/2022/03/03/over-17000dropped-out-of-govt-varsities-last-year-says-minister/
Gobeil-Proulx, J. (2019). La perspective étudiante sur la formation comodale, ou hybride
flexible. Revue internationale des technologies en pédagogie universitaire, 16(1), 56-67.
Green, K. (2021). Lecture modality: Student attendance choices and performance. In Advances in
Accounting Education: Teaching and Curriculum Innovations (Vol. 25, pp. 119-131). Emerald
Publishing Limited.
Harris, B. N., McCarthy, P. C., Wright, A. M., Schutz, H., Boersma, K. S., Shepherd, S. L., ... &Ellington, R. M. (2020). From panic to pedagogy: Using online active learning to promote
inclusive instruction in ecology and evolutionary biology courses and beyond. Ecology andevolution, 10(22), 12581-12612.
Irvine, V., Code, J., & Richards, L. (2013). Realigning higher education for the 21st century learnerthrough multi-access learning. Journal of Online Learning and Teaching, 9(2), 172.

882

Kakeshita, T. (2021, April). Improved HyFlex Course Design Utilizing Live Online and On-demand
Courses. In CSEDU (2) (pp. 104-113).
Kim, Y. (2018). Revisiting classroom practices in East Asian Countries: Examination of within-country
variations and effects of classroom instruction. Teachers College Record, 120(7), 1-42.
Kyei-Blankson, L., Godwyll, F., & Nur-Awaleh, M. A. (2014). Innovative blended delivery and learning:
exploring student choice, experience, and level of satisfaction in a hyflex course. International
Journal of Innovation and Learning, 16(3), 243-252.
Kyei-Blankson, L., Godwyll, F., & Nur-Awaleh, M. A. (2014). Innovative blended delivery and learning:
exploring student choice, experience, and level of satisfaction in a hyflex course. International
Journal of Innovation and Learning, 16(3), 243-252.
Lakhal, S., Bateman, D., & Bédard, J. (2017). Blended Synchronous Delivery Mode in Graduate
Programs: A Literature Review and Its Implementation in the Master Teacher Program. Collected
Essays on Learning and Teaching, 10, 47-60.
Lakhal, S., Khechine, H., & Pascot, D. (2014, October). Academic students’ satisfaction and learning
outcomes in a HyFlex course: Do delivery modes matter?. In E-Learn: World conference on ELearning in corporate, government, healthcare, and higher education (pp. 1075-1083).
Association for the Advancement of Computing in Education (AACE).
Latif, A., Choudhary, A. I., & Hammayun, A. A. (2015). Economic effects of student dropouts: A
comparative study. Journal of global economics.
Li, M., & Campbell, J. (2008). Asian students’ perceptions of group work and group assignments in a
New Zealand tertiary institution. Intercultural Education, 19(3), 203-216.
Liu, C. Y. A., & Rodriguez, R. C. (2019). Evaluation of the impact of the Hyflex learning model.
International Journal of Innovation and Learning, 25(4), 393.
New Straits Times. (2022). NST Leader: Dropout by degrees.
https://www.nst.com.my/opinion/leaders/2022/08/818575/nst-leader-dropout-degrees
Rhoads, D. D. (2020). Traditional, online or both? A comparative study of university student learningand satisfaction between traditional and hyflex delivery modalities (Publication No. 27995688)[Doctoral dissertation, Concordia University]. ProQuest Dissertations Publishing.
Robert, J. (2022). Modality Preferences. https://www.educause.edu/ecar/researchpublications/2022/students-and-technology-report-rebalancing-the-student-experience/modalitypreferences
Sprent, P. (2011). Fisher exact test. In International encyclopedia of statistical science (pp. 524-525).
Springer, Berlin, Heidelberg.
Verrecchia, P. J., & McGlinchey, M. J. (2021). Teaching during covid: The effectiveness of the HyFlex
classroom in a 300 Level Statistics class. Journal of Education and Training Studies, 9(3), 23-27.

883

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Assessment of Intelligent Teaching Preparationof EFL Teachers: Based on Two-Year Data
Comparison
Xin ANa*, Xi SHENa, Jiannan BAIa & Yushun LIaaschool of Educational Technology, Beijing Normal University, Beijing, China* 17888829110@163.com
Abstract: In order to guide the effective application of artificial intelligence (AI) in education(AIEd), China has set up a number of national AI education demonstration areas (AIEDA).
Teachers in this AIEDA have carried out action research about AIEd for over two years. For
the purpose of assessing teachers’ intelligent teaching preparation during the process, we
conducted consecutive two years of evaluation through questionnaires and AI-based teaching
platform data. This study developed a questionnaire of teachers’ intelligent teaching
preparation, including teachers’ perception of and behavioral intention to AIEd, AI
technological pedagogical and content knowledge (TPACK), and AI ethics knowledge. Besides,
the data obtained from an AI-based teaching platform were analyzed, which indicated teachers’
acceptance and trust towards AIEd. A total of 101 valid samples of English as a foreign language(EFL) teachers participated in data collection for two years. The results showed that teachers’
trust in AIEd increased, but their effort expectancy and perception of facilitating conditions of
AIEd decreased; Teachers’ AI ethics knowledge increased, while their self-report TPACK
decreased.
Keywords: Artificial intelligence, EFL teacher, intelligent teaching preparation, AI education
demonstration area

1. Introduction
Artificial intelligence (AI) is shaping the education sector and has become an important area of researchacross the world (Khan et al., 2021). Various countries such as the United States (National Science andTechnology Council, 2016), the United Kingdom (Department for Science, Innovation and Technology
et al., 2017), France (Government Francais, 2017), and Japan (Headquarters for Japanese Economic
Revitalization, 2016), are deploying AI technologies in education. China has designated AI education
demonstration areas (AIEDA) to explore the use of AI for subject learning (An et al., 2022), andpromote the development of teachers’ preparation of AI in education (AIEd) (The Teachers’ Office ofChina, 2018). Teachers in K-12 schools in the AIEDA have leveraged AI affordances through multiple
AI tools to facilitate personalized learning and immediate feedback since 2020 (An et al., 2022). The
integration of AI into education has presented new challenges to teachers, it is crucial to assess the
preparedness of teachers in AI environment.
However, existing research on the AIEd environment has primarily focused on students, while researchon teachers remains significantly lacking (Liang et al., 2021). Teachers need to be prepared to embrace
the changes brought about by AI and work towards developing their own skills and knowledge to meetthe demands of the modern-day classroom. Assessing the preparedness of teachers in AI environment
can provide them with targeted suggestions for improvement. Therefore, teachers’ intelligent teaching
preparations was studied in this research to fill the research gap. Intelligent teaching refers to theapplication of AI technology and algorithms by teachers to optimize the teaching and learning process
and improve students’ learning outcomes (Celik, 2023; Liu, 2020).
This study aimed to evaluate teachers’ preparation to incorporate AIEd by examining both external andinternal factors (An et al., 2022). The Unified Theory of Acceptance and Use of Technology (UTAUT)(Venkatesh et al., 2003) accounts for the perceptions of technology from usersȽ viewpoint and was澳
therefore used to explore teachersȽ perceptions and acceptance of AIEd as external factors. The

884

intelligent Technological Pedagogical and Content Knowledge (iTPACK) framework (An et al., 2022;
Celik, 2023) was used to assess teachers’ knowledge of AIEd as internal factors. The study conducted
a two-year longitudinal survey of the same group of teachers and collected data from bothquestionnaires and an AI teaching platform to provide a more comprehensive assessment.

2. Conceptual framework
2.1 UTAUTUTAUT is a widely used model for understanding users’ acceptance and usage of new technologies. It
was proposed by integrating the Technology Acceptance Model (TAM) and seven other models. It has
been shown to explain up to 70% of the variance in predicting users’ behavioral intention to use newtechnology (Venkatesh et al., 2003). UTAUT surveys various factors, including performance
expectation, effort expectation, social influence, facilitating condition, users’ behavioral intention, andbehavior. The definitions of these factors are shown in Table 2. Due to its effectiveness, UTAUT has
been widely used in education field (Hsu, 2022). As AI technology is increasingly used in education,
UTAUT could be applied to assess students’ acceptance of AIEd. For instance, An et al. (2022)
developed a questionnaire based on UTAUT, and explored teachers’ behavioral intention to use AIEd
systems and its related factors. However, there is still a lack of annual follow-up surveys on teachers,
making it difficult to demonstrate the changes in teachers’ perceptions and acceptance under the rapid
development of AI technology. In this study, UTAUT was adopted to explore teachers’ perceptions andacceptance of AIEd in two years.

2.2 iTPACK
Technological Pedagogical Content Knowledge (TPACK) is a framework that describes the knowledge
that teachers need to effectively integrate technology into their teaching practices. Developed by Mishra
and Koehler (2006), TPACK comprises three core areas of knowledge: technological knowledge (TK),
pedagogical knowledge (PK), and content knowledge (CK). These three types of knowledge areinterrelated and combine to form Pedagogical Content Knowledge (PCK), Technological Pedagogical
Knowledge (TPK), Technological Content Knowledge (TCK), and TPACK. TPACK provides aframework for professional development, helping teachers to identify areas where they may need tostrengthen their knowledge and skills. By leveraging this framework, teachers can effectively integratetechnology into their teaching practices.
When AI entered education, a series of ethical issues have appeared. It is believed that teachers should
master AI ethics knowledge as well as TPACK. Celik (2023) proposed iTPACK framework, including
teachers’ TPACK and ethics knowledge. Teachers should have a rational understanding of the ethicalissues in AIEd, such as privacy, fairness, and responsibility (Feng et al., 2020). In the process of
integrating AI into teaching, teachers need to ensure that they consciously abide by social moral norms
in information activities, pay attention to privacy and data security, and deal with issues related to
information cocoon, data privacy, AI bias and ethical decision-making in a targeted way (Burton, 2017).
However, previous studies typically only conducted one-time surveys, making it difficult to present theknowledge development of teachers in AI environments. In this study, we would conduct a follow-up
survey on teachers’ knowledge based on iTPACK framework to fill the research gap.

3. Methods
3.1 Participants
The participants of this study are teachers of English as foreign language (EFL) in middle school in
Dongcheng District, one of the AIEDA in China. In Dongcheng District, the Information Office of the
Education Commission, the Education Science Research Institute, universities, and enterprises arecooperating to promote the integration of AI and teaching among middle school EFL teachers. Teachers885conducted action research to explore the integration of AI and EFL subjects in this district. Each teacher
is guided by three interdisciplinary experts, which includes an EFL expert, an educational technology
expert, and a scientific research expert, contributing to the development of intelligent teaching
preparation among teachers.
To assess the development of teachers’ preparation for AIEd, questionnaires and AI teaching platform(E-listening and speaking, https://www.ets100.com/home/perintro.html) data analysis were used in thisstudy. Teachers can assign a variety of exercises, which are automatically fed back by the AI to students,
and the teacher can also give feedback on the platform濁澳 A total of 354 EFL teachers’ AI teaching
platform data was collected on April 30, 2022, including the users’ data in the period of January 1, 2021
to April 30, 2022.The first questionnaire was conducted from November 3, 2021 to November 9, 2021,
and the second questionnaire was conducted from December 5, 2022 to December 14, 2022. A total of
101 EFL teachers who effectively participated in both surveys were selected, while samples with
duplicate options for all scale questions were deleted. The basic information of the respondents is shownin Table 1.
Table 1. Demographic Information of Participants
Demographic
ProfileSex
Year of BirthYear of Starting
Teaching

Classification

Number

Percent (%)

Male
Female1960-19691970-1979
1980-1989
1990-1999
1981-1989
1990-1999
2000-20092010-20192020-2021

2
99
6
36
39
24
4
20
37
30
10

1.98%
98.02%
5.94%
35.64%
38.61%
23.76%
3.96%
19.80%
36.63%
29.70%
9.90%

3.2 Instrument
The questionnaire was developed mainly referred to An et al (2022) and Celik (2023), including
teachers’ perception of and behavioral intention to AIEd based on UATUT, and teachers’ knowledge
preparation of AIEd based on iTPACK framework. Table 2 lists the constructs and item examples of
the questionnaire. Five-point Likert scale was adopted in the questionnaire, with 1 representing strongly
disagree, 2 representing disagree, 3 representing general, 4 representing agree, and 5 representing
strongly agree. Based on the relevant literature and theory development, the scale was reviewed andfilled out by two educational technology experts, one staff member of the Education Commission of
Dongcheng District, two junior high school English teachers, and modified according to the feedback
to ensure the content validity.
Table 2. Structure of Teacher Intelligent Teaching Preparation Assessment Scale
Dimension

SubDimension
Performance
Expectancy
(PE)

UTAUT
EffortExpectancy(EE)

Definitions

Sample item

Performance Expectancy means the degree
to which a teacher believes that using the
AI system will help him or her attain gains
in job performance (Venkatesh et al.,2003).
Effort Expectancy means the degree ofease associated with the use of an AI
system (Venkatesh et al., 2003).

AI can help me
improve the quality ofteaching.

886

I think AI teachingsystems are very
simple.

Facilitating
Conditions(FC)

iTPACK

Facilitating Conditions means the degree
to which a teacher believes that anorganizational and technical infrastructure
exists to support the use of the AI system(Venkatesh et al., 2003).
SocialSocial Influence means the degree to
Influence (SI) which an individual perceives thatimportant others believe he or she should
use the new system (Venkatesh et al.,2003).
Behavioral
Behavioral Intention means the beliefs
Intention (BI)teachers have about the AI using behavior
under consideration (Davis, 1989).
AI language AI-TK means knowledge about AI
technological
technologies in language fields (Schmid etknowledgeal., 2020).
(AI-TK)AI
technologicalpedagogicalknowledge(AI-TPK)
AI
technologicalcontent
knowledge(AI-TCK)AI
technologicalpedagogicalcontent
knowledge(AI-TPACK)AI ethics

When I need to use AI
in teaching, my school
will provide help forme.
My colleagues think I
should use AI to
support teaching..
I intend to use AI inteaching in the future.

I know that speech
recognition
technology can scorepronunciation
accuracy, fluency, andphonological
tone
type.
AI-TPK means knowledge of how I know how to use AI
teaching may be changed as the result of tools to meet students’using AI technologies (An et al., 2022).
differentiated learningneeds.
AI-TCK Knowledge about how AI I know how to use AI
technology and subject content relate to to help students learn
vocabulary.
each other (Schmid et al., 2020).
AI-TPACK means knowledge for teachingwith AI technology which requires anunderstanding of how AI technologies can
support teaching subject matter (An et al.,2022).

I know how to use thestrategyofpersonalized guidance
to improve students’English skills with thehelp of AI.
AI ethics means that teachers can I will pay attention to
consciously abide by social moral norms in the privacy of students'
AI-supported teaching activities, have a data when using AI.
rational understanding of ethics and safetyissues in the process of integrating AI into
education and teaching, pay attention to thethreat of AI to education ethics and safety
and properly deal with it (An et al., 2022).

3.3 Data analysis
Following data collection, the reliability and structural validity of the survey scale were evaluated. Data
analysis includes two parts: Confirmatory Factor Analysis (CFA) and reliability analysis. Reliability
analysis of the scale was conducted with SPSS20.0, while CFA was conducted with Mplus 8.3.
The standards recommended by Hair et al., (2014) were adopted in the CFA. Accordingly, indices ofȤðGI  5RRW0HDQ6TXDUH(UURURI$SSUR[LPDWLRQ 506($   &RPSDUDWLYH)LW,QGH[ &), (> 0.90), and the Tucker-Lewis Index (TLI) (> 0.90) were utilized to assess the model fit. To examine
WKHFRQYHUJHQWYDOLGLW\RIWKHVFDOHIDFWRUORDGLQJV Ȝ ZHUHHPSOR\HGWRFDOFXODWH$YHUDJH9DULDQFH

887

Extracted (AVE) (>0.5) and Construct Reliability (CR) (>0.7). Discriminant validity of the scale wasdetermined by comparing the square root values of AVEs of components with the correlations between
components.
,Q WKH UHOLDELOLW\ DQDO\VLV RI WKH VFDOH &URQEDFK V Į YDOXH ZDV FDOFXODWHG WR DVVHVV WKH LQWHUQDO
consistenF\FRHIILFLHQWV7KH&URQEDFK VĮYDOXHRIWKHHQWLUHVFDOHDQGDOOVXE-scales should be above
0.7 (Fornell & Larcker, 1981).

4. Results
4.1 Validity and Reliability
To verify the structural validity of the scale, CFA was carried out. The model fit indices were assessed,
ZLWKDȤðGIRI  506($RI  &),RI ! DQG7/,RI !
0.90), indicating an acceptable fit for the scale items. Standardized factor loadings ranged from 0.719
to 0.949, all falling within an appropriate range. Additionally, the AVE values exceeded 0.5, and the
CR was higher than 0.7, indicating good convergent validity. Discriminant validity was also confirmed,
with the square root values of AVE for each component exceeding the respective inter-component
correlations. Furthermore, tKHUHOLDELOLW\FRHIILFLHQW&URQEDFK VĮRIVXEGRPDLQVUDQJHGEHWZHHQ
and 0.939, with an overall reliability of 0.952, indicating high internal consistency. In conclusion, thescale has good validity and reliability.

4.2 Acceptance and Trust for AI Teaching Platform
The real teaching behaviors of teachers are analyzed with the help of backend data from AI teaching
platform. The two kinds of backend data (number of assign homework and number of check homework)
represent two commonly used functions for teachers on the platform. The results are shown in Table 3.
In 2022, the average monthly number of teachers using this platform to assign homework has increased
significantly compared with 2021, but the proportion of teachers checking homework is declining. This
indicates that teachers’ acceptance of the AI teaching platform is increasing, and their trust in theautomatic feedback function of the platform is also increasing.
Table 3. The use of an AI teaching platform by EFL teachers in Dongcheng District
Classification
Number of assign
homework (AH) /month
Number of Check
homework (CH)/month
AH/CH (%)

Average in 2021

Average in 2022

Growth Rate

6.29

16.21

157.69%

1.96

2.74

39.97%

31.18

16.94

-45.68%

4.3 UTAUT of AIEd
In order to learn more about the situation of teachers, this study conducted a questionnaire survey.
Samples of scale questions in the questionnaire are normally distributed (range of kurtosis is [-1.441,
1.254], range of skewness is [-0.463, 0.394]), so it is possible to conduct paired sample t-tests for two
years of teacher scale data. The scores of various dimensions and the different test results of UTAUT
of middle school EFL teachers in 2021 and 2022 are shown in Table 4. There was no significant
difference in the three dimensions of teachers' performance expectancy, social influence and behavioral
intention in the two-year survey, while the scores of effort expectancy and facilitating conditions in
2022 were significantly lower than those in 2021.
Table 4. Analysis on the difference of UTAUT scores of middle school EFL teachers in 2021 and 2022

888

ItemsPair 1
Pair 2
Pair 3
Pair 4
Pair 5M

Performance Expectancy 2021
Performance Expectancy 2022
Effort Expectancy 2021
Effort Expectancy 2022
Facilitating Conditions 2021
Facilitating Conditions 2022
Social Influence 2021
Social Influence 2022
Behavioral Intention 2021
Behavioral Intention 2022

4.15
4.12
3.75
3.60
3.74
3.58
3.63
3.58
4.03
4.04

MD

SD

95%
confidence
interval
lower upper

0.02

0.59

-0.09

0.15

0.67

0.15

t

Sig.

0.14

0.41

0.68

0.02

0.28

2.25

0.03

0.69

0.02

0.29

2.24

0.03

0.05

0.71

-0.09

0.19

0.67

0.51

-0.01

0.58

-0.13

0.10

-0.22

0.83

4.4 iTPACK
The specific scores of iTPACK of EFL teachers in 2021 and 2022 are shown in Table 5. There is no
significant change in TK and TCK, but TPK and TPACK are significantly lower in 2022 than in 2021.
In the meanwhile, teachers’ knowledge of AI ethics improved significantly in 2022 compared with
2021.
Table 5. Analysis on the difference of iTPACK scores of middle school EFL teachers in 2021 and 2022

Pair 6
Pair 7
Pair 8Pair 9Pair 10

Items

M

TK2021
TK2022
TPK2021
TPK2022
TCK2021
TCK2022TPACK2021
TPACK2022
AI Ethics2021
AI Ethics2022

3.61
3.50
3.62
3.32
3.70
3.76
3.84
3.62
3.60
4.18

MD

SD

95% confidence
interval
lower
upper

0.11

0.78

-0.04

0.30

0.64

-0.07

t

Sig.

0.27

1.48

0.14

0.17

0.42

4.63

0.00

0.82

-0.23

0.09

-0.82

0.41

0.22

0.71

0.08

0.36

3.11

0.00

0.58

0.63

0.46

0.70

9.40

0.00

5. Discussion
5.1 Teachers' trust in AI has increased, as has their expectation for the development oftechnology environment
According to the teachers' behavior data collected from the AI teaching platform, teachers' trust andacceptance of AI are improving. This is the real behaviors of teachers using AI teaching platforms,
indicating that teachers have felt the reliable help of AI. However, compared with two years ofquestionnaire survey results, it can be seen that teachers' perceptions of effort expectancy andfacilitating conditions are declining. This indicates that with the deep integration of AI and subject
teaching, teachers have diversified and high-level demands in practice, and their expectations andrequirements for AI teaching products are increasing, while the support of intelligent teaching softwareand hardware environment increasingly lags behind teachers' expectations. In the future, we should
continue to promote the improvement of AI EFL teaching products, strengthen the support services oftechnical personnel, and escort the practice of the integration of AI and education for teachers.

889

5.2 Teachers' self-report AI pedagogical knowledge declined, but AI ethics knowledge
improved
From the comparison results of two years, the scores of teachers in AI ethics knowledge increased,
while TPK and TPCK of self-evaluation decreased. This indicates that with the development of expert
training and teacher action research, teachers have a more prudent attitude towards AI ethics issues,
such as the information cocoon, privacy disclosure, decision-making bias, and accountability
difficulties. However, the scores of teachers’ self-report AI technological pedagogical knowledge
declined. It might be because that EFL teachers have realized their shortcomings in AI-assisted
pedagogical knowledge as AI integrates with EFL deeper and deeper. In the follow-up practice, more
attention should be paid to teachers' ability to improve AI-supported teaching, so that they can use AI
to implement more teaching strategies and carry out personalized and precise teaching.

6. Implications
This study showed the current situation and development of teachers' intelligent teaching preparationin AIEDA in China. This study provides evaluation instruments for teachers' intelligent teaching
preparation in the process of AI and subject integration, and the research findings provide a practicalreference for the integration of AI and education in K-12 further. At the teacher level, this study provides
teachers with tools for self-evaluation and self-reflection, points out the direction for their future
professional ability development, and provides a guarantee for their independent and professional action
research.
Due to space limitations, this study does not present the evaluation content of qualitative data such as
interviews and analysis of subject text data. In addition, promoting the development of students is thefundamental purpose of integrating AI technology into education and teaching. This study lacks theevaluation of teachers from the perspective of students, which needs to be improved continuously in the
future.

Acknowledgements
This research was supported by the National Social Science Foundation "Research on Artificial
Intelligence Technical Specifications for Minors" (Grant Number 21&ZD328).

References
An, X., Chai, C. S., Li, Y., Zhou, Y., Shen, X., Zheng, C., Chen, M. (2022). Modeling English teachers’
behavioral intention to use artificial intelligence in middle schools. Education and Information
Technologies.
Burton E., Goldsmith J., Koenig S., Kuipers B., Mattei N., Walsh T. (2017). Ethical Considerations in Artificial
Intelligence Courses. AI Magazine, 22-34.
Celik, I. (2023). Towards Intelligent-TPACK: An empirical study on teachers’ professional knowledge toethically integrate artificial intelligence (AI)-based tools into education. Computers in Human Behavior,138, 107468.
Chun, H. (2020). Searching for the coexistence paradigm of human translator and machine translation: Focusing
on PBL-based translation practicum class with AI machine translation tools. The Journal of Interpretation
and Translation Education, 18(4), 59–96.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information
technology. Mis Quarterly, 13(3), 319–340.
Department for Science, Innovation and Technology, Department for Digital, Culture, Media & Sport,
Department for Business, Energy & Industrial Strategy. (2017). Growing the artificial intelligence industry
in the UK. https://www.gov.uk/government/publications/growing-the-artificial-intelligence-industry-inthe-uk.
Feng, R., Sun, J., Sun, F. (2020). Ethical Risk and Rational Choice of Artificial Intelligence in the Applicationof Education. Journal of Distance Education, 38(3),47-54.

890

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables andmeasurement error. Journal of Marketing Research, 24(2), 337–346. https://doi.org/10.2307/3151312.
Government Francais. (2017). Rapport De Synthèse France Intelligence Artificielle.
https://www.economie.gouv.fr/files/files/PDF/2017/Rapport_synthese_France_IA_.pdf.
Hair, J. F., Black, B., Babin, B. J., & Anderson, R. (2014). Multivariate data analysis. Prentice Hall.
Headquarters for Japanese economic revitalization. (2016). Japan Revitalization Strategy 2016-As We Move
towards the Fourth Industrial Revolution.
https://www.kantei.go.jp/jp/singi/keizaisaisei/pdf/2016_zentaihombun.pdf.
Hsu, L. (2022). EFL learners’ self-determination and acceptance of LMOOCs: the UTAUT model. Computer
Assisted Language Learning.
Khan, F., Jan, M.A., Rehman, A.U., Mastorakis, S., Alazab, M., Watter, P.. (2021). A secured and intelligent
communication scheme for IIoT-enabled pervasive edge computing. IEEE Trans Ind, 17, 5128–5137.
Liang, J. C., Hwang, G. J., Chen, M. R. A., & Darmawansah, D. (2021). Roles and research foci of artificial
intelligence in language education: an integrated bibliographic analysis and systematic review approach.
Interactive Learning Environments.
Liu Bin (2020). Exploring the Intelligent Education Literacy of Teachers in the Age of Artificial Intelligence.
Modern Educational Technology, 11, 12-18
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher
knowledge. Teachers College Record, 108(6), 1017-1054.
National Science and Technology Council. (2016). The National Artificial Intelligence Research andDevelopment Strategic Plan. https://www.nitrd.gov/PUBS/national_ai_rd_strategic_plan.pdf.
Schmid, M., Brianza, E., & Petko, D. (2020). Developing a short assessment instrument for Technological
Pedagogical Content Knowledge (TPACK.xs) and comparing the factor structure of an integrative and atransformative model. Computers & Education, 157(103967).
The Teachers’ Office of China. (2018). Notice of the Ministry of Education of China on carrying out the pilot
work of the Action to boost the construction of teachers by artificial intelligence.
http://www.moe.gov.cn/srcsite/A10/s7034/201808/t20180815_345323.html.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology:
Toward a unified view. Mis Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540.
Yu, S., Wang, Q. (2019). Analysis of Collaborative Path Development of “Al+Teachers”. e-Education
Research, 40(4), 14-22,29.

891

" !   !" " "   #" !
#" !"$  #" ! #"                   
"$& &! !$&!   !' &!  !!%
 ($%&&!"' $" ;/@1 03/@2A:3GC>4 32C   '45? <-<1> <>1?1:@? @C; /-?1 ?@A051? >18-@10 @; 01B18;<5:3 10A/-@5;:-8
@1/4:;8;351? <-5>10 C5@4 @1-/41> <>;21??5;:-8 01B18;<91:@ $ /;A>?1? @; ?A<<;>@<>-/@5/10>5B1: >1?1->/4 -9;:3 @1-/41>? '41 /-?1 ?@A051? 191>310 2>;9
9A8@505?/5<85:->E <>;61/@? 5:B;8B5:3 ?/4;;8@1-/41>? -:0 >1?1->/41>? '41 25>?@ <>;61/@
?A<<;>@?@1-/41>A?1;21B501:/1.-?10@1-/45:3?@>-@1351?@4-@-853:C5@4@41?/51:/1
;2 81->:5:3 ?A/4 -? >1@>51B-8 <>-/@5/1 -:0 05?@>5.A@10 <>-/@5/1 '41 ?1/;:0 <>;61/@
?A<<;>@?@1-/41>?5:418<5:3?@A01:@?81->:@;9-:-31?@>1??A?5:31B501:/1.-?10
?182>13A8-@5;:@1/4:5=A1??A/4-?95:02A8:1??-:0/;3:5@5B1>1-<<>-5?-8'41<>;61/@?
2A?1 @;31@41> /;:/1<@? ;2 @1-/41> 5:=A5>E 81->:5:3 01?53: ;:85:1 /;99A:5@51? ;2
@1-/41>?-:0;<1:10A/-@5;:-8>1?;A>/1?@;?A<<;>@@1-/41><>-/@5/10>5B1:>1?1->/4: ;B1>B51C ;2 @1-/41>31:1>-@10 >1?1->/4 -:0 59<>1??5;:? ;2 @41 $ /;A>?1?
3-@41>100A>5:3<58;@5:3->1<>1?1:@10-8;:3C5@401?/>5<@5;:?;2@41@C;?A<<;>@5:3
@1/4:;8;351?9<85/-@5;:?2;><>-/@5/1->1-8?;?4->10 
 '1-/41> 5:=A5>E '1-/41> <>;21??5;:-8 01B18;<91:@ $>-/@5/10>5B1:
>1?1->/4 1->:5:301?53:#:85:1/;99A:5@51?;2@1-/41>? %#!&%! '@/1B7132@7D3<@3A3/@167AMC<23@B/93<7<=@23@B=5/7<<3E9<=E:3253>/@B:G0G;3/<A=4
>@/1B713/<2B63=CB1=;3A=4B6/B>@/1B713N/<2G  <D=:D7<5B3/163@A7<AC16@3A3/@16
7A / E/G 7< E6716 M/ AB@=<5 /<2 1=63@3<B 0=2G =4 >@=43AA7=</: 9<=E:3253 =E<32 0G B63
B3/167<5 >@=43AA7=<N )16:37163@ >   1/< 03 0C7:B   B3/163@ >@=43AA7=</:
23D3:=>;3<B'/>>@=/16B6/B/:75<AE7B6>@/1B7132@7D3<@3A3/@167AB3/163@7<?C7@G* 
7<B=ABC23<B:3/@<7<5E67167<D=:D3AB3/163@A3F>:=@7<5B63CA3=42/B/B=/<AE3@?C3AB7=<A
B6/B3;3@534@=;B3/167<5>@/1B713A$716=A3B/:  <<=D/B7D3 32C1/B7=< B316<=:=573A /@3 E3:: >=A7B7=<32 B= 7<1@3/A3 B3/163@=>>=@BC<7B73A /<2 /07:7B73A B= >/@B717>/B3 7< >@/1B7132@7D3< @3A3/@16  #3/@<7<5 23A75< #
B==:A 4=@ 7<AB/<13 AC>>=@B B3/163@A 7< B63 >@=13AA =4 1=<13>BC/:7H7<5 /CB6=@7<5 /<2
2=1C;3<B7<5 :3/@<7<5 B/A9A -/AA=<  "7@A16<3@  # B==:A 1/< 03 3;032232 7<B=
=<:7<31=;;C<7B73A=4B3/163@A&>:/B4=@;AB6/BAC>>=@B7<B3@/1B7=<A/;=<5B3/163@AE63<
1=1@3/B7<5A6/@7<53F>:=@7<5/<21=;;3<B7<5=<:3/@<7<523A75<A7<>@=13AA3A=4;CBC/:
7<A>7@/B7=< 3@<I<23H#3= 3B /:   *63 A6/@32 :3/@<7<5 23A75<A @3>@3A3<B =>3<
32C1/B7=</: @3A=C@13A &( B6/B 1/< 03 4@33:G @3CA32 /<2 @347<32 0G B63 32C1/B7=</:
1=;;C<7BG  *63A3 B316<=:=573A 4/17:7B/B3 1=::31B7D3 9<=E:3253 0C7:27<5 /<2 *  B63@30G
/::=E7<5B3/163@AB=3F>/<2B637@>@/1B713A$716=A3@<I<23H#3= -67:3B63@3/@3/ <C;03@ =4 3F7AB7<5 # B316<=:=573A 4=1CA7<5 =< 27443@3<B >32/5=571/: />>@=/163A /<2
43/BC@3A ;=AB 2= <=B 7<1=@>=@/B3 B63 @3?C7@32 43/BC@3A B= AC>>=@B >@/1B7132@7D3< @3A3/@16
/2D/<17<53D723<130/A32B3/167<5=>>=@BC<7B73A AGAB3;B6/B2=3A7<B35@/B3=>B7=<A4=@
B63@3?C7@3243/BC@3A7AB63 <B35@/B32#3/@<7<53A75<<D7@=<;3<B #  #6/A033<CA32 E7B6 =<:7<3 1=C@A3A AC>>=@B7<5 ' B= 7;>@=D3 B3/163@AO A97::A E63< 23A75<7<5 4=@
B316<=:=5G3<6/<132 :3/@<7<5 /@@3B/=;7<5= 3B /:   -7B6 / 63756B3<32 4=1CA =<
4/17:7B/B7<5*  >@=13AA3A B= /2D/<13 >@/1B7132@7D3<@3A3/@167<B=3D723<130/A32>@/1B713A
/;=<5 B3/163@A BE= >@=831BA E3@3 C<23@B/93<  *63A3 >@=831BA  3F>:=@3 B63 7<B35@/B7=< =4

892

7<<=D/B7D3 32C1/B7=</: />>@=/163A /<2 B316<=:=573A AC16 /A *  # & /<2 &(  *63
4=::=E7<5A31B7=<A>@3A3<B1/A3ABC273A=4B63BE=>@=831BA  $ '  $  #!  %! 
M*3/163@A =4B3< CA3 7<AB@C1B7=</: >@/1B713A 9<=E< B= 03 E@=<5 7 3  ;/AA7<5 @/B63@ B6/<
7<B3@:3/D7<5 3F/;>:3A B= 3F>:/7< / B=>71N (=32753@  'G1 >     -7B6 7<1@3/A32
/D/7:/07:7BG=4<3EB316<=:=573A&&(B=2=1C;3<B/<2A6/@3>@/1B713AB63@37A/@7A9=4
27AA3;7</B7<57<34431B7D3>@/1B713A 3<13/;C:B727A17>:7</@G>@=831B ::C;7<3E/A7<7B7/B32
B=AC>>=@BB3/163@CA3=43D723<130/A32B3/167<5AB@/B3573A/<22757B/:B316<=:=573A  #!$$!  '!" % !&#$ $ AC@D3GABC2G1=<2C1B32/BB63=<A3B=4B63>@=831B4=C<2B6/BB3/163@;=B7D/B7=<7A67563@
4=@:3/@<7<5B=CA33D723<130/A32B3/167<5AB@/B3573A/<22757B/:B316<=:=573A4=@B3/167<5/<2:3/@<7<5B6/<4=@* 4=@'3/@2A:3G:0J *=/:75<E7B6B63A347<27<5AB63'
1=C@A3E/A23A75<32B=4=1CA=<7<B@=2C17<5B3/163@AB=3D723<130/A32B3/167<5AB@/B3573A
/<22757B/:B316<=:=573AB=AC>>=@BB637@CA3=4B63A3AB@/B3573A #/<2* E3@3>@3A3<B32/A
B63>@=13AA3AB6@=C56E6716B3/163@AE=C:2;/93CA3=4/<2@34:31BC>=<B637@CA3=4B63
AB@/B3573A  6=C@1=C@A33:3D3< 6=C@E=@9A6=>A/<26=C@A=4/AG<16@=<=CAE=@9
E/A>@3>/@32 *6347@AB E=@9A6=>A7<B@=2C132A173<13=4:3/@<7<5/<2:3/@<7<523A75<B63=@G -=@9A6=>A/:B3@</B3203BE33<7<B@=2C17<5B63=@G/<2@3A3/@16ABC273A@3:/B32B=B3/167<5
AB@/B3573A/<2>@=D727<5B7;34=@B3/163@AB=1=23A75<7;>:3;3<B/B7=<A *6347</:E=@9A6=>A
7<D=:D32A6/@7<5@3AC:BA/<2@34:31B7<5=<B63=D3@/::1=C@A3  #   $  !!(%!!#% #' $#
-7B6 / 4=1CA =< 7<B@=2C17<5 AB@/B3573A /<2 A1/44=:27<5 *  B63 B316<=:=5G
6BB>A 7::C;7<3 C>4 32C 7A1=<13>BC/:7H32/A/<&4=@B3/163@AB=:3/@</0=CB/<2>@/1B713
B637@CA3=43D723<130/A32B3/167<5AB@/B3573AD7/B631@3/B7=<=4B3/163@53<3@/B32@3A3/@16
:3AA=<A @3A3/@16:3AA=<7<B67A1=<B3FB1/<03D73E32/A/A7;>:747327<B3@>@3B/B7=<=4/!/>/<3A3 :3AA=< ABC2G @3A3/@16 :3AA=< */9/6/A67  $1=C5/:
 7< B6/B B3/163@A
1=<2C1B B63 :3AA=< E7B6 B63 >C@>=A3 =4 :3/@<7<5 /0=CB B637@ >@/1B713A  (3A3/@16 :3AA=<
1@3/B7=<7AA1/44=:232D7//AB3>B3/163@:327<?C7@G23A75<B3;>:/B3 * 271B/B3AB63A3:31B7=</<2A3?C3<13=4AB3>AE7B6#>@=D727<5B634C<1B7=</:7BG4=@/227<5:3/@<7<5/1B7D7B73AA6/@7<5/1B7D7BG@3A=C@13A/<2037<5/0:3B=1=>G/2/>B/<2@3CA3<=B=<:GB63:3/@<7<5;/B3@7/:A
0CBB63=D3@/::@3A3/@16:3AA=< *63@3CA/0:3@3A3/@16:3AA=<A/<2:3/@<7<5/1B7D7BG;/B3@7/:
E7B67<B63;/@3&(1=<B@70CB32B=B6332C1/B7=</:1=;;C<7BG/AA6=E<7<75C@3 '$# ::C;7<3@3A3/@16:3AA=<A((3;7F32=>3<32C1/B7=</:@3A=C@13A


893



::7</::B63B316<=:=5GA1/44=:2A>@/1B7132@7D3<@3A3/@160GB3/163@A=<B63CA3=43D723<13
0/A32B3/167<5AB@/B3573AB6@=C567BAB3/163@:327<?C7@G23A75<B3;>:/B3B6/B5C723AB3/163@A
B6@=C561@3/B7<57;>:3;3<B7<5@34:31B7<5C>=</<2A6/@7<5B637@@3A3/@16:3AA=<A/A&(  $ %& %$ !&%%#$$  % #!  %!  :/19 =4 AB@3AA ;/</53;3<B A97::A 1=;07<32 E7B6 =<5=7<5 AB@3AA=@A @3:/B32 B= /1/23;71
23;/<2A1/<<35/B7D3:G7;>/1BABC23<B/1/23;71>3@4=@;/<13;3<B/:63/:B6/<2E3::037<5
'/A1=33B/:  3<13/;C:B727A17>:7</@G>@=831B(3;7FE/A7<7B7/B32B=AC>>=@B
B3/163@ CA3 =4 :3AA=< ;/B3@7/: &( 7<B@=2C17<5 3D723<130/A32 AB@3AA ;/</53;3<B
B316<7?C3AAC16/A;7<24C:<3AA/<21=5<7B7D3@3/>>@/7A/:  #!$$!  '!" % !&#$ $ (3AC:BA=4/AC@D3GABC2G1=<2C1B32/BB63=<A3B=4B63>@=831B7<271/B3B6/BB3/163@A6/D3/
AB@=<57<B3@3AB7<:3/@<7<5/0=CBB63B=>71=4AB@3AA:0J3/@2A:3G
0CBB637@AB@3AA;7<2A3BA2347<32/AB637@>3@13>B7=<A=4B6334431BA/<2@=:3=4AB@3AA@C;3B/: /@3=D3@:G <35/B7D3 /<2 >@3A3<B /< =0AB/1:3 B= =D3@1=;3  2/>B7<5 B= B63A3 7<A756BA B63 '
1=C@A3 E/A 23A75<32 A= B6/B >/@B717>/<BA 1=;>:3B32 23;=<AB@/B7=< :3AA=<A =< AB@3AA;7<2A3BA/<2A3:4@35C:/B7=<B316<7?C3A/A/E/GB=3<AC@3>/@B717>/<BA:3/@<32/0=CB>=A7B7D3
AB@3AA;7<2A3BA *63A3:3AA=<A>@=D7232/<7<7B7/:3F>3@73<13>/@B717>/<BA1=C:20C7:2C>=<
E63</2/>B7<5/227B7=</:&(4=@B637@ABC23<BA #/<2* E3@37<B@=2C132/A>@=13AA3A
B6@=C56E6716B3/163@AE3@3B=>:/</<2@34:31BC>=<B637@CA3=4&(  6=C@1=C@A3A7F
6=C@E=@9A6=>A/<26=C@A=4/AG<16@=<=CAE=@9A6=E<7<*/0:3 E/A>@3>/@32 */0:3 $  !'$%!  (  %$'&!  #'% %
&1??5;:
& 
& 
& 

&
&
&

(117








';<5/
1@@5:3?@->@10D<1/@-@5;:?019;:?@>-@5;:81??;:?
$>1<->-@5;::@>;0A/@5;:@;81->:5:301?53:/;:/1<@?
$8-::5:3%1?1->/481??;:?4->5:32110.-/7
A@;:;9;A?C;>7;:0A/@5:3>1?1->/481??;:?C5@4?@A01:@?
&1:?19-75:3%1?1->/481??;:>1281/@5;:?
$A.85?45:3$>1?1:@5:3C;>7#%<A.85/-@5;:
1??;:?81->:10%1281/@5:3;:@41;B1>-88/;A>?1

 #   $  !!(%!!#% #' $#-7B6/4=1CA=<5C727<5B3/163@AB=63:>B637@ABC23<BA:3/@<B=;/</53AB@3AACA7<53D723<13
0/A32A3:4@35C:/B7=<B316<7?C3AB63B316<=:=5G6BB>A @3;7F32 C>4 32C 7A1=<13>BC/:7H32
/A/<&4=@B3/163@AB=27A1=D3@/2/>B/<2A6/@3&(@3:/B32B=AB@3AA;/</53;3<BA33
75C@3   & 7A 3;>6/A7H32 B6@=C56 B63 1=;;C<7BG 7<B3@4/13 B6/B 4/17:7B/B3A 7<B3@/1B7=<A
/;=<51=;;C<7BG;3;03@A #7A/BB6313<B3@=4B63B316<=:=5G7<0=B6B631@3/B7=<A6/@7<5/<2@3CA3=4:3/@<7<5/1B7D7B73A/<2:3AA=<A *6@=C56B63#7<B3@4/13AB3/163@A/@3/0:3B=
C>:=/2 47:3A B= A6/@3 2=1C;3<B B637@ /2/>B/B7=<A /<2 >@=D723 3F/;>:3 CA3 1/A3A  * @34:31B7=<>@=;>BAE=@9E7B67<B63/1B7D7BG/<2:3AA=<1@3/B7=<7<B3@4/13AB=A1/44=:2B3/163@
B3AB7<5/<2@3>=@B7<5=<B63CA/53=4B63@3A=C@13AA6/@32E7B6B631=;;C<7BG ::7</::B63
B316<=:=5G AC>>=@BA >@/1B7132@7D3< @3A3/@16 0G B3/163@A =< B63 @3CA3 =4 &( @3:/B32 B=
AB@3AA ;/</53;3<B 4=@ ABC23<BA B6@=C56 B63 7<B35@/B7=< =4 *  @34:31B7=< >@=;>BA 7<B= B63
/1B7D7BG/<2:3AA=<1@3/B7=<7<B3@4/13A *63A3>@=;>BA03<347B0=B6B631@3/B=@A=4@3A=C@13A
B6@=C56/727<5B637@>:/<<7<5/<2@34:31B7<5=<B637@B3/167<5>@/1B713A/<2B631=;;C<7BG0G
2=1C;3<B7<5/<2A6/@7<53F/;>:3A4@=;B63/CB63<B71CA3=4B63@3A=C@13A 894#% #' $#!%$ "%! $ $# $  23A75<0/A32 @3A3/@16 />>@=/16 =:3 3B /:  E/A 4=::=E32 7< E6716 B63 >@=831BAO
B316<=:=573A /<2 ' 1=C@A3A E3@3 7B3@/B7D3:G 23D3:=>32  < 0=B6 >@=831BA AC@D3G @3A3/@16
ABC273A E3@3 1=<2C1B32 B= 7<4=@; B63 7<7B7/: 23A75< =4 B63 ' 1=C@A3 /<2 B316<=:=573A
3/@2A:3G:0J
:0J3/@2A:3G
 '1=C@A3AE3@3B63<>7:=B327<;C:B7>:3
C@=>3/< 1=C<B@73A /A3  )>/7< '=@BC5/: )3@07/ AB=<7/ /A3  )>/7< 3<;/@9G>@CA  <B63A31=C@A3A>/@B717>/<BAE=@9327<A;/::5@=C>A B=B3/163@AB=23A75</<2
7;>:3;3<B@3A3/@16:3AA=<A/A/4=@;=41=::/0=@/B7D3* @=;B63>7:=BA/227B7=</:2/B/E/A
1=::31B324@=;1=C@A3>/@B717>/<BAB6@=C56>@3/<2>=AB1=C@A3AC@D3GA@34:31B7=</1B7D7B73A
4=1CA 5@=C>A /<2 /@B74/1BA >@=2C132 7< B63 1=C@A3 7 3  @3A3/@16 :3AA=<A   C<7D3@A7BG
@3A3/@163B671A1=;;7BB33/>>@=D32B63@3A3/@167<0=B6>@=831BA  % !%!  ($$*67A>/>3@>@3A3<BA2/B/@3:/B32B=B63/@B74/1BA@3A3/@16:3AA=<A@34:31B7=</1B7D7B73A/<2
4=1CA 5@=C>A 4@=; >7:=BA @C< 7< )>/7<  )31=<2/@G /<2 D=1/B7=</: 32C1/B7=< B3/163@A>/@B717>/B327<B631=C@A3A/A3  B3/163@A/A3  B3/163@A  <0=B61/A3AB3/163@A
2=1C;3<B32 B637@ @3A3/@16 :3AA=<A CA7<5 / *  B3;>:/B3  *63 2/B/ 4@=; B63A3 B3;>:/B3A 7A
AC;;/@7H32 7< B63 47<27<5A  227B7=</::G 0=B6 1/A3A 6/2 B3/163@A @34:31B =< B63
7;>:3;3<B/B7=<=4B631=C@A3  </A3 /</AG<16@=<=CA@34:31B7=</1B7D7BGE/A@C</BB63
;72>=7<B=4B631=C@A3/<2/AG<16@=<=CA@34:31B7=</1B7D7BGE/A@C<7<B6347</:E=@9A6=> =@
/A3 /<=<:7<34=1CA5@=C>CA7<5/1=<D3@A/B7=</:;3B6=23D7::/@23B/: E/A63:2
E7B6 A7F B3/163@A /B B63 ;72>=7<B =4 B63 1=C@A3 /<2 / AG<16@=<=CA @34:31B7=< /1B7D7BG E/A
1=<2C1B327<B6347</:E=@9A6=> *63;3AE3@3723<B747324@=;B63@34:31B7=</1B7D7BG/@B74/1BA
B@/<A1@7>BA4@=;B634=1CA5@=C>A/<2=0A3@D/B7=</:<=B3A4@=;B6347</:E=@9A6=>ACA7<5/
1CBB7<5/<2A=@B7<5/>>@=/16(G/<3@</@2    $ # #%$#$$! $*/0:3>@=D723A/AC;;/@G=4@3A3/@16:3AA=<A@C< /A3 6/24=C@@3A3/@16:3AA=<A *6@33
E3@303BE33<5@=C>1=;>/@7A=<A7<E6716/B@/27B7=</::3AA=<7 3 6=EB63B3/163@AE=C:2
@35C:/@:GB3/16AC16/1:/AAE/A1=;>/@32E7B6/:3AA=<CA7<5/<3D723<130/A32AB@/B35G *634=C@B6@3A3/@16:3AA=</2=>B32/E7B67<5@=C>ABC2G/>>@=/16CA7<5>@3>=AB;3/AC@3A */0:3 '$)!%$%%! %$!')$%
"A9.1>;2>1?1->/481??;:?
&@>-@1351?1D<8;>10
-@-/;881/@5;:

&@A01:@?<->@5/5<-@5:3

-?1 &<-/1081->:5:3.>-5:.>1-7?
>12>-95:3 >1@>51B-8 <>-/@5/1
-:005?@>5.A@10<>-/@5/1
#:85:1
@;;8?
;;381
8-??>;;9 ;;381 ;>9?
-4;;@<-<1>.-?10
-??53:91:@? -:0 @1-/41>
;.?1>B-@5;:?
G 

-?1 %12>-95:3 ?=A->1 .>1-@45:3
<>;3>1??5B19A?/A8->>18-D-@5;:
95:02A8:1??
#:85:1 @;;8? $-081@ 0$AFF81
8-??!;;0<<;;381;>9?
;;381 -9.;->0? ;;381
8-??>;;9 -4;;@ <-<1>
.-?10-??53:91:@?-:0@1-/41>
;.?1>B-@5;:?
G

@=;/A3 A7F@3A3/@16:3AA=<AE3@3@C< ::4=::=E32/E7B67<5@=C>ABC2G/>>@=/16 ::
B63A3:3AA=<A7<B@=2C132B63=@G=<B63A173<13=4AB@3AA/AE3::/AA3:4@35C:/B7=<B316<7?C3A ,723=A/<2>@3A3<B/B7=<A:723AE3@3B63;=AB4@3?C3<B:GCA32&( $=AB:3AA=<ACA32>@3

895

>=ABAC@D3GA@3:/B32B=AB@3AA;7<2A3BA@C;3B/: /A/<3D/:C/B7=<;3/AC@3/<2/:A=
1=::31B322/B/=<ABC23<B7;>@3AA7=<A=4B63:3AA=<A    $ #%! $
6/::3<53A;3<B7=<320G>/@B717>/<BAB6/B4CBC@37B3@/B7=<A=4B63'1=C@A3A<332B=/22@3AA/@3AC;;/@7H327<*/0:3  <@34:31B7<5=<B631=C@A3A13<B3@32=<@3A3/@16:3AA=<A/A3
 >/@B717>/<BA A6/@32 6=E B63G 4=C<2 B63 AB@/B3573A AC7B/0:3 /<2 34431B7D3 B= CA3 E7B6 B637@
ABC23<BA0/A32=<;3/AC@3A=4ABC23<B>3@4=@;/<13/<2 =@16/<53A7<1:/AA@==;1:7;/B3  <
E@7B7<5 /0=CB E6/B B63G E=C:2 16/<53 7< B637@ 4CBC@3 @3A3/@16 :3AA=<A >/@B717>/<BA E=C:2
/2=>B/;=@3AGAB3;/B71/<2@75=@=CA/>>:71/B7=<=4B63/1B7D7B73A=D3@B7;37<1:C23ABC23<B
43320/19=<B637@7;>@3AA7=<A=4B63/1B7D7B73A/<2 16/<536=EB63G7;>:3;3<B32A>317471
AB@/B3573AAC16/A57D7<57;;327/B343320/19=<@3B@73D/:>@/1B713/1B7D7B73A &D3@/::B63CA3
=4AB@/B3573A7<@3A3/@16:3AA=<A53<3@/B327<A756B4C:@34:31B7=<A/;=<5>/@B717>/<BAA33;7<5:G
/::=E7<5B63;B=1=<B@/ABB637@3F>3@73<13E7B6B63B@/7<7<5/<2B63>@=D7232;/B3@7/:A 
/A3 >/@B717>/<BA<=B3203<347BAB=B63@3A3/@16:3AA=<AAC16/A67563@;=B7D/B7=<
03BB3@ >3@4=@;/<13 /<2 /< 7;>@=D32 1:/AA@==; 1:7;/B3  *63G 6756:756B32 16/<53A 7< B63
>3@13>B7=<A /<2 /BB7BC23A B=E/@2A AB@3AA /;=<5 ABC23<BA  -63< @34:31B7<5 /0=CB
7;>@=D3;3<BA /<2 4CBC@3 7;>:3;3<B/B7=<A >/@B717>/<BA 6756:756B32 B63 7;>=@B/<13 =4
3F>=C<27<5B63A173<B74713D723<130367<2B63>@/1B713A/<26/D7<51=::3/5C3AA6/@3B637@
>=A7B7D3 3F>3@73<13A 5/7<32 4@=; CA7<5 B63 @3A=C@13A  < AC; 6/D7<5 B3/163@A >3@4=@;
@3A3/@16 /1B7D7B73A B= <=B =<:G >:/< B637@ :3AA=<A 0CB /:A= @34:31B =< :3AA=< =CB1=;3A 7< /
1=::/0=@/B7D3;/<<3@E/A>=A7B7D3:G@3137D32 */0:3   % &)$&"& $%'4191?
(;>78;-0-:0@591
<>1??A>1

&/4;;8/A8@A>1

-?1 -B5:3 @; 81->: @41;>E />1-@1 -:0
59<8191:@>1?1->/481??;:?C-?@;;
9A/4C;>78?;@591@;/;88-.;>-@1
C-?:;@9-01-B-58-.81C5@45:@415>
/A>>1:@@1-/45:3?/410A81?
$11>C;>7C-?:;@<->@;2@41?/4;;8
/A8@A>1

-?1 8;@;2@591:11010@;.15:B1?@10@;
01?53:-:0@41:59<8191:@?1??5;:?
8?; @41 /;A>?1 C-? /;3:5@5B18E
019-:05:3-?-8;@;25:2;>9-@5;:

 /A8@A>1 ;2 ?4->5:3 5: C45/4 @591 5?
-B-58-.81 @; -:-8EF1 -:0 05?/A??
9-@1>5-8? -:0 1D<1>51:/1? C-? :;@E1@1?@-.85?4105:@41?/4;;8
>5035:3@41;>E-:0 @ C-? - /4-881:31 @; @>-:?8-@1 "<>-/@5/1
>1?1->/4 ?@A051? ;: @1-/45:3
?@>-@1351?5:@;<>-/@5/1

 "%! $!##%<B631=<B3FB=4>@/1B7132@7D3<@3A3/@16'1=C@A3>/@B717>/<BAE3@3/0:3B=:3/@</0=CB
3D723<130/A32B3/167<51=;3C>E7B67<?C7@G?C3AB7=<A@3:/B32B=B637<B@=2C132AB@/B3573A
/<2 ;/B3@7/:A 23A75< :3/@<7<5 /1B7D7B73A B= 3F>:=@3 B637@ ?C3AB7=<A 1=::31B 2/B/ B6@=C56
7;>:3;3<B7<5B63/1B7D7B73AE7B6ABC23<BA@34:31B=<B637@3F>3@73<13A/<2>@=2C13:3AA=<A
:3/@<32  </A3<A3B63GE3@3/0:3B=1=<2C1B>@/1B7132@7D3<@3A3/@16/<253<3@/B3=CB>CBA
B6/B 1/< 03 CA34C: B= B63 32C1/B7=</: 1=;;C<7BG L /A 1/A3 ABC273A =4 B637@ 3F>3@73<13A %=<3B63:3AAB63@3/@3AB7::5/>AB=1:=A3034=@3AC16E=@91/<;/935@3/B3@1=<B@70CB7=<AB=
>@=43AA7=</:9<=E:3253=E<320GB63B3/167<5>@=43AA7=< )7;7:/@B==0AB/1:3A723<B74732B6/B
6/D3A:=E32B63/2=>B7=<=4!/>/<3A3#3AA=<)BC2G=CBA723=4!/>/<6=9A673@</<23H
E3=0A3@D32B6/B;/<G>/@B717>/<BAAB@C55:32B=4@33C>3<=C56B7;3B=/23?C/B3:G
3<5/53 7< 7<?C7@G >@=13AA3A /<2 1=::/0=@/B3 E7B6 >33@A :/1932 >@=132C@/: @3A3/@16
9<=E:3253/<22/B/:7B3@/1GB6/B/4431B32B63?C/:7BG=42/B/1=::31B32/<2B637@7<B3@>@3B/B7=<A
=4 B63 2/B/ /<2 <33232 B= CA3 B63 @3A=C@13A 7 3  AB@/B3573A /<2 ;/B3@7/:A E7B6 ABC23<BA;=@3B6/<=<3B7;3B=031=;3>@=47173<B7<CA7<5B63;1=;4=@B/0:G4=@@3A3/@16:3AA=<A 896

< B3@;A =4 7;>:71/B7=<A 4=@ >@/1B713 /<2 >=:71G 7<B3@>@3B/B7=<A =4 B63A3 BE= 1/A3A
AC553ABB6/B1=::/0=@/B7D3* 3D723<130/A32B3/167<5/<22757B/:B316<=:=573A/@3B63;3A
B6/B1/<03CA32B=3<5/53B3/163@A7<1=<B7<C7<5' .3BB=6/D3B3/163@A1=<B@70CB3B=/>@=43AA7=</:9<=E:32530/A35@=C<2327<3D723<13B6@=C56>@/1B7132@7D3<@3A3/@16344=@BA/@3<33232B==D3@1=;35/>A7<>32/5=571/:/<27<?C7@G>@/1B713A3 5 >@=132C@/:@3A3/@16
9<=E:3253 /<2 2/B/ :7B3@/1G /<2 B= :3AA3< >3@137D32 0C@23<A E=@9:=/2 B7;3 >@3AAC@3
A16==: 1C:BC@3  &<5=7<5 ' B6/B AC>>=@BA B3/163@ 23D3:=>;3<B =4 0=B6 >32/5=571/: /<2
7<?C7@G >@/1B713A 7< >/@/::3: 7A :793:G B= >:/G / 1@7B71/: @=:3 7< 3AB/0:7A67<5 /<2 ;/7<B/7<7<5
B3/163@=E<32 >@=43AA7=</: 9<=E:3253  C@B63@ 7<<=D/B7D3 B316<=:=573A B6/B 4=AB3@ =<:7<3
1=;;C<7B73A=4B3/163@AA1/44=:2#/<2* >@=13AA3A>@=D723@3CA/0:3@3A3/@16:3AA=<A
&(LAC16/AB633F/;>:3A>@3A3<B327<B67A>/>3@L6/D3B63>=B3<B7/:B=AC>>=@B/<2
ACAB/7<B3/163@'B6/BB/@53BA0=B6>@/1B713A  !( %$*63/CB6=@AE=C:2:793B=B6/<9 ::C;7<3/<2(3;7F>@=831B;3;03@A/<2>/@B717>/<BA4=@
B637@1=<B@70CB7=<A *67AE=@96/A033<>/@B7/::G4C<2320G@/A;CA ) "   ) ")  3>/@B;3<B=4(3A3/@16/<2+<7D3@A7B73A=4
B63 =D3@<;3<B =4 /B/:=<7/ )(  /<2 $  %    
' ( # $
8.K 1->0?81E!
'1-/41><>;21??5;:-801B18;<91:@@;418<?@A01:@?9-:-31?@>1??
.->>51>?-:01:-.81>?*!-:A?/>5<@?A.95@@102;><A.85/-@5;:+1->0?81E!8.K 
'1-/41>!;@5B-@5;:';C->0$>;21??5;:-81B18;<91:@;88;C5:3
91>31:/E%19;@1'1-/45:3D<1>51:/1?:1/$&&%*.(2/'3)& 3).3&1."3*/.",/.'&1&.$&
/./-043&1400/13&%/,,"#/1"3*5&&"1.*.(  00 :@1>:-@5;:-8&;/51@E
;2@41 1->:5:3&/51:/1?
4;7?45&1>:-:01F
4-881:31?@;59<;>@5:3-<-:1?181??;:?@A0E;:/1>:?
95?/;:/1<@5;:?-:0:A-:/1?)*&,3""00".   
;81%$A>-;&%;??5!&15:!
15:3<>;-/@5B1C41>1-/@5;:>1?1->/4911@?
01?53:>1?1->/4  1/$&&%*.(2 
>A9&-8;B1E$/4;>&
%1@45:75:3?@>1??@41>;81;295:0?1@?5:01@1>95:5:3@41
?@>1??>1?<;:?1/41.",/'&12/.",*36".%/$*",26$)/,/(6  
1B588->01?>;/41?!>-:FJ!A0-:K$-F;?N
04.3&2-&3/%/,8(*$/22/#1&,"
$/.5&12"$*8.&.&,31"#"+/&3./(17'*$/
->>1@-;95:3;!&8;1<$1>:H:01F 1;!;>)
 1->:5:301?53:2;>@1-/41>
<>;21??5;:-801B18;<91:@.3&1."3*/.",/41.",/'%4$"3*/.",&$)./,/(6*.*()&1%4$"3*/.   
1>:H:01F 1;?1:?5;$I>1F1>:@8!$;FF54-/K:$>51@; $$1>?5/;
:5:@13>-@101:B5>;:91:@2;>81->:5:301?53:1/.3*&12*.
!5/4;?1>:H:01F 1;8.K '1-/41>8105:=A5>E5:@1/4:;8;3E?A<<;>@10?/4;;8
/;99A:5@51?1*3*2)/41.",/'%4$"3*/.",&$)./,/(6   !5/4;?1>:H:01F 1;
/;881/@5B15:=A5>E2>-91C;>7@;?@A0E-:0?A<<;>@@1-/41>?-?01?53:1>?5:@1/4:;8;35/-81:B5>;:91:@?/-043&12%4$"3*/. $-?/;1!1@>5/7&$->71>
'4159<-/@;2?@>1??;:?@A01:@?5:?1/;:0->E
?/4;;8-:045341>10A/-@5;:.3&1."3*/.",/41.",/'%/,&2$&.$&".% /43)   %;10531> $E/!
:1D<1:?5B1@1/4:5=A1?@;59<>;B110A/-@5;:<<8E5:3
/;3:5@5B1<?E/4;8;3E@;1:4-:/110A/-@5;:-8<>-/@5/1/41.",/'00,*&%&2&"1$)*.&-/16".%/(.*3*/.   %E-:(1>:->0%
'1/4:5=A1?@;501:@52E@4191?5:=A-85@-@5B10-@-*&,%
&3)/%2   &/4815/41>'1-/45:3-:0 1->:5:3:@1>:-@5;:-8&A>B1E' &:?534@?-:0:@1><>1@-@5;:?

4@@< CCC;1/0;>3 10A/-@5;: @-85? ' &
,5:?534@?,-:0,5:@1><>1@-@5;:?<02'-7-4-?45!/;A3-8';88-.;>-@5B181??;:>1?1->/4!-D595F5:3@4159<-/@;2
81??;:?@A0E!   897

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Impact of Preschool Teacher Training
on STEAM Education: Professional
Preparedness and Confidence
Wan-Chen CHANG
Department of Early Childhood Education, National Tsing Hua University, Taiwan*wc_chang@mx.nthu.edu.tw
Abstract: This study aims to explore the impact of teacher training in the field of
STEAM education on preschool teachers' preparedness and confidence. The
participants of this study were nine education and childcare professionals from a publicpreschool in northern Taiwan. A small-scale workshop was conducted to train theteachers, covering topics such as an overview of STEAM education, STEAM learningfor preschoolers, teaching methods, and curriculum design. The results revealed thatprior to the training, the teachers had a high awareness of the importance and expected
outcomes of STEAM education. However, they scored relatively low in terms ofteaching experience and self-efficacy. After the training, the teachers showedsignificant improvement in their knowledge of STEAM teaching methods andenvironmental support. Nevertheless, their ratings for confidence and teaching efficacy
remained relatively low. These findings highlight the need for further professional
development and support for teachers in the field of STEAM education. The results ofthis study provide valuable insights and recommendations for teacher training andinstructional practices in preschool settings.
Keywords: preschool teacher training, STEAM education, confidence, teachingefficacy, curriculum design

1. Introduction
Under the impetus of policies and educational trends, both parents and teachers are highly
supportive of implementing STEAM education at the preschool stage. However, adults needadditional knowledge and support to facilitate the execution of STEAM education (e.g.,
Margot & Kettler, 2019; McClure et al., 2017; Simoncini & Lasen, 2018). In particular,teachers play a crucial role in STEAM education at the preschool level (Karademir et al.,2020; Simoncini & Lasen, 2018; Monkeviciene et al., 2020; Wahyuningsih et al., 2020). Thisis because teachers can only successfully incorporate it into the curriculum when they fully
understand the components of STEAM education (Pang & Good, 2000). However, tosuccessfully integrate STEAM education into the curriculum, teachers need to grasp
additional knowledge and skills. Therefore, it is vital to provide training and support forpreschool teachers' professional development in STEAM education. This study aims tounderstand preschool teachers' cognition and beliefs about STEAM education, theirprofessional needs, and challenges in STEAM education, and to explore how to provide
effective professional development training and support in STEAM education. The goal is toenhance preschool teachers' professional capabilities and confidence in STEAM education,
thereby promoting children's learning and development.

2. Research Method
2.1 Research participants

898

The participants of this study are nine teachers from five classes in a public kindergarten innorthern Taiwan. The kindergarten, which adopts a thematic teaching curriculum model,
selected the themes for its courses after receiving STEAM education training.

2.2 Research Instruments
2.2.1 STEAM Teaching Questionnaire for Teachers
This study will develop a questionnaire on teachers' beliefs about STEAM teaching based on
previous research. Pre-tests will be administered when the teachers participate in the firstphase, and post-tests will be administered after the second phase concludes. The
questionnaire structure includes five major dimensions: the importance of STEAM education,teaching methods for implementing STEAM education, support for implementing STEAM
education, self-efficacy, and expected outcomes. The questionnaire uses a seven-point
scale, with respondents selecting an option from a continuum ranging from "strongly agree"
to "strongly disagree" (7 6 5 4 3 2 1).
2.3.2 Mid-term Progress Review of Teachers' STEAM Teaching
To understand how teachers are implementing the STEAM curriculum during the semester,
the researchers will have discussions and interviews with the teachers from each classduring the semester to understand the challenges they face while implementing thecurriculum.

3. Preliminary Data Analysis
This study analyzed the data from the teachers before the training, and the results areshown as below. Looking at the means, teachers have a positive attitude in all fivedimensions. Particularly, teachers believe in the importance of STEAM teaching (M = 5.65,
SD = 1.09) and have a higher average score for the expected outcomes of STEAM teaching(M = 5.58, SD = 1.06), indicating that teachers have a more positive attitude towards thelearning outcomes of STEAM teaching for children. The average score for teachingexperience is relatively low (M = 4.23, SD = 1.31), indicating that teachers' previous
experience with STEAM teaching is quite lacking, even including a lack of experience inscience inquiry teaching.
Further analysis on the importance of STEAM activities indicates that teachers
believe effective STEAM education is crucial for enhancing children's hands-on capabilities
and problem-solving skills (M = 6.19, SD = 0.71), as well as stimulating their curiosity andinterest in everyday phenomena (M = 6.10, SD = 0.79). On the other hand, they do not place
as much emphasis on enhancing children's acquisition and understanding of STEAM
knowledge (M = 4.67, SD = 0.92).
In response to open-ended questions, the majority of teachers (85%) believe that theessence of STEAM education lies in using inquiry-based problems to help children develop
relevant skills, such as learning by doing, logical thinking, problem identification, problemsolving, and collaborative abilities. Regarding potential future challenges, most teachers(94%) are concerned about their lack of STEAM teaching expertise, particularly in preparing
teaching methods. This result aligns with the findings from the questionnaire.
After the implementation of the thematic courses, a mid-term review revealed thatteachers still lack confidence in their teaching (M = 3.94, SD = 1.27). The teachers' efficacy
in STEAM teaching (M = 4.29 , SD = 1.20) and encouragement for children to use scientificinquiry methods in STEAM teaching (M = 5.04, SD = 1.35) were also examined. This showsthat, compared to before the training, teachers are now able to utilize more scientific inquiry
methods, but there has been no significant difference in their teaching efficacy. Therefore, itis necessary for university professors to provide more explicit guidance to give themconcrete goals for STEAM teaching.

899

4. Conclusion and Discussion
The results of this study indicate that teachers have high expectations for the implementationof STEAM education in early education stages, and they highly value the development ofchildren's abilities. After the training courses, teachers utilized more child-centered teaching
methods in STEAM courses, but their self-efficacy in teaching remained insufficient. In
addition, teachers moderately utilize exploratory methods and dialogue to foster thecultivation of higher-order thinking skills, as well as reflection and self-reflection abilities(Monkeviciene et al., 2020). Therefore, professional development in STEAM domain
knowledge and teaching methods are necessary for teachers in STEAM education.

Acknowledgements
The authors would like to thank National Science and Technology Council of the Republic ofChina, Taiwan, for financial support (NSTC 111-2410-H-007-063 - )References
Margot, K. C., & Kettler, T. (2019). Teachers’ perception of STEM integration and education:
A systematic literature review. International Journal of STEM education, 6(1), 1-16.
McClure, E. R., Guernsey, L., Clements, D. H., Bales, S. N., Nichols, J., Kendall-Taylor, N., &Levine, M. H. (2017). STEM starts early: Grounding science, technology, engineering,and math education in early childhood. New York: The Joan Ganz Cooney Center at
Sesame Workshop.
Simoncini, K., & Lasen, M. (2018). Ideas about STEM among Australian early childhood
professionals: How important is STEM in early childhood education?. International
Journal of Early Childhood, 50(3), 353-369.
Karademir, A., Kartal, A., & Turk, C. (2020). Science education activities in Turkey: A
qualitative comparison study in preschool classrooms. Early Childhood Education Journal,
48(3), 285-304.
Monkeviciene, O., Autukeviciene, B., Kaminskiene, L., & Monkevicius, J. (2020). Impact ofinnovative STEAM education practices on teacher professional development and 3-6 year
old children’s competence development. Journal of Social Studies Education
Research, 11(4), 1-27.
Wahyuningsih, S., Nurjanah, N. E., Rasmani, U. E. E., Hafidah, R., Pudyaningtyas, A. R., &Syamsuddin, M. M. (2020). STEAM learning in early childhood education: A literature
review. International Journal of Pedagogy and Teacher Education, 4(1), 33-44.
Pang, J., & Good, R. (2000). A review of the integration of science and mathematics:
Implications for further research. School Science and Mathematics, 100(2), 73–82.

900" !   !" " "   #" !
#" !"$  #" ! #"                #   "   

#&%!!! !#%$*!%! '#$%*" &%!  %  %#%!  $%%&%!# &%! !*$# " 
 %#!#!"&%  %&$*!%! '#$%*" 
I1=1E389 D19C59 GCD ;I?D? E 13 :@        9 ?34> ;,;0= B0 ;=:;:>0/ 49?0=A09?4:9 80>>,20> ?3,? .:9>4/0= ?30.:8;70C4?D :1 >?@/09? ;0=>:9,74?40> 0.,@>0 :1 ?30 49.=0,>492 48;:=?,9.0 :1
49/4A4/@,77D :;?484E0/ 70,=9492 ?30 49?0=A09?4:9 1:= ?30 >?@/09? 4> ,7>: /0>4=0/ ?: -0
?,47:=0/*3470;=0A4:@>>?@/40>3,A01:@9/?30,;;=:;=4,?080>>,20.:9?09?=07,?0/?::90:=>0A0=,7;0=>:9,74?D?=,4?>9:>?@/D3,>-009.:9/@.?0/?:>09/49/4A4/@,77D
:;?484E0/80>>,20>B4?3.:9>4/0=,?4:9:10,.3>?@/09?H>.:8;70C;0=>:9,74?D?=,4?>9
?34>>?@/DB0.:9/@.?0/?30.7@>?0=492:1>?@/09?>-,>0/:9424A0?0>?>.:=0>,9/
?309/0?0=8490/?309,?@=0:10,.3.7@>?0=,9/?30?D;0:149?0=A09?4:980>>,20=07,?0/?:?3,?9,?@=0..:=/492?:?30.7@>?0=492,9,7D>4>?30>?@/09?>B0=0.,?02:=4E0/49?:
?3=00 2=:@;> :9>4/0=492 ?30 ;0=>:9,74?D ?=,4? :1 ?30>0 2=:@;> B0 /0.4/0/ ?,47:=0/
80>>,20>1:=0,.32=:@;  !0>>,20 49?0=A09?4:9 -42 14A0 49A09?:=D 49/4A4/@,77D :;?484E0/ 70,=9492
;0=>:9,74?D.7,>>414.,?4:9  ! 
+85 EC5 ?6 D85 6<1=9>7 9>D5BF5>D9?> 9C D85 9>DB?4E3D9?> ?6 >E475C 9> 54E31D9?>1< CDE495C
1=711B4&95<C5> 1>49D31>251>56653D9F5G1I?69>49F94E1<<I?@D9=9J54=5CC175
9>D5BF5>D9?> *?=5CDE495CEC9>7>E4759>D5BF5>D9?>C9>C9CD?>D859=@B?F5=5>D?6CDE45>DCM
@5B6?B=1>35%?DJ5D1<   'M3?>>5<<$1>7  -89<5@B5F9?ECCDE495CC8?GD81D
6<1=9>79>D5BF5>D9?>31>251@@<954D?9>49F94E1<<I?@D9=9J54=5CC1759>D5BF5>D9?>-1<<5D1<  .1=1E3895D1<  EC9>7979F5!>F5>D?BI"?8>5D1<   D85C5CDE495C6?3EC
?> ?><I ?>5 ?B C5F5B1< DI@5C ?6 @5BC?>1<9DI DB19DC 1>4 D85I 494 >?D @B?@?C5 D85 C@539693
=5CC175DI@56?B51383<ECD5B 
5>35 D85 19= ?6 ?EB B5C51B38 9C D? CE775CD 1> 1@@B?138 6?B 945>D96I9>7 56653D9F5
9>D5BF5>D9?>=5CC175C3?>C945B9>7CDE45>DCM3?=@<5H@5BC?>1<9DIDB19DC -5DBID?9>F5CD971D5
D856?<<?G9>7B5C51B38AE5CD9?> (%#%&$%#$!!")"#$! %*%*"$!$!!$%& %$ (%$%!"%+$$!#!%!$&$%#$  !"#  

(5BC?>1<9DI9>F5>D?B95C1B513?==?>1@@B?1386?B5F1<E1D9>7@5BC?>1<9D95C9>@CI38?<?7I +85I9>F?<F51C5B95C?6AE5CD9?>>19B5C45C97>54D?E>3?F5B1CE2:53DMC@5BC?>1<9DI '>5
G5<< ;>?G>9>F5>D?BI9CD85979F5!>F5>D?BI"?8>5D1<   G893831D57?B9J5C
@5BC?>1<9D95C9>D?69F57B?E@C +89C=5D8?49C8978<I@?@E<1B1>4B5C51B385BC81F55F5>
45F5<?@54<9CDC?61@@B?@B91D514:53D9F5C6?B513831D57?BI?<425B7  ?6CD55

901

)114 "?8>C?>'CD5>4?B6  +85B5<1D9?>C89@C25DG55>5138979F5DB19D1>4
14:53D9F5C1B5B5@B5C5>D549>D856?<<?G9>79D5=CG85B5=51>CD85@B?69<5D8?C5G8?
81F5D858978979F5C3?B51B5<9;5<ID?81F51>4 =51>CD85?>5CD8?C5G8?81F5D85
<?G979F5C3?B51B5<9;5<ID?81F5
 '@5>>5CCD?5H@5B95>359>F5>D9F53EB9?EC3?>C9CD5>D 31ED9?EC  HDB1F5BC9?>13D9F5D1<;1D9F5C8I AE95D  7B5512<5>5CC;9>43??@5B1D9F5C5<69C8 3?<4  ?>C395>D9?EC>5CCB5C@?>C92<5?B71>9J545HDB1F171>D 31B5<5CC  &5EB?D939C=1>H9?EC1>7BI31<= E>5=?D9?>1< !>?EBB5C51B38G5G9<<EC5D85"1@1>5C5979F5!>F5>D?BID?45D5B=9>55138CDE45>DMC
@5BC?>1<9DI      

!>54E31D9?>CDE495CD85EC5?6>E475C81CB5359F54=E381DD5>D9?>1=711B4&95<C5>
   &E475 9C 4569>54 1C K1<D5B/9>70 @5?@<5MC 2581F9?BC 9> 1 @B5493D12<5 G1I G9D8?ED
6?B29449>71>I?@D9?>C?BC97>96931>D<I381>79>7D859B53?>?=939>35>D9F5CLG89389CD85D5B=
?6 53?>?=93C +81<5B  *E>CD59>     CDE4I CE775CD54 D81D CDE45>DC 4? >?D =9>4
B5359F9>7>E475C=?B5?6D5>96D85I@5B359F5D85>E475CD?25EC56E<D?D85=1D1B55D1<    +859>DB?4E3D9?>?6>E475C9>54E31D9?>CDE495C9C31<<5416B1=9>79>D5BF5>D9?> F5>C=1<<381>75C9>D856B1=9>7?69>6?B=1D9?>81F5D85@?D5>D91<D?1<D5B2581F9?B1>45<9=9>1D5
291C5C4E5D?3?7>9D9F51>41DD5>D9?>1<<9=9D1D9?>C1=711B4&95<C5>  !>D89CCDE4I
G5DBID?EC56<1=9>79>D5BF5>D9?>6?B1>9>49F94E1<<I?@D9=9J54CICD5= !D9C@?CC92<5D?3B51D5F1B9?ECDI@5C?6>E475CD81D3?>C945BD85CDE45>DMC@5BC?>1<9DI
G8938 =1I 81F5 F1BI9>7 9=@13DC ?> 8?G CDE45>DC 1@@B?138 D859B AE9JJ5C  '>5 CDE4I
45D5B=9>54 ?> 1> 9>49F94E1< 21C9C G8938 =5CC175C 1B5 =?CD 1@@B?@B91D5 21C54 ?> @B?69<5
381B13D5B9CD93C.1=1E3895D1<   33?B49>7D?D89CCDE4ID85B55H9CD541>?D5G?BD8I
9>F5BC51CC?391D9?>25DG55>D85B1D5?6B5C@?>49>7D?>?D96931D9?>C1>43?>C395>D9?EC>5CC6?B
@55B>?D96931D9?>C 449D9?>1<<ID85B5G1C<9DD<561F?B12<53?BB5<1D9?>25DG55>D856B5AE5>3I?6
>?D96931D9?>B5C@?>C5C1>43?>C395>D9?EC>5CC6?B4514<9>5>?D96931D9?>C %?B5?F5B>5EB?D939C=
G1C3?BB5<1D54G9D8@55B>?D96931D9?>CG89<5?@5>>5CCD?5H@5B95>35G1C3?BB5<1D54G9D8@55B
>?D96931D9?>C1CG5<< 
<D8?E78C?=5B5C51B385CC8?GD8556653D9F5>5CC?69>D5BF5>D9?>B5<1D54D??>5DI@5?6?BC5F5B1<DI@5C?6@5BC?>1<9DI-1<<5D1<  .1=1E3895D1<  >?CDE4IC8?GC
D85 1@@<931D9?> ?6 9>49F94E1<<I ?@D9=9J54 3?>D5>D =5CC175 3?>C945B9>7 5138 CDE45>DMC 1<<
3?=@<5H @5BC?>1<9DI  !D 9C 496693E<D D? 453945 G81D DI@5 ?6 =5CC175 C8?E<4 25 C5>D D? 5138
CDE45>D2531EC55F5BICDE45>D81C13?=@<5H@5BC?>1<9DI !>D89CCDE4IG5381<<5>754D85
3<ECD5B9>7?6D85CDE45>DC9>D5B=C?6D859B@5BC?>1<9DI1>4=14514539C9?>?6C5>49>7G81D
DI@5?6=5CC1751>4D85>1@@<I9DD?CDE45>DC  
+89CCDE4IG1C3?>4E3D549>18978C38??<9>"1@1>  CD 7B145CDE45>DC  I51BC?<4@1BD939@1D54 9> D89C 5H@5B9=5>D  +? 5CD9=1D5 CDE45>DCM @5BC?>1<9D95C G5 EC54 1 "1@1>5C5
F5BC9?>?61AE5CD9?>>19B5CEBF5IG89383?>C9CDC?6 AE5CD9?>C12?EDD859B@5BC?>1<DB19DC
%EB1;1=9%EB1;1=9  +89C31>2585<@6E<G85>C?=5?>5D1;5CD85CEBF5ID?D8?C5
G8? 31>>?D E>45BCD1>4 D85 ?B979>1< >7<9C8 AE5CD9?>>19B5  +85 97 9F5 1CC5CC=5>D
3?=@B9C5C AE5CD9?>C?ED?6G8938 1B5@B5C5>D541CDBE5 61<C5AE5CD9?>CB5<1D9>7D??>5
?6D8569F5@5BC?>1<9DIDB19DC 138CDE45>DMC@5BC?>1<9DIC3?B59C45D5B=9>542I1449>7E@D85
C3?B5C 6?B 5138 9D5= ?> 1  @?9>D C31<5 2I D85 CDE45>DMC 1>CG5BC  -5 3?<<53D54 97 9F5
@5BC?>1<9DI41D16B?=  CDE45>DC?61<<9>D89C5H@5B9=5>D -5 45C97>54 1>4 45F5<?@54 1 =5CC175 C5>49>7 CICD5= 31<<54 K$)*%5CC175L
G8938 3?=29>5C $51B>9>7 )53?B4 *D?B5 $)* $51B>9>7 %1>175=5>D *ICD5= $%* 1>4
>1<IC9C+??< 97EB5 45C3B925CD85381BD?6D?G8?=D85CICD5=C5>4C1=5CC1751>48?G
D?C5>4=5CC175CD?D855>F9B?>=5>D -5C3854E<5D85=5CC175D?;5>D?CD?B5D85=5CC175

902

<?7C9>D?$)*2I@EC89>7D853853;2?H5C1>41C@5396932EDD?>9>>1<IC9C+??<CEBB?E>454
2I1B54CAE1B59>97EB5  '>D85?D85B81>4G5EC54$)*%5CC175D?C5DC?=55=19<C
G9D8C5>49>73?>D5>DCD9=9>7C1>441D5C '>35D85D9=581C3?=5$)*%5CC17579F5C$)*
D85AE5BID?453945D?G8?=D85CICD5=C5>4C1C3854E<54=5CC175 $)*%5CC1751<C?
75DCD85B5CE<D?6D85AE5BI1>4D85>9DC5>4C1=5CC175D?$%*133?B49>7D?D85B5CE<D?6D85
AE5BI -531>1<C?C5>41=5CC175=1>E1<<IG9D8>1<IC9C+??<C?G5EC549DD?C5>41@1BD
?6D85=5CC175 
&# %5CC175 C5>49>76<?GEC9>7$)*%5CC175

531EC5?6D85=1>IF1B912<5C?6@5BC?>1<9DIDB19DCG531>MD51C9<I453945G9D8C?=5
3?=@B585>C9F5BE<5CG81DDI@5?63?>D5>DC?6=5CC175CD85CDE45>DB5359F5C +85B56?B5G5
3?>4E3D54 D85 3<ECD5B9>7 G9D8 97 9F5 D5CD C3?B5C ?6 69F5 @5BC?>1<9DI DB19DC 1C 5H@<1>1D?BI
F1B912<5C1>4D85>G54539454D85@B?@5B=5CC1753?>D5>DC?651383<ECD5B9>CD514?6=1;9>7
BE<5CD81D31>1CC97>@B?@5BDID?5138CDE45>D 1C54?>D85B5CE<D?6D85; =51>C5<2?G@<?D?6D81D3<ECD5B9>7G56?E>4D81D3<ECD5BCC55=D8525CD>E=25B?63<ECD5BC2531EC59D9CD85
@?9>D1DG8938D8512C?<ED5F1<E5?6D85381>759>C<?@59C7B51D5CD (B5F9?ECCDE495C6?<<?G
D89CB5CE<DD81DG531>3<1CC96ID85CDE45>DC9>D?3<ECD5BCC5>4?B@6F1>;5> 
)?29>C5D1<  +1;1=95D1<  -1<<5D1<    

1C54?>D85B5CE<D?6D855<2?G@<?DG53<ECD5B54  CDE45>DC9>D?D8B553<ECD5BC +12<5 
C8?GC D85 3<ECD5B >1=5 D85 >E=25B ?6 @5?@<5 9> D85 3<ECD5B D85 C3?B5 =51>C 1>4 * ?6
CDE45>DCM @5BC?>1<9DI DB19DC ?6 5138 3<ECD5B 1>4 D85 7B1@8C ?6 D85 49CDB92ED9?> ?6 CDE45>DCM
@5BC?>1<9DI DB19DC  33?B49>7 D? D85C5 B5CE<DC G5 >1=54 D85C5 3<ECD5BC G8?C5 >1=5C
B5@B5C5>DD85@B?@5BDI?651383<ECD5B +8569BCD3<ECD5B81C1785B D81> 1F5B175C3?B56?B
1<<!@5BC?>1<9DIDB19DCC?G5>1=54D89C3<ECD5BK 9L +85C53?>43<ECD5B81C189785B
D81> 1F5B175C3?B56?B7B5512<5>5CC1>4HDB1F5BC9?>1>41<?G5B D81> 1F5B175C3?B56?B
5EB?D939C= -5>1=54D89C3<ECD5BK>L21C54?>D85381B13D5B9CD93C3?B5 +85D89B43<ECD5B
81C 1 89785B D81> 1F5B175 5EB?D939C= C3?B5 1>4 81C <?G5B D81> 1F5B175 @5>>5CC D?
5H@5B95>35 ?B HDB1F5BC9?> C3?B5  33?B49>7 D? D85 651DEB5C G5 69>1<<I >1=54 D85 3<ECD5B
K&?5L  -5 1<C?6?E>4 6B?= &', 1>1<IC9C ?6 D85 41D1 D81D 1<< ?6 D85 97 9F5 @5BC?>1<9DI
C3?B5C?6?>5C@5396933<ECD5B1B5C97>96931>D<I49665B5>D6B?=D85?D85B?>5C +89CB5CE<D61CD5>C
D85F1<949DI?6D851CC97>=5>D?6D85@B?@5BD95C?651383<ECD5B DD85C53?>4CD5@G51CC97>54G8938=5CC175D?C5>4D?51387B?E@?6D853<ECD5B *DE45>DC9>D85 97B?E@81F5D8589785CD1F5B175C3?B5?6D85D8B553<ECD5BC '>5@B5F9?EC
B5C51B38C8?GCD81D9D9C25DD5B6?BCDE45>DCM5>7175=5>DD?C5>44514<9>5 DI@5=5CC175D?
@5?@<5G8?81F518978C3?B5.1=1E3895D1<   +8ECG5D89>;9D9C25CDD?C5>41
4514<9>5 DI@5=5CC175CD?D85= *DE45>DC9>D85&?57B?E@81F5D85<?G5CD1F5B175C3?B5
?6D85D8B553<ECD5BC '>5CDE4I1<C?C8?GCD81D9D9C25DD5B6?BCDE45>DCM5>7175=5>DD?C5>4
@55B DI@5=5CC175CD?@5?@<5G8?81F51<?GC3?B5.1=1E3895D1<   +8ECG5
3?>3<E454D81D9D9C25CDD?C5>4@55B DI@5=5CC175CD?D85= *DE45>DC9>D85>7B?E@81F5
D85 <?G5CD & C3?B5 ?6 1<< D8B55 7B?E@C  *DE45>DC G8? 81F5 1 <?G & C3?B5 1B5 E>5=?D9?>1<903

 ?6CD551>4)114 1>431<="?8>C?>'CD5>4?B6  +8ECG54539454D?C5>4
=5CC175CG893881F5D853?>D5>D?>@B9?B13D9?>C?685BC5<6 89=C5<6 ?>C945B9>7D85@5BC?>1<9DIDB19D?65138=5CC175DI@5G54539454G81D3?>D5>DC?6
D85=5CC1751B5D8525CDD?C5>49>D89C5>7175=5>D3?>D5HD +856?<<?G9>7C5>D5>35C1B5
5H1=@<5C?6D853?>D5>D 514<9>5K.?E81F5>?DG?B;54?>D85B53?==5>41D9?>CICD5=?BD5CDC5D=1;5B9>D85
<1CD41IC +855H1=G9<<2579>?>52BE1BI +855H1=41D59C?><I 41IC1G1I !B53?==5>41D9?>G9<<85<@I?E<51B>=?B55669395>D<I L
 (55BK.?E81F5>?DG?B;54?>D85B53?==5>41D9?>CICD5=?BD5CDC5D=1;5B9>D85<1CD41IC EB9>7D85@5B9?4D85B53?==5>41D9?>CICD5=?BD5CDC5D=1;5BG1CEC542I
@5?@<5 1>4  5H5B39C5C G5B5 C?<F54  ! B53?==5>41D9?> G9<< 85<@ I?E <51B> =?B5
5669395>D<I L ?==9D=5>DK.?E81F5>?DG?B;54?>D85B53?==5>41D9?>CICD5=?BD5CDC5D=1;5B9>
D85<1CD41IC .?E81F5C?<F54?>5B53?==5>41D9?>5H5B39C5C?61B .?E>554D?C?<F5
=?B55H5B39C5CD?EC5+5CD*5D%1;5B1>46EBD85B9=@B?F5I?EB13145=93@5B6?B=1>35 !B53?==5>41D9?>G9<<85<@I?E<51B>=?B55669395>D<I L
+12<5 $$%! "#!"#%*
+I@5?6@5?@<5
9

&?5
 
>
 

%51>**3?B5?65138@5BC?>1<9DIDB19D

&
'


 
 
 
 
 
     
 
 
 
 
 
      
 
 
 
 
 
       !  
!> ?B45B D? @B?F945 1@@B?@B91D5 =5CC175 9>D5BF5>D9?>C D81D D1;5 9>D? 133?E>D 3?=@<5H@5BC?>1<9D95C3<ECD5B9>7G1C@5B6?B=54?>D85CDE45>DC1>4D85381B13D5B9CD93C?651383<ECD5B
G5B5<??;541DD?45D5B=9>51@@B?@B91D59>D5BF5>D9?>=5CC175C +85B5CE<DCC8?G54D81DD85
25CD 1@@B?138 G1C D? 49F945 D85 CDE45>DC 9>D? D8B55 3<ECD5BC 1>4 G5 1<C? 6?E>4 D81D D85
1@@B?@B91D59>D5BF5>D9?>=5CC175DI@56?B51389CK514<9>5LK(55BL?BK?==9D=5>DL21C54
?> @B5F9?EC <9D5B1DEB5 -1<< 5D 1<    .1=1E389 5D 1<     ?<<?G9>7 >?D5C 1B5 D85
651DEB5C?651383<ECD5BB?E@K 9L+89C7B?E@81C189785BC3?B5D81>D85?D85BC1>4@B5F9?ECCDE4IC8?GC
@5?@<5G9D889785BC3?B5D5>4CD?25@E>3DE1<13;5D1<   '>5CDE4I81CC8?G>D81DD85=?B54514<9>5CD85B51B5D8525DD5BCDE45>DCD5>4D?@5B6?B=)129>'M?>?78E5  
B?=D81D@?9>D?6F95G 9CDE45>DC31>25CEC35@D92<5D?14514<9>5 DI@5=5CC175 
B?E@K&?5L+85?D85B7B?E@81C189785B&C3?B51>4<?G5BC3?B5D81>D81D?6D85
?D85BC 1>4 @B5F9?EC CDE4I C8?GC D81D @5?@<5 G9D8 D89C @5BC?>1<9DI 1B5 ?25I9>7 D8?C5 9>
1ED8?B9DI1>4D85I651BC?391<49C1@@B?F5=5>D-1<<5D1<   (B5F9?ECCDE495C1BB5<<5D
1<    (1@1I 5D 1<     )?75BC  5<<5B   81F5 C8?G> D81D 9>D5BF5>D9?> G8938
5>3?EB175C @55B D? @55B 3?<<12?B1D9?> 31> 25 56653D9F5 9> @B?=?D9>7 1 C5>C5 ?6 C?391<25<?>79>75CD12<9C89>7C?391<>?B=C6?BCDB9F9>71>49=@B?F9>7C;9<<CD8B?E78@1BD>5BC89@C 
B?E@ K>L +89C 7B?E@ 81C 1 89785B  C3?B5 1>4 1 <?G5B & C3?B5 D81> D81D ?6 D85
?D85BC1>4@B5F9?ECCDE495CC8?G54D81D@5?@<5G9D8189785BC3?B51>41<?G5B&C3?B5
1B5<9;5<ID?=19>D19>3?>C9CD5>D25<956CD85I1B5?G>9>791<49>9-1<<5D1<   +85I
1B5=?B5CEC35@D92<5D?3?==9D=5>D@5BCE1C9?>CDB1D57I +8?C5G8?81F5CDB?>7C?391<C;9<<C
1B5=?B5<9;5<ID?259>6<E5>354D?@5B6?B=12581F9?B969D1<97>CG9D8D859B5H9CD9>725<956C?B
@B9?B13D9?>C91<49>9-1<<5D1<   +8ECG5C5<53D54D?C5>413?==9D=5>D DI@5=5CC175D?D85= 904

'>5<9=9D1D9?>?6D855H@5B9=5>D1<B5CE<D9CD81D9D9C>?D>535CC1B9<IDBE5D81D5F5BI?>5
31>141@DD853<ECD5BMCBE<5 ?B5H1=@<5G531>C556B?=+12<5 D81DD85B59C1CDE45>DG8?
81C1 @?9>DC3?B55F5>9> 9 ?BD85C?<ED9?>?6D85C5@B?2<5=CG53?E<4EC5=?B5
9>4931D?BCD?5F1<E1D5CDE45>DCM@5BC?>1<9DI<9;51B;+B914+B19DC"?>1C?>-52CD5B
?B+I@5 5>?<<5D   ! ! !!> D89C @1@5B G5 CE775CD54 9>D5BF5>D9?> =5CC175C 3?>C945B9>7 5138 CDE45>DMC 3?=@<5H@5BC?>1<9DI1>4@B5C5>D54D538>9AE5C6?BF9CE1<9J9>7D85?ED3?=5CEC9>7B5<5F1>D9>4931D?BC5HDB13D54  9BCD CDE45>D @5BC?>1<9D95C G5B5 9>F5CD971D54 EC9>7 D85 97 9F5 D5CD 1>4 D85>
CDE45>DC G5B5 3<ECD5B54 EC9>7 9D  +85 3<ECD5B9>7 B5CE<DC 1>4 @B5F9?EC @1@5BC G5B5 D85>
3?>CE<D54 D? 45D5B=9>5 D85 >1DEB5 ?6 5138 3<ECD5B 1>4 D85 DI@5 ?6 9>D5BF5>D9?> =5CC175
141@D54D?D81D>1DEB5 +853<ECD5B9>7B5CE<DCC8?G54D81DD85CDE45>DCG5B549F94549>D?D8B55
7B?E@C1>4G53?>3<E454D81DD8525CD=5CC175C6?B5138?6D85C57B?E@CG5B5K514<9>5L
K(55BL1>4K?==9D=5>DLB5C@53D9F5<I (B5F9?ECG?B;C?6<9D5B1DEB56?<<?GD85B5CE<DC?6D89C
3<ECD5B9>71>4D85F1<949DI?6D851@@B?@B91D59>D5BF5>D9?>=5CC175C -5 1<C? 3?>4E3D54 1> 9>D5BF5>D9?> 5H@5B9=5>D G9D8 D85 C1=5 3?8?BD ?6 CDE45>DC D?
E>45BCD1>4D8556653DC?6=1D38541>4B1>4?==5CC179>7  ?G5F5B4E5D?D85C=1<<>E=25B
?6CDE45>D5>7175=5>D<?7CD85B5CE<D9C>?D3?>3<EC9F51>4G?E<4B5AE9B56EBD85BCDE4I # +89CB5C51B38G1CCE@@?BD542I"*(*"(  "(#  &'*@5391<!>>?F1D9?>
(B?7B1=?>!1>4971D1"(&(  &'"(&(
>09/:=;1   :=609,@ $ #>?09/:=1   ),9 609 !    ,=A492 ;0=>:9,74?D
/0>.=4;?4:9,?4?>5:49?>:914=8,?4:9:1?3=00=0;74.,-70;0=>:9,74?D;=:?:?D;0>1:=-:?3.347/=09
,9/,/@7?>)&$%#$)&#!$&'$#! (,   >09/:=;1A,9609! &0>47409?:A0=.:9?=:770/,9//0.:9?=:770/;0=>:9,74?D;=:?:?D;0> 49 .347/3::/ &0;74.,-474?D ;=0/4.?4A0 ;:B0= ,9/ ?30 ?=,4??D;0 4>>@0$)&#! $&'$#! (,#$ !',$!$,    ,.6!'.38@670'27:11
*3:4>7,?0,9/B3:4>0,=7D4214A0;0=>:9,74?D
1,.?:=>,9/;@9.?@,74?D49,??09/492;>D.3:7:24.,70C;0=4809?>$)&#!$'& #&'$#! (,  
,==077 '  ',.0=/:?0    *0>?    =:8 9,?@=,7 A,=4,?4:9 ?: :;?48,7 ;:74.D (30
48;:=?,9.0:109/:209:@>;00=2=:@;1:=8,?4:9$#$"(&    
4,7/494&
(30>.409.0:1;0=>@,>4:9 #(  "& #   
,82,,=/!("407>09'
"@/2492490/@.,?4:9$#$" '$)( $#* + 
09:770?
' '?,9/,=/,>>0>>809?:1902,?4A0,110.?4A4?D>:.4,74934-4?4:9,9/(D;0
;0=>:9,74?D',$'$"( "  #  
:7/-0=2 &    (30 /0A07:;809? :1 8,=60=> 1:= ?30 -4214A0 1,.?:= >?=@.?@=0 ',$!$ !''''"#(   
:1>?00*&,,/  9?02=,?4:9:1?30-4214A0,9/.4=.@8;70C,;;=:,.30>?:?=,4?
>?=@.?@=0$)&#!$&'$#! (,#$ !',$!$,     
:39#$:9,3@0!09?70&   4214A049A09?:=D$)&#!$&'$#! (,#$ !
',$!$,:39>:9#>?09/:=1  7,=414.,?4:9:1?3014A01,.?:=8:/07B4?3?30,-=4/20/-4214A0
/4809>4:9,7.4=.@8;70C$)&#!$&'$#! (,#$ !',$!$,   
:9,>:9 $   *0->?0=   
 (30 /4=?D /:E09  .:9.4>0 80,>@=0 :1 ?30 /,=6 ?=4,/',$!$ !''''"#( 



905

!:?E   !,77:9 !  %@4.6    @?:8,?0/ 0/@.,?4A0 9@/20> ?: =0/@.0 84>>0/
,>>429809?>49.:77020&#'( $#'$#&# ##$!$ '  !@=,6,84 +  !@=,6,84    '.,70 .:9>?=@.?4:9 :1 , F42 4A0G ;0=>:9,74?D 49A09?:=D 
%#'$)&#!$&'$#! (,    #H:99077' ,92
,9;0=>:9,74E0/9@/20>48;=:A070,=9492493D-=4/.7,>>0>
C;0=4809?,7 0A4/09.0 1=:8 ,9 49?=:/@.?:=D @9/0=2=,/@,?0 .:@=>0 $)&#! $ '& $#
#$!$, #)( $#   &,-49!#:9:23@0( :4924?9:B:=7,?0="& #$#$" * +  $,;,D$(,D7:='(D70= ,>64!
 0,=94925:->6477>1=:8.:770,2@0>,?
B:=6 A4/09.0 1=:8 , 1407/ 0C;0=4809? @>492 ?0,.30= ;0=1:=8,9.0 /,?, "& # $#$" $)&#! $#$" $! ,   &:-49> & * :39 # $ ,>;4  !:114?? (   '?:@?3,80= :0-0= !   &0>47409?
:A0=.:9?=:770/ ,9/ @9/0=.:9?=:770/ -:D> (3=00 =0;74.,-70 ;0=>:9,74?D ?D;0>$)&#! $&'$#! (,#$ !%',$!$,   &:20=> (  0770=   4>.:@=,20/ -D ;00= 0C.07709.0 C;:>@=0 ?: 0C08;7,=D ;00=;0=1:=8,9.0.,@>0><@4??492',$!$ !' #        !!     &     ")$ &$'%&)"$&+ *#!$"
!&"!  $%"!&+ % &"$ *#!&"! "$  #$"(! $!!
! !&% ! "&(&"! &" $!             
(3,70= &   '@9>?049  &  "@/20 8;=:A492 /0.4>4:9> ,-:@? 30,7?3 B0,7?3 ,9/
3,;;490>>$#'($! ($#    *,77   ,8;-077   ,D0   0AD   3@77,= "  $0=>:9,74?D ;=:1470> ,9/
;0=>@,>4:9,90C;7:=,?:=D>?@/D49A0>?42,?492?30=:70:1?3042(D;0;0=>:9,74?D,9/?30
,=6(=4,/:9>@>.0;?4-474?D?:;0=>@,>4:9&'$#! (,## * )! &#' 
+,8,@.34 ( (,6,84  7,9,2,9   #2,?,  
 "@/20 80>>,20> 1:= 070,=9492
092,20809? ,9/ >?@/09?H> ;0=>:9,74?D ?=,4?> 110.?> ,9/ 48;74.,?4:9 1:= ;0=>:9,74E,?4:9
&$ #'$(#(&#( $#!$#&#$#$"%)(&' #)( $#  906

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education)JXNLS'FXJI.RUQJRJSYFYNTS7JXJFWHM&
(TQQFGTWFYN[J&UUWTFHMYT*IZHFYJ4ZYTK
8HMTTQ(MNQIWJS
Faisal BADAR * & Jon MASONCharles Darwin University, Australia*Faisal.Badar@students.cdu.edu.au
Jon.Mason@cdu.edu.au
Abstract: Design-Based Implementation Research (DBIR) is characterised by
iterations of a solution informed by collaboration with stakeholders and often used incontexts involving digital technology. It is also useful in building the capacity for longterm collaborations between research and practice for sustainable and equitablechange within an educational system. In this study, DBIR has informed a scalable
solution to provide quality and inclusive education to underprivileged out-of-school
children, using the Digitally Operated One-Room Schoolhouse (DOORS) model. In
doing so, it responds to questions of importance in the local context and builds thecapacity of systems to engage in continuous improvement. This paper presents asummary.
Keywords: Design-Basededucation, design thinking

Implementation

Research,

out-of-school

children,

1. Introduction
Design-Based Implementation Research (DBIR) is a research approach based on principles
of Design-Based Research (DBR) that foreground the role of digital technologies in
improving educational practices through an iterative, collaborative process with key
stakeholders (Amiel, & Reeves, 2008; Anderson & Shattuck, 2012). SDG-4 is one of the
UN’s Sustainable Development Goals that aims to provide universal primary and secondary
education. However, as of UNESCO’s 2022 report, there are still an estimated 244 million
children aged between 6 and 18 years out of school globally. Over the years, global
stakeholders have launched several initiatives such as Education for All (EFA), theMillennium Development Goals (MDG) and One Laptop per Child (OLPC) to ensure theprovision of primary education to all children. However, these initiatives were unsustainable
in meeting their objectives and their lack of delivery of expected progress may be directly
attributed to the failure of considering contextual factors in shaping such initiatives (Badar &Mason, 2020). As an iterative and collaborative approach to developing solutions, DBIR can
assist in adaptable, scalable, and sustainable outcomes for educating OOSC.
This paper reports on a DBIR-based educational initiative that utilises the integrationof education technology tools and the one-room schoolhouse concept. The Digitally
Operated One-Room Schoolhouse (DOORS) model implements an innovative andoverarching curriculum and pedagogy to provide inclusive and quality education to OOSC
using blended learning techniques. This program progressed pragmatically from exploratory
study to design-oriented research, in which feasibility and fine-tuning of learning
environments within rural and remote location contexts was the focus. It then progressed
towards rigorous collaborative efforts among all stakeholders including researchers,
practitioners, OOSC parents, influencers around OOSC, and policymakers. The iterations
were intended to respond to deeply entrenched problems and pragmatically implement anadaptable, sustainable, and scalable system to educate underprivileged OOSC, suited totheir local challenges and limitations.

907

2. EDvantage Digital Learning System-A DBIR Approach to Educate OOSC
EDvantage Digital Learning System (EDLS) is a Digitally Operated One-Room Schoolhouse(DOORS) model, which is a result of extensive research and an iterative implementation
process based on the DBIR approach. EDLS is a complete schooling system, built throughthe collaboration of researchers and practitioners over the period of the last five years. The
project is currently operational in remote and rural areas of Pakistan and Tanzania, providing
free, inclusive, and quality education to OOSC through the integration of modern digital
educational technologies with the centuries-old one-room schoolhouse concept, suited totheir local challenges and limitations. The EDLS-DOORS’s basic design and implementation
process follows design thinking basics as shown in Figure 1 below.

Figure 1. Design and Implementation Process of EDLS-DOORS Model.
The above six-phase process was not implemented linearly but rather iteratively,
especially during the first six months of the prototype phase of the pilot campus. Aftercompletion of six months iterative process of the pilot campus. The subsequent development
and growth from a single prototype campus to 14 campuses during the next 20 months. The
EDLS-DOORS model adopted a pragmatic approach and conduct a series of iterations to
implement best-suited academic and pedagogical processes as shown in Figure 2.
Contextual factorsof underprivileged
OOSC in diﬀerent
global regions

EDLS-DOORS IteraƟve DBIR Model

5. TestContextual factors ofunderprivileged OOSC ateach Campus
Contextual
factors ofunderprivileged
OOSC

1. Focus &Understand

IteraƟon
Series #1
4. Prototype

IteraƟon
Series #3
4. Build

5. Test2. Define

1. Understand3. Ideate

IteraƟon
Series #2
5. Test

1. Understand4. Build

2. Define

Scalability TesƟng
Mul ple Regions Phase

3. Ideate

2. Define

Adaptability &Sustainability TesƟng
Mul ple Campuses Phase

3. Ideate

Concept TesƟng
Pilot Campus Phase

Figure 2: EDLS-DOORS Iterative Model using DBIR Approach

908

The EDLS_DOORS Model consists of three series of iterative stages under a DBIR
framework – the pilot campus phase for concept testing, the multiple campuses phase foradaptability and sustainability. The first series of iterations involved the initial testing of theconcept through the establishment of the pilot campus in 2021. From the initial idea
generation in 2018 to the launch of the pilot campus in 2021, initial planning and researchwere conducted to understand the OOSC problem, gain familiarity with contextual factors ofunderprivileged OOSC through community visits, gauged the local sentiments towards
education for feasibility analysis, studied academic best practices suitable for the contextual
requirements, and understand the EdTech tools suitable for underprivileged OOSC
education. The pilot campus, which was crucial for testing the EDLS-DOORS concept's
applicability and identification, was launched in a rural area of Karachi, Pakistan andobserved closely for six months and iterations were continuous and extensively applied to
improve the overall model.
The second series of iterations involves adaptability and sustainability testing throughmultiple campuses at different locations. The understanding of unique local contextual
factors of OOSC in each location was established for ten new campuses in rural areas.
Through site visits and meetings with community stakeholders, the local dynamics andlimitations are understood and required adjustments to the model were made. Issues suchas language barriers, parental reluctance and cultural restraints were understood andapplicable solutions were designed, making the model sustainable and adaptable.
The third series of iterations involved testing the scalability of this model throughexpansion to multiple regions outside of Pakistan to understand and focus on local
contextual factors of OOSC in different regions of the world. Reaching out to andcollaborating with local stakeholders in those regions was found to be crucial and yet
stakeholders from five different countries approached. The scalability testing in this iteration
has been initiated through the launch of online Spoken English classes in rural Tanzania.

3. Conclusion
DBIR is a systematic approach for working with stakeholders to iteratively design researchthat focuses on implementation that responds to questions of importance in the local context.
As outlined in this paper DBIR has been pivotal in guiding the early success of the EDLSDOORS implementations. DBIR poses its own set of challenges and limitations to theresearcher. Establishing researcher-practitioner partnerships and building trust may bringalong a challenge for researchers, particularly with new stakeholders with whom no previous
relationship exists (Fishman, Cheng, & Penuel, 2019). EDLS-DOORS model provides anopportunity to approach out-of-school children’s education differently and shows promisingoutcomes in terms of its adaptability, sustainability, and scalability.

References
Amiel, T., & Reeves, T. C. (2008). Design-based research and educational technology: Rethinking
technology and the research agenda. Journal of educational technology & society, 11(4), 29-40.
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education
research? Educational researcher, 41(1), 16-25.
Badar, F. B., & Mason, J. (2020). Numbers are Alarming, Solutions are Scant - Out of School Children
in Pakistan. Proceedings From ICSD 2020 (pp. 1-16). International Conference on Sustainable
Development. Retrieved from https://ic-sd.org/wp-content/uploads/2020/11/Faisal-Badar.pdf
Badar, F., & Mason, J. (2020). Towards Digital Multigrade One-room Schoolhouses for
Underprivileged Communities in Rural Pakistan.
Badar, F., & Mason, J. (2019). Digital learning transformation for one-room schoolhouses in rural
Pakistan. Proceedings of the 27th International Conference on Computers in Education. Taiwan:
Asia-Pacific Society for Computers in Education. Retrieved fromhttp://ilt.nutn.edu.tw/icce2019/dw/ICCE2019%20Proceedings%20Volume%20II.pdf
Fishman, B. J., Cheng, B., & Penuel, W. R. (2019). CIRCL primer: Design-based implementation
research. Retrieved from CIRCL Primer Series: http://circlcenter.org/dbir/909

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Study of Versatile Tutor Training
Programs for Universities in Japan
Yumi ISHIGEa*, Kazuhiro KABEYAb, Kayoko NAGAOc & Hirotoshi TANIGAWAdaModern Social Studies, Otemae University, Japanb
School of Health and Sport Sciences, Osaka University of Health and Sport Sciences,Japan
c
School of Health and Sport Sciences, Osaka University of Health and Sport Sciences,Japanc
Department of Early Childhood Care and Education, Shikoku University, Japan*y-ishige@otemae.ac.jp
Abstract: This research reports on the creation of online tutor training programs forlearning support centers in Japanese universities. As this topic is relatively new inJapanese higher education institutions, there is a lack of practical training programs
that can serve as references. To address the issue, we have opted to develop programs
based on a framework established in the United States. The programs were plannedin accordance with two core concepts: “usable by any university” and “to be selflearning materials.” The primary objective was to create training programs that can beuniversally used in Japanese universities, allowing tutors to learn independently even
without a lecturer. In conclusion, it can be affirmed that the programs serve as versatiletutor training resources. While some cultural differences may have emerged due to theadoption of a US-based framework, it is believed that these variances can be partiallyresolved by making adjustments during on-site programs.
Keywords: tutor training program, learning support center, online learning materials

1. Introduction
The main propose of this research is to develop training programs for tutors based on aframework used in the United States and to validate the effectiveness of the programs forJapanese universities. There are two primary reasons for undertaking this research. First,
the necessity of a learning support center (LSC) in higher education has been recognized inJapan since around the latter half of the 2000s (The Japan Association for Developmental
Education, 2012).In this research, a LSC is defined as an organization that assists students
with their learning outside of classroom(Ministry of Education, Culture, Sports, Science, andTechnology, 2010). In light of this need, the importance of training programs for tutors
working at LSC has gained attention recently. However, there are currently no authorized
and established tutor training programs in Japan, despite individual institutions implementing
their own programs. Several LSCs appear to require more information on tutor training. The
creation of such programs is expected to improve the situation better.
Second, there is a significant disparity between the abundance of tutor training
materials available in the United States compared to Japan. While these resources canserve as valuable references, it is challenging to adopt them directly without modifications.
This is because the requirements for academic support in Japanese universities areperceived to differ from those in the United States. Thus, it becomes necessary to adapt thematerials to suit the Japanese context. Therefore, we made the decision to design our own
tutor training programs referring to the research of The College Reading and Learning
Association (CRLA). CRLA is known for issuing certifications for international tutor training
programs, making it suitable for our purposes. This paper will present the contents of thetutor training programs we have developed and examine their effectiveness.

910

2. Method: Creating tutor training programs
Two key concepts guided the creation of the tutor training programs in this research: one is“usable by any university” and the other is to be “self-learning materials.” While it isundeniable that each university, faculty, and department has its own study styles andlearning contents, it is unlikely that a single teaching method would be universally suitable
for all subjects. However, we believe tutoring skills aimed at supporting tutees can be
developed regardless of the differences between universities, instead of focusing onteaching methods for specific subjects. Tutor training programs that we aim to develop
should adhere to these principles. Secondly, it is important to note that not all universities inJapan with LSC have a lecturer who can train tutors. The reasons for this absence may varydepending on the university, such as budget constraints, lecturers being occupied with other
responsibilities, and/or a lack of understanding of training needs. In order to make thesituation better, providing self-training materials can be an effective solution. Thus, we havedecided to develop materials that can be utilized for both collective and individual learning.
Based on the aforementioned principles, we have selected the article “Standards,
Outcomes, and Possible Assessments for ITTPC Certification” by CRLA as basic reference.
ITTPC stands for “International Tutor Training Program Certification,” which was establishedto provide “recognition and positive reinforcement for tutoring program’s successful workfrom an international organization,” and it also requires “set an internationally-accepted
standard of skills and training for tutors” (The College Reading and Learning Association,2018). The article presents ideas on tutoring skills that are applicable internationally, rather
than specific to particular fields of study. We anticipate that it will be well-suited for our
intended purpose. While our research does not aim to develop programs by which thecertification can be acquired, the standards can be a good example what tutors are required
at higher education.
Five researchers, including myself, have actively participated in the research. We
engaged in extensive discussions and evaluations of the contents of ITTPC and of ourprograms, then we were assigned to topics and made each of them, and brought them in forfurther discussion. One of the distinguishing features of the programs is that they were
designed as online materials, allowing for easy accessibility and reference. Consequently,
we refer to these online tutor training programs as “OTT.” Each topic of OTT consists of ashort video, a note containing key sentences, and a worksheet with assignments. These
assignments consist of both individual and multi-person tasks. The note with key sentences
and the worksheet with assignments are provided as downloadable PDF files, allowing foreasy access and use.

3. Result
Tutor training at Otemae University was conducted even before the existence of OTT, during
which a diverse range of materials were utilized based on the identified necessary topics.
However, compared to that time, the implementation of OTT brings several advantages.
First, having a single set of training materials enables both tutors and lecturers to adhere toa consistent policy, providing a sense of stability and security. The establishment of OTT has
indeed resulted in increased reliability, which is believed to be associated with the reliabilityof ITTPC. Tutors have also provided feedback stating that the expectations placed uponthem have become clearer through the use of OTT.
Second, tutors use worksheets with assignments to help them in preparing theirworkshops during the training. Those materials can be valuable resources for tutors,
assisting them in organizing specific segments of tutor training. Third, the availability of
online materials has made it easy for tutors to work on preparation and review, and createup work for absent students is easier than when there were no such materials available.
Although the benefits of OTT have been highlighted above, some challenges haveemerged. OTT’s universal design occasionally caused confusion among tutors as certain
descriptions did not align with their specific LSC contexts. Moreover, misunderstandings ofassignment intent led to workshops being perceived as ineffective. These issues currently911pose negative aspects of OTT. Another challenge lies in the widespread dissemination of
OTT to Japanese universities.

4. The effectiveness of OTT
To sum up, topics offered by ITTPC serve as an effective framework for tutor training at
Japanese universities. The existence of such programs has been helpful, allowing students
to develop their skills and awareness through the use of OTT. The well-designed framework
serves as a valuable reference for tutors. Even if a lecturer does not use OTT directly for
their own tutor training, presenting the topics of OTT as a guide to the required skills andcompetencies of tutors can still be effective. It is worth noting that OTT is available inJapanese, further enhancing its accessibility and relevance in the Japanese context.
Nonetheless, the impact of cultural differences on the use of OTT cannot becompletely ignored. In the case of Otemae University, a topic “Adult learners” was a notable
example. In Japan, the majority of university students enroll immediately after graduatingfrom high school, resulting in a limited number of mature students. For instance, at myuniversity, there are only one or fewer mature students per grade level who utilize the LSC.
Thus, the topic is not used in tutor training at Otemae University as an opportunity to takeadvantage of is really rare. This situation is not unique to Otemae University but is common
among many universities in Japan (e-Stat, 2021). Another case is “Supplemental
Instruction.” While the instructional method is explained in the material, it is not typicallyorganized or utilized in the context of Japanese higher education institutions. Therefore, atOtemae University, Supplemental Instruction is also omitted from tutor training due to its
limited applicability in the typical Japanese educational setting.
Hence, it is evident that not all content in OTT is invaluable to a specific university.
This limitation arises from the nature of versatile programs. However, it is expected that thisissue can be addressed through the way individual implementation and operation of OTT byeach university. Overall, it can be concluded that the objective of creating versatile tutor
training programs using the ITTPC framework has been achieved, and significant results
have been obtained.
In the future, we have plans to improve the content by addressing any aspects that
may lead to misunderstandings among tutors. Besides, as OTT is a freely accessible online
resource, we aim to collaborate with educational societies to promote its widespread use.
While we have only presented the practices of one university in this study, our intention is togather operational data from multiple universities and conduct future comparisons.

Acknowledgements
This work was supported by Grants-in-Aid for Scientific Research 17K01163 and 20K03112.

References
e-Stat. (2021). Basic School Survey, University Enrollment by Age Group. https://www.estat.go.jp/stat- search/files?page=1&layout=datalist&toukei=00400001&tstat=000001011528&cycle=0&tclass1=000001161251&tclass2=000001161252&tclass3=000001161253&tclass4=00
0001161255&tclass5val=0. (2023.05.03).
Ministry of Education, Culture, Sports, Science, and Technology. (2010). Third: Student Support andLearning Environment Improvement. https://www.mext.go.jp/b_menu/shingi/chukyo/ chukyo4/
houkoku/attach/1297027.htm. (2023.04.22).
The College Reading and Learning Association. (2018). Purpose.
https://crla.net/index.php/certifications/ittpc-international-tutor-training-program.(2023.03.12).
The Japan Association for Developmental Education. (2012). Building a System of Developmental
Education at Universities and Colleges in Japan. Nakanishiya Shuppan.
Yumi ISHIGE, Yuri NONAMI, Naoya HONDA, Yutaka ASATANI. (2021). A Report on the Learning
Support Centre at Otemae University: Activities during the COVID 19 pandemic in 2020. Otemae
Ronshy 21, 139-160.

912

" !   !" " "   #" !
#" !"$  #" ! #"               
    

 "!%   $ !%# !% "!% " "!"# !% 
=%1%>%/-81,80-:) %7%,-/%;% 9 %' .4  /31&$*"-*341206*%&%(5*%"/$&'02"'*44*/(120$&33"*.&%"4&/)"/$*/()20.&#00,53"#*-*49'02"$)*-%7*4)$&2&#2"-1"-399("4)&2*/('&&%#"$,'20.4)&$)*-%2&/"/%0#3&26*/(4)&*2*/4&2"$4*0/37*4)4)&%&6*$&"%+534.&/437&2&."%&53*/("$$&33*#*-*49 '5/$4*0/3 40 *.1206& &"3& 0' 53& 81&24 "%6*$& "%%2&33&% $)"--&/(&3#&90/%"$$&33*#*-*49*.1206&.&/432&35-4*/(*/"35$$&33'5-'*44*/(120$&33    $&2&#2"- 1"-39  )20.&#00, '*44*/( "$$&33*#*-*49 31&$*"- 3511024
&%5$"4*0/  
!,) ',3302-8-%8-:),%7463:-()(32)():-')4)6789()28-27',3307%'6377%4%2
-2'09(-2+8,37)*3674)'-%02))(7 6)4368-2(-'%8)78,%8,631) '3278-898)7 3*8,)7)():-')7-140=-2+;-()746)%(97) 3;):)6%(%48-2+,631)&33/7*36789()287;-8,
')6)&6%04%07=-7',%00)2+-2+(9)838,)-692-59)4,=7-'%00-1-8%8-327 !%2-3/%)8%0  78%8)(2+)2)6%04,=7-'%00=(-7%&0)(4)340);-8,')6)&6%04%07=1%=,%:)-2:30928%6=
%'8-32773;,)28,)='327'-3970=86=8313:)32)43-28328,)-6&3(=8,)=1%=238&)%&0)
8313:)-84634)60=(9)838,)*36')3*8,)-6)28-6)&3(= )1)28-32)(8,%8%7=78)18,%8
)2%&0)78)<8-2498%2(34)6%8-32&=:3-'),%7&))2():)034)( 3;):)68,)6)%6)
(-**)6)2')7-28,)():-')7%2(8,)-6 -278%00)(-274)'-%0 2))(77',3307-2)%',3*8,)03'%0
192-'-4%0-8-)7%2(8,)78%8)3*',-0(6)2-87)0*:%6-)7-8-7(-**-'90883'327869'8%7=78)1
8%-036)(83)%',',-0(%7!%2-3/%,%779++)78)( 2%((-8-320%'/3*8)%',)67/23;0)(+)3*
!():-')71%=0)%(838,)4377-&-0-8=8,%81%2=74)'-%0 2))(77',3307(32388%/)
%(:%28%+)3*8,)%')77-&-0-8=*92'8-3273*8,)-6():-')7 28,-7789(=8,)6)*36);-8,8,)
%(:-')3*)<4)687%*-88-2+;%7'32(9'8)(97-2+%'')77-&-0-8=*92'8-327831%/)8,)
,631)&33/7)%7-)68397)*36',-0(6)2;-8,')6)&6%04%07=   
 
!,)496437)3*8,-7789(=%-1)(83)2,%2'),631)&33/97%&-0-8=*36%',-0(;-8,')6)&6%0
4%07=8,639+,)<4)68 +9-()('97831->%8-32*3'97-2+328,)*-88-2+463')7783*%'-0-8%8)8,)
',-0(7)**)'8-:)97%+)   !,)789(=*3'97)732#%.92-36,-+,7',330789()28;-8,')6)&6%04%07=%2(-28)00)'89%0
(-7%&-0-8=%8$ 4)'-%0 944368 ',330 #98-0->)7%7',330 463:-()(,631)&33/*36:%6-397

913

%'8-:-8-)7&38,%87',330%2(,31) 3;):)6(9)83-2:30928%6=13:)1)287 7,))2'3928)67(-**-'908-)70-/)92-28)2()(8%47328,)7'6))23692-28)28-32%0(6%++-2+;,-0)97-2+8,)
():-')

-88-2+7;)6)'32(9'8)(*3968-1)7*36%838%03* 1-298)74)67)77-32 !,))**)'8-:)2)77
3*8,)*-88-2+%2(-87)**)'8-:)1)8,3(;-00&)):%09%8)(&=%2%0=>-2+8,):-()378%/)23*8,)
*-88-2+%2(8,))4-73()7*631-28)6:-);7;-8,#%*8)68,)*-88-2+  !,)6)7)%6',-2:30:)(-28,-746)7)28%8-32,%7&))2'32(9'8)(;-8,%446346-%8))8,-'%0'327-()6%8-327%2(,%2(0-2+3*4)6732%0-2*361%8-32%2('327)28,%7&))23&8%-2)(*631
8,)-2(-:-(9%04%6)287 +9%6(-%27%2(8,),)%(3*8,)7',33083;,-',8,)79&.)'8&)032+7 
96-2+8,)7)77-32 #;%7-28)6:-);)(%&398,)6',%00)2+)7;-8,97-2+8,),631)&33/
%2(,)6-28)6%'8-327;-8,8,)():-');)6)3&7)6:)( 2%46):-397-278%2');,)279&1-88-2+
%2%77-+21)283233+0)0%776331#)2'3928)6)((-**-'908-)78%44-2+8,)839',(-740%=
%''96%8)0=6)7908-2+-21908-40)79&1-77-32%88)1487 2%79&7)59)288%7/7,)%88)148)(8397)8,)839',4%(*36'0-'/-2+3''%7-32%00=%440=-2+)<')77-:)46)7796) ((-8-32%00=7,)
()132786%8)(8,)%&-0-8=837)%6',8,)-28)62)897-2+%'31&-2%8-323*839',%2(/)=&3%6(-24987%08,39+,8,) 71%007->)3*7)%6',)0)1)287'%97)(,)6833*8)28%48,);632+%6)%7
0)%(-2+83*69786%8-32 %780=94(%8-2+%24)2-%6='6)%8)(;-8,33+0) 46)%(7,))8
463:)(8639&0)731)*36#%78,)71%00'309127*368=4-2+'311)287%2(7)0)'8-2+(%8)7
46)7)28)(8%44-2+',%00)2+)7

 
96-2+8,)7)'32(7)77-32)**3687;)6)1%()83%((6)77#77869++0);-8,(-740%=7->) *8)68)78-2+:%6-3977->)7%
(-740%=;%77)0)'8)(%78,)137879-8%&0)*36,)6 !3%((6)77(-**-'908-)7-2'0-'/-2+8,)%983'0-'/*92'8-32;%7-2863(9')(;-8,%32) 7)'32(()0%=&)8;))2'0-'/783%''3113(%8)#746)*)6)2')%2(-2:30928%6=13:)1)287 !,)
43-28)6437-8-32'366)'8-32*92'8-32;%7%073-140)1)28)( (.9781)287;)6)1%()838,)
1397)43-28)677->)'3036%2(74))(&%7)(32#7-2498 !,)43-28)67->)6)1%-2)(
()*%9088,)'3036;%7',%2+)(836)(,)6*%:36-8)%2(8,)()*%90874))(;%76)8%-2)(*36
'31*368%&0)97) 3003;-2+8,)7)%(.9781)287-8;%7()'-()(8,%8#;390('328-29)97-2+8,)'97831->)(7)88-2+7 928-08,)2)<87)77-32

 "7-2+%839',4%(%032)

" (.978-2+*92'8-327;-8,8)%',)67914

 
96-2+8,)8,-6(7)77-32%2)<4)68-2,631)&33/1%29*%'896-2+%2(7%0)7463:-()(320-2)463*)77-32%0+9-(%2')*36*968,)6*-88-2+%(.9781)287 #7,%6)(,)6'32')627%&398
-1%+)7)%6',)7;,)6),)6*-2+)677869++0)(8313:)%''96%8)0=0)%(-2+83-2'366)'8'0-'/7 !,))<4)688%9+,8,)68397)8,)8%&/)=*367)0)'8-32%-(-2+,)6-2',337-2+8,)()7-6)(
-1%+) !)',2-59)7*36)%7-0=2%:-+%8-2+7)0)'8-327):)2;-8,4%6%0=7-7;)6)()132786%8)(
97-2+8,) ,-*8%2(8%&/)=7 #%0736)')-:)(+9-(%2')328=4-2+'311)287-28,)4)2
-%6= !,))<4)68%(:-7)(97-2+8,)%(.%')281)29'30912837-140-*=7,))87)0)'8-32*36
',%2+-2+(%8)7;,-',#*392(46)*)6%&0)3:)6,36->328%07'6300-2+ *8)68,)*-88-2+#'327908)(8,)74)'-%0-78;,38%9+,8,)67,))8 7;-8',-2+7,368'987 3;):)6(9)83,)6
4%07=7,)*392(-8',%00)2+-2+8346)771908-40)/)=77-1908%2)3970=%2(348)(%+%-278
97-2+7,368'987

 
28,)*3968,7)77-32#)<46)77)(,)67%8-7*%'8-32;-8,8,)'97831->)(,631)&33/*-2(-2+-819',)%7-)68397)8,%2&)*36)  ,),-+,0-+,8)(8,)'32:)2-)2')3*8,)4,=7-'%0/)=&3%6(8,)32) ,%2()((-740%=%(.9781)28%2(8,)348-32836)78,)6,%2(328,)
(-740%=7&)>)0;,-0)839',-2+-8 7%6)79087,)-28)2(783'328-29)97-2+8,),631)&33/

" 20-2)%(:-')*631%2%(:-736

" !%44-2+;-8,%*-2+)6328,)(-740%=  
=%((6)77-2+#7-779)7%2(3&7)6:-2+,)697)3*8,),631)&33/)**)'8-:)*-88-2+;%7%',-):)( 08)6-2+8,)(-740%=7->)-1463:)(8%44-2+%''96%'=;,-0)8,)%9831%8-''0-'/
*92'8-32%-()('0-'/-2+()74-8)*-2+)64%6%0=7-7 (.978-2+8,)1397)43-28)67'3036%2(74))(&%7)(32#746)*)6)2')7238320=)2,%2')(97%&-0-8=&98%073*378)6)(%88%',1)28
838,)():-') <4)68+9-(%2')328%&%2(7,368'98/)=76)730:)(',%00)2+)7238730:%&0)&=
%'')77-&-0-8=*)%896)7%032) !,)*-88-2+463')777 79'')7778)11)(*63192()678%2(-2+8,)
',-0(6)27463&0)17%(.978-2+)59-41)28&%7)(328,)-6-2498%2(7))/-2+)<4)68%(:-')*36-779)7&)=32(%'')77-&-0-8=-1463:)1)287

 
*/*34290'%5$"4*0/5-452&10243$*&/$&"/%&$)/0-0(9  &10240' "2*053526&930/4)&$)00-/*4*"4*6& " '$#  !!$$!# &&&'$ ! $$  
'$ 
! "$$&33&%    
"/*0,"03)*."3"("3)*2" *2095,*","4""95.*!"4"/"#&&/:*0/%0*20,*  
&6&-01.&/40'"934&./"#-&3)"2"$4&2/154"/%1&2"4*0/6*" 0*$&'02")93*$"--9*3"#-&%&230/7*4)"1&&$).1&%*.&/4 %" $!# $( " "$
$   915

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Exploring the Relationship between 21stCentury Skills and Motivation: A Study
Using Contextual Inquiry Project-based
Learning
Jirapipat THANYAPHONGPHATa*, Preeyada TAPINGKAEb, Kannika
DAUNGCHARONEa, & Krittawaya THONGKOOaaCollege of Arts, Media and Technology, Chiang Mai University, Thailandb
Ban San Pa Sak School Chiang Mai, Thailand*jirapipat.than@cmu.ac.th
Abstract: Motivation has gained considerable attention as educators and researchers
seek to understand and nurture the diverse set of 21st century skills required forindividuals to thrive in the rapidly evolving landscape of the 21st century. These skills,
including critical thinking, collaboration, problem-solving, creativity, and self-direction,
among others, are crucial for success in various professional and personal
contexts.This study investigates the relationship between 21st-century skills andmotivation within the framework of Contextual Inquiry Project-based Learning (CI-PBL).
The purpose of the study is to gain insights into how motivation influences thedevelopment of various 21st-century skills. The research employs Pearson's
correlation coefficient to analyze the relationship between motivation and these skills.
The study aims to provide a comprehensive understanding of the strength and directionof the correlations, revealing potential patterns and associations between motivationand skill development.
Keywords: Contextual Inquiry Project-based Learning, Project-based learning,
Motivation, 21st Century Skills, ICT innovation in education

1. Introduction
In the rapidly evolving landscape of education, there is a growing recognition of the need toequip students with skills that are relevant to the demands of the 21st century. These skills,
often referred to as 21st century skills, encompass a wide range of abilities andcompetencies that enable individuals to navigate and thrive in the modern world. Some of
the key skills emphasized in this context include critical thinking, creativity, collaboration,
problem-solving, self-direction, meaningful learning with Information and communication
technology, and knowledge creation efficacy.
Individuals' levels of interest, perseverance, and effort in learning and using new
skills are greatly influenced by their motivation. It is well known that it plays a crucial role ineducational settings since it affects students' capacity to set objectives, control their learning
processes, and deal with difficulties. Despite the fact that the importance of motivation in theclassroom is generally known, research on the precise connection between motivation and21st century abilities is still underway.
Explore the 21st Century Skills and competencies that are essential for learners inAsia to succeed in the new millennium. The rapid changes brought about by globalizationand technological advancement require learners to possess a new set of skills andcompetencies that go beyond traditional academic knowledge and skills.

916

Through a study utilizing contextual inquiry project-based learning, this researchseeks to analyze this relationship. The study will look at how this strategy can help students
develop both of these skill sets more effectively. We intend to learn more about theconnections between these two sets of talents and how they might be effectively cultivatedusing cutting-edge teaching strategies via the course of this research. The results of thisstudy could have a big impact on how educators and policymakers create educational
programs that support the growth of computational thinking and 21st century abilities. We
can prepare pupils for life in a complicated, technologically advanced environment by giving
them these abilities.
The investigation of this relationship study aims to contribute to the existing body ofresearch on both motivation and 21st century skills. By uncovering potential correlations andpatterns, educators and researchers can gain valuable insights into the dynamics between
motivation and skill development, informing instructional practices and curriculum design.
The research questions in this study are as follows:xx

What is the nature and strength of the correlation between motivation and various 21st
century skills?
How does motivation influence the development and application of 21st century skills
within the CI-PBL framework?

2. Literature Review
2.1 21st Century Skill
21st-century skills refer to a set of competencies that are considered essential for success inthe modern world. These skills go beyond traditional academic knowledge and include
abilities such as critical thinking, problem-solving, collaboration, communication, creativity,and digital literacy. According to the World Economic Forum, these skills are becoming
increasingly important due to factors such as globalization, technological advances, and thechanging nature of work (World Economic Forum, 2018).
Several frameworks have been developed to define and organize 21st-century skills.
One popular framework is the Partnership for 21st Century Skills (P21) Framework, which
identifies four key categories of skills: (1) communication and collaboration, (2) criticalthinking and problem-solving, (3) creativity and innovation, and (4) digital literacy(Partnership for 21st Century Skills, 2015).
Research has shown that developing 21st-century skills can lead to a range of
positive outcomes for students, including improved academic performance, increased
engagement, and better preparation for future careers (Trilling & Fadel, 2009; Griffin,
McGaw, & Care, 2018). However, there are also challenges associated with integrating 21stcentury skills into traditional educational practices, such as resistance to change, lack ofteacher training, and difficulty in assessing these skills (Voogt et al., 2015).
21st Century Skill by Chai Ching Sing, et al. (2015) is a research paper that proposeda framework for assessing students' perceptions of 21st-century learning practices. The
framework includes seven dimensions of 21st-century skills:
1. Self-directed learning (SDL): the ability to take responsibility for one's own learning.
2. Meaningful learning with Information and communication technology (MLT) : theability to use technology as a tool for promoting student-centered, collaborative,
and authentic learning experiences that align with the demands of the 21st-century
workplace (Jang et al., 2017)
3. Collaborative learning(CoL): the ability to work effectively in teams and shareknowledge with others
4. Critical thinking(CriT): the ability to analyze information and make informed
decisions
5. Creative thinking(CreT): the ability to generate and develop innovative ideas andsolutions917

6. Authentic problem-solving(APS): the ability to solve real-world problems using
critical and creative thinking skills
7. Knowledge creation efficacy (KCE) : the ability to develop new knowledge throughcollaboration and efficient learning process.
The paper argues that these seven dimensions of 21st-century skills are essential forstudents to succeed in the modern world and that educators should prioritize thedevelopment of these skills in their teaching practices.

2.2 Motivation
Motivation is the driving force that underlies behavior and actions. It refers to the internal andexternal factors that influence the direction, intensity, and persistence of behavior (Ryan &Deci, 2020). Motivation plays a critical role in learning and achievement, as it can impact
students' engagement, effort, and performance.
Researchers have identified several strategies for fostering motivation in educationalsettings, such as providing meaningful and challenging tasks, promoting autonomy andchoice, giving feedback, and creating a supportive learning environment (Brophy, 2021).
The previous study highlights the effectiveness of project-based learning as a
strategy for promoting computational thinking skills and motivating students in computerscience education. By engaging students in authentic and hands-on programming projects,
the approach allows them to apply computational thinking principles in real-world contexts,
enhancing their understanding and interest in the subject matter (Thanyaphongphat et al.,2022). There exist six distinct dimensions that encompass the concept of motivation.
1. IM (Intrinsic Motivation): Intrinsic motivation is the natural desire and interest in
doing something for its own sake, such as because you enjoy it or are curious
about it. It's important to include this factor because it helps show how interested
and passionate people may be about the subject. It's a key way to figure out whatreally drives people to be engaged.
2. PG (Personal Goals): Personal goals are the hopes, dreams, and goals that aperson has for their own academic or personal growth. This factor shows how
students' motivation is tied to their own goals, which can be different and can drive
their hard work and dedication.
3. EM (Extrinsic Motivation): When someone does something for external benefits or
to avoid bad things, this is called extrinsic motivation. This can be based on things
like grades, awards, or social acceptance. It's important to include this factor toaccount for the different motivations that can come from outside sources, which
can affect how engaged students are.
4. AC (Anxiety about Computer Science Assessment): Worrying about tests in acertain topic, like computer science, can have a big effect on how motivatedstudents are to learn. This dimension focuses on the emotional side of motivation,
showing how worry can both help and hurt motivation. It's important to think aboutworry because it can change how you learn.
5. SD (Self-Determination): Self-determination is a person's sense of being in chargeof their own acts and decisions. This factor is very important for figuring out how
much control students feel they have over their learning. Self-determined students
are more likely to be self-motivated and interested in school.
6. SE (Self-Efficacy): Self-efficacy is a person's belief that they can do a job or reacha goal on their own. It's a very important factor because it directly affects how hard
students are willing to work on a topic. High self-efficacy can help students work
harder and get more done.

2.3 Project-based learning
Project-based learning has emerged as a promising approach to fostering both 21st century
skills and computational thinking. Contextual Inquiry Project-based Learning, in particular,
has been shown to be an effective way of promoting these skills. According to Hsu et al.

918

(2018), this approach involves students in a collaborative project that is based on a realworld problem. The students are guided through the project with the aim of developing 21st
century skills and computational thinking. This approach has been shown to be effective inenhancing students' problem-solving abilities, critical thinking, and collaboration skills (Hsuet al., 2018).

2.4 Contextual Inquiry
Contextual inquiry is a qualitative research method used to understand users' behaviors andneeds in the context of their work or daily lives. The method involves observing andinterviewing users in their natural environment and analyzing the data collected to identify
patterns and insights. Contextual inquiry is commonly used in user-centered design and canbe applied to a variety of contexts, including healthcare, education, and technology.
One of the key features of contextual inquiry is the emphasis on observation andparticipation. Researchers not only observe users but also participate in their activities togain a deeper understanding of their experiences. This approach allows researchers to
identify implicit needs and behaviors that may not be apparent through traditional surveys orinterviews.
Another important aspect of contextual inquiry is the use of contextual data to informthe design process. Researchers collect data on the physical environment, tools, andartifacts used by users, as well as their social and organizational contexts. This information
can be used to develop design solutions that are tailored to users' needs and contexts.

2.5 Contextual Inquiry Project-based Learning
Contextual Inquiry Project-based Learning (CI-PBL) is an instructional approach thatcombines the Contextual Inquiry research method with Project-based Learning (PBL) to
promote meaningful and authentic learning experiences. This approach involves having
students engage in inquiry-based projects that are grounded in real-world contexts and thatinvolve the use of technology. These steps are seven steps for learning:
1. Motivate and Prepare: This step involves building motivation and preparing
students with the foundational knowledge and necessary skills for the project.
2. Analyze: In this step, students analyze the problem and make connections tothe context in which the project is taking place.
3. Plan: This step involves planning the project and determining how the problem
will be solved.
4. Do: In this step, students implement their plan by taking action and solving the
problem.
5. Examine & Evaluation: In this step, students examine and evaluate their work
to determine its effectiveness and to identify areas for improvement.
6. Explore & Expand: This step involves exploring new knowledge and expanding
the scope of the project beyond the original problem.
7. Develop & Distribute: In the final step, students develop and distribute theirwork to others, sharing their solutions and knowledge with a wider audience.
By following these seven steps, students can engage in authentic, inquiry-basedlearning that helps them develop critical thinking, problem-solving, communication, andcollaboration skills. The CI-PBL approach allows students to take ownership of their learningand connect academic concepts to real-world contexts, making learning more meaningful
and relevant to their lives.
Research has shown in Figure 1 that the CI-PBL approach can be effective inpromoting the development of 21st-century skills such as critical thinking, creativity,communication, and collaboration. For example, a study by Lee and Wu (2019) found thatusing a Contextual Inquiry Project-based Learning approach in a college-level computerscience course led to improvements in students' critical thinking and problem-solving skills.

919

Figure 1. The Contextual Inquiry Project-based Learning approach.

2.6 Relationship between 21st-Century Skills and Motivation in PBL using Contextual
Inquiry
There is growing evidence that the development of 21st-century skills is positively related to
students' motivation to learn. Research suggests that when students are engaged in tasks
that require them to use 21st-century skills such as critical thinking, creativity, andcollaboration, they are more likely to be intrinsically motivated and engaged in their learning(Kirschner & van Merriënboer, 2013).
In addition, project-based learning (PBL) is an instructional approach that is often
used to develop 21st-century skills and foster student motivation. PBL involves students
working on real-world problems or challenges and requires them to use a range of 21stcentury skills such as communication, collaboration, and problem-solving (Hung et al.,2008). Studies have found that PBL can increase student motivation, engagement, andachievement (Krajcik et al., 2014).

3. Methodology
3.1 Participants
The students from northern Thai high schools (grades 7) participated in the pilot project. The
same teachers taught the same unit content for groups. This class had twenty-five combined
students: thirteen males and twelve females.

3.2 Instrument
The instrument used to assess student motivation was modified from the questionnairedeveloped by Panjaburee and Srisawasdi (2016), which had a scale reliability of 0.89. The
estimated value of 29 items on Likert's Scale (Likert's Scale). The questionnaire scores wereallocated a five-level Rating Scale: least, less, moderate, very, and most, which was thentranslated into Thai by a language expert. Then, retest the confidence with a Cronbach's
alpha confidence of 0.928, which indicates extremely high confidence.
The 21st Century Skills Questionnaire was made by Chai Ching Sing, et al(2015)
with a Cronbach's alpha confidence score of 0.95. We then translated it into Thai. There
were 42 questions on the poll, six for each scale. The poll questions were shown on a fivepoint Likert scale, with 1 being strongly disagree and 5 being strongly agree. The Thai
version was used, and Cronbach's alpha was 0.90, which means there was a very high levelof confidence and that it could be used to evaluate 21st Century Skills reasoning in sevendifferent areas.

920

An open-ended questionnaire focused on the activities that students enjoyed and theexperiences that left a significant impression on them.

3.3 Research Design
To investigate the relationship between 21st century skills and motivation skills.The
Contextual Inquiry Project-based Learning approach is combined with the context-based
inquiry-based learning management to promote 21st-century skills and enhance themotivation of grade 7 students to learn science subjects, specifically Computational Science.
The program consists of Level 1, with 1 hour per session, totaling 18 hours, and includes thefollowing units:x Unit 1: Digital Intelligence (2 hours)x Unit 2: Digital Problem Solver (2 hours)x Unit 3: Digital Inventor (10 hours)x Unit 4: Digital Data Management (4 hours)
The learning activities are designed to integrate skills, processes, and experiences.
The content is arranged in order to increase difficulty, catering to the original skills of thelearners. Both individual and group activities are included. For group activities, students will
be divided into groups of 3-4 people and work together on activity sheets.
The Learning Management Plan for Unit 3, "Digital Inventor," consists of a 10-hoursession for students, with 1 hour dedicated to problem-solving steps. The breakdown ofactivities is as follows:x Coding Heart Rate: 1 hourx Car Coding: 2 hours
x Fan Coding: 3 hours
x Digital Inventor: 3 hours
To support the learning process, various materials will be utilized, including activity
sheets, knowledge sheets, slides, and teaching media. Additionally, each student will be
provided with one Micro:bit board.

Figure 2. Example Car Coding activity in Digital Inventor unit.

4. Results and Discussions
In order to determine the impact of CI-PBL on Computer Science education, Table 1
displays the level of student motivation after learning. It asserts that the majority of student921

learning incentives were positive. In contrast, the evaluation component generated the least
amount of anxiety. It indicates that students are not anxious about the test. These results
demonstrate that CI-PBL can motivate all students to study Computer Science.
Table 1 Means and SD of the Students’ Motivation Towards the Contextual Inquiry Project-based
Learning in Computer Science.

Learning Motivations Dimensions

Mean

SD.

SE.

RemarkIM (Intrinsic Motivation)

3.79200.61232

0.12246Agree

PG (Personal Goals)

3.69600.750820.15016Agree

EM (Extrinsic Motivation)
3.5840AC (Anxiety about Computer Science
2.1360Assessment)
SD (Self-Determination)
3.67200.720930.14419Agree

0.77399

0.15480

Disagree0.591270.11825Agree

SE (Self-Efficacy)

0.57178

0.11436Agree

3.5120For another query data analysis, Pearson's correlation coefficient was used to examine
the relationship between the six motivational dimensions. From Table 2, it was shown that thehighest positive correlation existed between the intrinsic (IM) dimension and the selfdetermination (SD) dimension significantly (r = 0.867, p < 0.01) means that students
participate in learning activities with intrinsic motivation and self-determination motivation.
They can complete their projects in other ways. Both motives can increase interest andenjoyment in learning. Other significant associations were found between anxiety about thecomputer science (AC) assessment and self-determination (SD) dimensions. However, thefear of failing the computer science exam affected their decision-making.
Table 2 Pearson’s correlation coefficient between motivation and twenty-first century
motivation.
Learning Motivations Dimensions

IM

PG

EM

AC

IM (Intrinsic Motivation)

1

0.796** 0.532** 0.319

PG (Personal Goals)

0.796** 1SDSE

0.867** 0.674

0.797** 0.174** 0.798** 0.661**

EM (Extrinsic Motivation)
0.532** 0.797** 1
0.118
AC (Anxiety about Computer
0.319
0.174
0.118
1
Science Assessment)
SD (Self-Determination)
0.867** 0.798** 0.636** 0.397*

0. 397*0.288

10.695**SE (Self-Efficacy)

0.695** 1**

0.674** 0.661** 0.655** 0.288

0.636** 0.655**** Correlation is significant at the 0.01 level (2-tailed).
* Correlation is significant at the 0.05 level (2-tailed).

According to the findings presented in Table 3, the implementation of the learningmodel resulted in significant improvements in the 21st century skills of the student sample
group. Specifically, there was a notable enhancement across five key aspects of these skills:
self-directed learning (SDL), collaborative learning (CoL), meaningful learning with ICT (MLT),
critical thinking (CriT), creative thinking (CreT), authentic problem-solving (APS), andknowledge creation efficacy (KCE). All of these skills showed a statistically significant increaseat the .01 level. Notably, meaningful learning with ICT skills exhibited the highest averagescore of 4.0240, showcasing the notable impact of this learning approach.

922

Table 3 Paired-samples t-test analysis Twenty-first-century skills comparing pre-test andpost-test scores.

SDL
MLT
CoL
CriT
CreT
APS
KCE

Mean

SD.

SE.

Pre-test

3.00800.60685

0.12137

Post-test

4.00800.49826

0.09965

Pre-test

2.85600.623220.12464

Post-test

4.02400.55474

0.11095

Pre-test

3.35200.90052

0.18010

Post-test

4.00800.644150.12883

Pre-test

3.04000.82977

0.16595

Post-test

3.96000.68927

0.13785

Pre-test

3.17000.94846

0.18969

Post-test

3.74000.686480.13730Pre-test

2.74400.564270.11285

Post-test

3.90400.60033

0.12007

Pre-test

2.77600.742340.14847

Post-test

3.82400.666630.13333tp-8.492.000-11.947.000-3.604.001-6.006.000-2.861.009-10.200.000-7.636.000**p < .01

In Table 4, Intrinsic Motivation is positively correlated with most of the twenty-first
century skills. Positive correlations indicate that as motivation increases, the corresponding
twenty-first century skills also tend to increase. Motivation has a positive and significant
correlation with MLT (0.638), CriT (0.775), CreT (0.679), APS (0.265) , SDL (0.443), and KCE(0.388).
In contrast, Motivation shows weaker or non-significant correlations with Social andEmotional Learning (SEL) and Active Citizenship (AC). The correlation coefficients for theseskills range from -0.223 to 0.126, indicating a weaker or negligible relationship between
motivation and these particular skills.
Table 4 Pearson’s correlation coefficient between motivation and twenty-first century skills.
SDL

MLT
**

CoL
**

CriT

CreT

APS

0.443

0.638

0.773

0.775

0.679

0.265

0.388*

PG

0.430

0.691**

0.801

0.732**

0.778**

0.502**

0.591*

EM

0.404**

0.772

0.725

0.602**

0.623**

0.659*

0.701**

AC-0.223

0.085

0.081

0.015*

0.018

0.065

0.126

SD

**

SE

0.422

**

0.645

0.642

0.593

0.377**

0.653**

0.672

0.681**

**

KCE

IM

*

**

**

**

0.633

**

0.443

0.516*

0.672

0.630*

0.666***p < .01

There are positive correlations between most of the twenty-first century skills
factors.Positive correlations indicate that as one skill factor increases, the corresponding skill
factors also tend to increase. The highest positive correlations are observed between
several pairs of skills factors, indicating strong associations between them. For example,
there are strong positive correlations between CoL and CriT (0.820) , CoL and MLT( 0.769),
CoL and CreT(0.697), CoL and SDL (0.493) , CoL and APS (0.433), and Collaboration andKnowledge creation efficacy (KCE) (0.578). These correlations range from 0.697 to 0.820,

923

suggesting that these skills factors tend to be positively related and may complement eachother in practice.
Table 5 Pearson’s correlation coefficient among the seven twenty-first century skills factors.
SDL

MLT

*

CoL
*

CriT

CreT
*

APS

KCE
*

SDL

1

0.518

0.493

0.648

0.475

0.315

0.556

MLT

0.518**

1**

0.769

0.598**

0.460**

0.638*

0.643**

CoL

0.493**

0.769**

1

0.820**

0.697**

0.433*

0.578*

CriT

0.648**

0.598**

0.820

1**

0.732**

0.434**

0.584*

CreT

0.475**

0.460**

0.697

0.732**

1**

0.245*

0.547

APS

*

0.315

**

0.638

0.433

*

0.434

0.245

1

0.743*

KCE

0.556**

0.643**

0.578

0.584**

0.547**

0.743

1*

**

**p < .01

Furthermore, the students were engaged in an open-ended questionnaire thatspecifically centered on identifying activities that they derived pleasure from. The
results of the study revealed that students placed a high level of importance on
engaging in group activities, participating in coding exercises that involved theutilization of Micro:bit boards, and engaging in programming tasks. They cherishedopportunities to initiate projects and independently produce work, particularly inactivities such as Fan Coding and Car Coding. Students appreciated the chance topresent their personal work, especially in the Fan Coding activity where they wereable to program a Micro:bit board to control a fan based on their design
specifications and create fan models reflecting their innovative concepts. In terms ofexperiences that left a lasting impression on the students, it was discovered thatstudents were struck by the teacher's support during activities, the ability to
independently troubleshoot problems, and the collaboration with peers. Students
articulated, "Initially, I was unable to complete the task, but after acquiring knowledge
and making an effort, I felt a sense of pride and enjoyment." This feedback illustrates
that such activities can enhance motivation for learning and cultivate competencies
for the 21st century, specifically in the realms of cooperative and self-directed
learning.
5. Conclusion
Research has shown that Contextual Inquiry Project-based Learning can be effective inpromoting the development of 21st-century skills such as critical thinking, collaboration, andcommunication. Similarly, a previous study by Chen and Chang (2016) found that students
who participated in a Contextual Inquiry Project-based Learning project showed significant
improvements in their critical thinking skills. This aligns with the existing body of research on
the utilization of suitable digital media, which has been shown to enhance student motivationand learning efficiency (Thanyaphongphat, 2019).
These results only show associations between motivation and twenty-first century skills,
but they do not indicate that one directly causes the other. Furthermore, other factors notconsidered in this analysis may also contribute to the development of these skills.
Future research projects could commence by analyzing student behavior throughobservations and conducting in-depth interviews to gather qualitative insights. Employing
this approach would provide a comprehensive understanding of the situation. In subsequentstudies, robust statistical tests could be employed to validate findings, leveraging a larger
participant pool. Additionally, it is recommended that future investigations explore the924adaptability of the Contextual Inquiry Project-based Learning method across various digitallearning contexts. These aspects share commonalities and possess the potential tocontribute to our existing knowledge.

References
Beyer, H., & Holtzblatt, K. (1998). Contextual inquiry: A participatory technique for system design. In
Participatory Design: Principles and Practices (pp. 53-68). Lawrence Erlbaum Associates.
Brophy, J. (2021). Motivating Students to Learn (5th ed.). Routledge.
Chai, C. S., Deng, F., Tsai, P. S., Koh, J. H. L., & Tsai, C. C. (2015). Assessing multidimensional
students’ perceptions of twenty-first-century learning practices. Asia Pacific Education Review.
2015(-), 16, 389-398. http://doi.org/10.1007/s12564-015-9379-4
Chen, P., & Chang, C. (2016). A study of the effectiveness of project-based learning combined with
contextual inquiry on students' critical thinking skills. Eurasia Journal of Mathematics, Scienceand Technology Education, 2016(12(5)), 1155-1165. http://doi.org/10.12973/eurasia.2016.1259a
Griffin, P., McGaw, B., & Care, E. (2018). Assessment and teaching of 21st century skills. Springer
Science & Business Media. 2018(-). https://doi.org/10.1007/978-3-319-65368-6
Jang, S. J., Chen, C. H., & Yen, C. H. (2017). The effects of digital storytelling on student
achievement, social presence, and 21st-century skills in a social studies course. Interactive
Learning Environments, 2017(25(8)), 1058-1072. http://doi.org/10.1080/10494820.2016.1226487
Lee, C. S., & Wu, C. S. (2019). A contextual inquiry project-based learning approach for enhancing
critical thinking skills in computer science education. Computers & Education, 2019(140),
103611. http://doi.org/10.1016/j.compedu.2019.103611
Partnership for 21st Century Skills. (2015). P21 framework definitions. https://www.p21.org/ourwork/p21-framework
Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory
perspective: Definitions, theory, practices, and future directions. Contemporary Educational
Psychology, 2020(61), 101860. https://doi.org/10.1016/j.cedpsych.2020.101860
Thanyaphongphat, J. (2019). Effects of Personalized Learning With Preferred Digital Media Types on
Learning. In Proceedings of 18th European Conference on e-Learning.
https://doi.org/10.34190/eel.19.041
Thanyaphongphat, J., Tapingkae, P., Thongkoo, K., & Daungcharone, K. (2022). Promoting
Computational Thinking with Visualization Programming through Project-Based Learning in
Computer Science. In Proceedings of the 2022 WEI Vienna International Academic Conferenceon Education, Humanities, and Social Sciences (pp. 29-36). Vienna, Austria.
Trilling, B., & Fadel, C. (2009). 21st century skills: Learning for life in our times. Jossey-Bass.
Voogt, J., Erstad, O., Dede, C., & Mishra, P. (2013). Challenges to learning and schooling in thedigital networked world of the 21st century. Journal of Computer Assisted Learning, 2013(29),217-220. http://doi.org/10.1111/jcal.12029
World Economic Forum. (2018). The future of jobs report 2018. World Economic Forum.
https://www.weforum.org/reports/the-future-of-jobs-report-2018.

925

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Data-Driven Competency Assessment
Supporting System for Teachers
Taito KANOa*, Izumi HORIKOSHIb, Kento KOIKEb & Hiroaki OGATAbaGraduate School of Informatics, Kyoto University, Japan
bAcademic Center for Computing and Media Studies, Kyoto University, Japan*kano.taito.87a@st.kyoto-u.ac.jp
Abstract: As many countries seek to promote competency-based education, formative
assessments are important to capture the learning processes of learners. However, as
yet there are no assessments that can fully capture the learning process. Recently, theuse of ICT tools for learning has become more general, and learning log data has been
accumulated. Using these data, it has become possible to capture learning processes
in detail; therefore, data-driven assessment has attracted increasing attention.
However, as conventional data-driven competency assessments require experts tomap data to competencies, they can only be applied in a defined context. In this study,
we proposed an assessment framework that allows teachers to assess their students’competency by freely combining data collected as students used the Learning &Evidence Analytics Framework (LEAF) platform. We created an assessment in a
scenario in an assumed educational setting using the proposed framework andexamined what kind of assessment would be possible. Then, we created a system forthe framework. Finally, interviews were conducted with three teachers regarding thesystem. The results suggest that the system can achieve context-independent andflexible data-driven assessment, contributing to the continuous improvement oflearning and teaching from multiple perspectives in activities that use the system.
Keywords: Competency assessment, data-driven assessment, learning analytics

1. Introduction
Competency-based learning, which emphasizes the development of competencies in K12
education, has been widely implemented (Henri et al., 2017). To develop learners’
competencies, it is important to assess their current competencies during the learning process(Guerrero-Roldán & Noguera, 2018). In doing so, it is possible to assess how they havedemonstrated and developed their competencies. Competency assessments have also been
implemented in K-12 education in Japan (MEXT, 2018). However, most assessment methods
focus on products such as tests, reflection sheets, and portfolios; process-based assessments
have not yet been developed (Guerrero-Roldán & Noguera, 2018).
Recently, ICT tools for learning have been introduced into the educational field, allowing
learners’ activities to be recorded in a database as log data that facilitate capturing the learning
process in more detail than ever before (Sung et al., 2016; Van der Kleij et al., 2015). Therefore,
data-driven competency assessment has attracted considerable attention.
However, in conventional data-driven competency assessments, experts must define thecorrespondence between data and competencies theoretically (Greene & Azevedo, 2010;
Winne & Perry, 2000). Although this approach achieves a robust and highly valid data-driven
competency assessment, it can only assess data as defined and cannot respond to diverse
contexts.
Therefore, we created a framework that allows teachers themselves to assess
competencies by combining data and implementing a data-driven competency assessment
system. We believe that this system will also afford each teacher the flexibility to create
assessments tailored to their context and improve the quality of observation. We addressed
these RQs in this study.
RQ1: How can the data-driven assessment be realized from trace data?
RQ2: How can the data-driven assessment be implemented for teachers' use?

926

RQ3: How do the teachers expect, use and evaluate the data-driven assessment system?

2. Literature Review
2.1 Data-Driven Competency Assessment
To assess competencies such as critical thinking and self-regulated skills (SRSs), it isnecessary to capture the learning process (Guerrero-Roldán & Noguera, 2018). Althoughmany methods have been proposed to capture the learning process, they have certain
limitations. For instance, assessments based on observations of daily learning activities arelimited to what teachers can capture (Greene, 2015). Even if the teacher asks learners to writea reflection sheet, there is still much noise in the sheet, such as the learners’ ability to express
themselves and remember (Renninger & Bachrach, 2015). Among the many methods
available, assessment based on learning log data has attracted attention as appropriate forassessing the learning process (Siadaty et al., 2016). To assess competencies from the data,
the following procedure is necessary.
First, target competency must be defined. Second, the competencies must be subdivided
into activity levels. For example, SRSs are divided by Greene and Azevedo (2009) into
“macro-level” and “micro-level” activity phases. Third, we must determine the activities in thesystem that correspond to the activity phase definitions. For instance, if the target competency
is SRSs, they are subdivided into “Planning,” “Engagement,” and “Evaluation & Reflection”;“Engagement” is subdivided into “Working on the Task” and “Applying Strategy Changes”; and“Working on the Task” is described as “to consistently engage with a learning task using tactics
and strategies” (Siadaty et al., 2016, p. 192). Based on the description, it can be considered
to correspond to an activity such as using a marker function to highlight an important word in“Working on the task”. In this case, the system activity must be recorded as data for assessing
the activity. By linking activities and data in this way, the assessment value of activities can bequantified from the data, from which the competency can be quantified.
However, such assessment processes are mostly conducted by experts and researchers, andtheir applicable context is limited. Essentially, the axis of competency assessment,interpretation of data, and context of assessment can be changed by the teacher. In this study,
we propose a method that allows teachers to flexibly create assessments according to theirown contexts.

2.2 Competency Assessment in Japan
Japan’s K12 program uniquely fosters three pillars of competencies as shown in Table 1 with
their descriptions which are considered based on MEXT (2018). Each pillar consists of multiplecompetencies that are mainly assessed based on learning products such as tests,
presentations, and interview tests.
Table 1. Pillars of Competency and their Descriptions (MEXT, 2018)
Pillar of Competency
Knowledge & Skills
Thinking, Judgment& Expression
Meta-Cognitive &Self-Regulated Skills

Description
How much knowledge and skills learners have acquired
How much acquired knowledge and skills learners can use
How well learners can recognize their state and adjust theirlearning to acquire knowledge and skills

The recent global digitalization of education has also progressed in Japan. All learners
have begun using devices in accordance with the digital education reform known as the GIGA
School Concept (MEXT, 2020). There are high expectations regarding the use of learning-log
data obtained from ICT tools for competency assessment. However, the use of learning-log
data for assessment has not yet been fully realized.

927

3. Method
3.1 Learning & Evidence Analytics Framework (LEAF)
The LEAF system that our laboratory is developing is an integrated learning platform that
integrates LMS (Moodle), e-books (BookRoll), a learning support module, a learning analytics
module (LogPalette), and a database (LRS) (Ogata et al., 2018). LEAF can serve as a sensor
that records learners’ detailed learning behaviors and a database that stores learning logs.
The following table presents example BookRoll log data. In this study, a data-driven
assessment system was created using the learning-log data obtained from LEAF. However,this assessment method can be applied in other environments as well.
Table 2. Example BookRoll Learning Log Data
Operation_time
2022-04-05 13:32:01
2022-04-06 07:14:43
2022-04-06 08:01:02

Student_id
S_1
S_2
S_2

Course_id
C_1
C_1
C_1

Operation_name“ADD_MARKER”“OPEN”“NEXT”

3.2 Research Questions
ίRQ1: How can the data-driven assessment be realized from trace data?
We began by creating an assessment framework that allows teachers to combine dataand create assessments. As an example application of this framework, we attempted toassess learners’ SRSs in a first-year junior high school mathematics course using a set of
created indicators. The results indicated the type of assessment possible using the framework.
ίRQ2: How can the data-driven assessment be implemented for teachers' use?
This section describes the implementation of this assessment. In addition, we describe
some of the support functions added to assist teachers in the assessment.
ίRQ3: How do the teachers expect, use and evaluate the data-driven assessment
system?
We conducted interviews with three Japanese teachers who used the implemented
system. We compared it with the current competency assessment and asked participants
about their expectations and concerns. We then asked them to use the system, and based on
observations of their behavior, we discussed its positive aspects and possible improvements.

4. Results and Discussions
4.1 RQ1: Data-Driven Assessment Framework
Here, the proposed assessment framework is introduced and applied to a given scenario.
4.1.1 Indicators and Data Processing Flow
The following explains the process for calculating the assessment value using trace log data,
as shown in Figure 1.

Trace
Log
(Table 3)

Aggregate

Indicator

Logarithmic
Transformed
Standardized

Transformed
Indicator(Figure 2)(Table 4)

Figure 1. Data Processing Flow

928

Selection
Weighting
Summation

Assessment
Value
Summative(Table 6)
Formative(Figure 3)

First, the indicators were created by aggregating the trace log data obtained from theLEAF. Second, the indicators were narrowed down by the time range of the activity to beassessed, and summed. Next, the logarithmic transformation of each indicator across learners
was perfomed to make the distribution closer to a normal distribution, and standardization was
performed to align the distribution means of different indicators. In this way, the influence ofoutliers can be suppressed, and different indicators can be summed. Finally, by selecting theindicators to be used for assessment and setting weights for each, the values of the selectedindicators were weighted and summed to calculate the assessment value. In conventional
assessment activities, the scores on several tests or the results of a presentation are weighted
and summed to create a rating. Based on these suggestions, ratings were created by adding
weights to the indicators. In addition, summative assessments can be created by aggregating
the indicators over the entire period, and formative assessments by aggregating the indicatorsfor each week.
This procedure was expected to enable not only the generation of learner ratings fromthe data but also continuous monitoring of the changes and growth of learners. Nine indicators
were created in this study, as listed in Table 3.
Table 3. List of Indicators
TS
NT
NJ
NYM
NRM
NB
NM
NHM
NAQ

Indicator Name
Time Spent
Number of Trans
Number of Jumps
Number of Yellow Markers
Number of Red Markers
Number of Bookmarks
Number of Memos
Number of Hand Writing Memos
Number of Attempts of Quiz

Explanation
Time viewed (minutes)
Number of page transitions madeNumber of page jumps madeNumber of times yellow marker was drawn
Number of times red marker was drawn
Number of times bookmarks were used
Number of notes
Number of handwritten notes
Number of quiz responses

4.1.2 Scenario Analysis
To demonstrate the application of the framework, we conducted an assessment with thesenine indicators in a scenario that assumed an actual educational setting as follows:
A teacher teaching a first-grade math course in 2022 will assess his or her students’ firstsemester SRSs. This course will also have a test on June 3. Test scores will be used to assess
Knowledge and Skills; therefore, learning activities in the LEAF from April 1 to June 3 will beassessed as a component of SRSs. Figure 2 shows the results of the assessments of the
SRSs of the three students based on the indicators shown in Table 3 using the assessment
process shown in Figure 1.
Figure 2(a) shows the value of the assessment indicators calculated for the three
students during the entire period. Red and blue lines are drawn for indicators in the top andbottom 30% of the values, respectively. These show that student_1 scored high only in thenumber of red markers, student_2 in four indicators, and student_3 in five indicators. Eachstudent varied in whether indicators were high or low, which may indicate differences instudents’ approaches. We believe that these figures are important for understanding thecharacteristics of learning in LEAF enhanced learning activities.
Second, we considered the actual assessment of SRSs based on these nine indicators.
Among SRSs, we will focus only on Working on the Task activities for the assessment target
because the indicators are considered to be related to Working on the Task, which is describedas “to consistently engage with a learning task using tactics and strategies” (Siadaty et al.,2016, p.192).

929

Figure 2. The Transformed Indicators and the Assessment Values
The activities performed in Working on the Task were further classified into three
categories (Table 4), and each indicator was mapped to a category based on its characteristics.
Table 4. Correspondences between Categories and Indicators
Category
Reading
Applying Strategy
Exercise

Indicator
Viewing time, number of page transitions, number of page jumps
Number of yellow markers, red markers, bookmarks
Number of notes, handwritten notes, quiz responses

The calculated assessment value of each activity was assigned A for the top 30%, B for
the top 30–70%, and C for the remaining activities to be graded. The grades for each activity
from April 1 to June 3 are listed in Figure 2(b). Test grades are also provided for reference
purposes.
The results of this summative assessment provide insights into students’ learning
activities that were not found in test scores alone. For example, student_1, who also scoredhigh on the test, did not perform many Reading or Exercise activities but was highly involvedin the Apply Strategy activity, suggesting that he or she was trying to learn creatively.
Student_3, who has high scores in each activity but a low score on the test, is considered tohave some problem with the way he or she performs the activity. These differences in thelearning activities of each student afford a perspective for assessment that cannot be seenfrom grades alone.
Thus far, we have learned how to incorporate the data obtained from LEAF into a
conventional summative assessment. Such summative assessments provide suggestions on
how to conduct learning activities but do not capture the learning process. Therefore, we
calculated the indicators for each week and performed weighted additions to calculate theassessment values for the time series. Figure 2(c) shows the transitions in the assessment
values of the three activities for all students.
This formative assessment makes it possible to see how long the activities had continued
and when they were decreasing or increasing. For example, student_2 improved on Reading
and Apply Strategy activities from the first week of May to the third week of May, but then930

dropped from the fourth week of May. Student_3 improved on Reading and Apply Strategy
activities as well from the first week of May, and continued to do so until the test. Using a timeseries graph allows us to capture student activities’ real-time changes and also allows us toobserve multiple activities simultaneously. We believe that this will allow for a more
multifaceted and continuous formative assessment in LEAF enhanced classes.
In other cases, when a teacher makes a specific intervention for students in a LEAF
enhanced classroom activity, it is possible to evaluate their intervention based on how thestudent activities have changed. We believe that this will help activate the cycle of classroom
practice and improvement.
To summarize the assessment process, summative and formative assessments werecreated by selecting indicators from a given set of indicators according to their purpose. Thus,
the framework allowed us to flexibly tailor the assessment to each context. Furthermore, this
could serve not only as a grading instrument but also to improve learning and teaching in thesystem.
However, such freedom in assessment may reduce its validity. Similar issues have beenraised in previous studies (Azevedo, 2015; Guerrero-Roldán & Noguera, 2018). Whenassessing competencies based on data, it is necessary to establish a correspondence
between the indicators and competencies. This mapping is mostly performed by experts andresearchers who are familiar with both data and competencies, as we did here, and isconsidered difficult for teachers, whose comprehension of competencies and indicators varies
widely. However, teachers should be able to perform the mapping to create assessments
flexibly according to their own context. That is, it is necessary to consider how teachers canlearn to define competencies and make appropriate correspondences. Accordingly, in thisstudy we incorporated a function to store and share the correspondences made by teachers
between indicators and competencies in the system.

4.2 RQ2: Implementation of the Proposed Assessment Method
Figure 3 shows the three main modules included in the system. The main components of thesystem are the Setting Module, which sets the context of the assessment and selects
indicators; the Sharing Module, which shares information to support the selection of indicatorsin the Setting Module; and the Visualization Module, which visualizes the calculated
assessment values.

Figure 3. Main Modules of the System931

4.2.1 Setting Module and Sharing Module
The Setting Module is shown in Figure 3(a). It is divided into two sections, one for inputting
the context of the assessment and the other for selecting indicators. In the context input
section, the name of the activity to be assessed, competency to be assessed, type of activity,
and assessment period are input. In the indicator selection section, a combination of indicators
can be set to calculate the assessment values by setting a weight of 1 or more for theindicators to be used, and a weight of 0 for the others.
In participatory design systems, such as the Setting Module, where data use is teacherdependent, teachers’ lack of expertise is often problematic (Dilmore et al., 2013; Holt et al.,2015). Sharing prior knowledge of data use with teachers is said to be effective in addressing
this issue (Dollinger et al., 2019). This can complement teachers’ expertise. Furthermore,
showing examples of data use by other teachers improves the reliability of these examples.
Therefore, in this study, after inputting the context information into the Setting Module, thefollowing information in a similar context was shared by pressing the search button in the
Sharing Module (Figure 3(b)).
b-1. The three most recent assessment cases
b-2. The percentage of selected indicatorsb-3. A weighted average of indicators when selectedb-4. Estimated weights predicted by the system
With this information, we attempted to bridge the gap between the nature of theindicators and the teachers’ understanding of them. As the assessments created by the projectcontinue to accumulate as case studies, the generalizability of shared information will increase.
We believe that this will lead to improvements in the validity of the correspondence between
the selected indicators and competencies.
4.2.2 Visualization Module
The Visualization Module, shown in Figure 3(c), provides the visualization of teachergenerated assessments. It is divided into the following two sections
c-1. Visualization of summative assessments
c-2. Visualization of formative assessments
Based on the results obtained in 4-1-2, we considered the requirements for each section.
c-1 has two requirements: (i) multiple assessment values should be visible together, and (ii)the average of the indicators over the entire period should be available. First, to achieve (i),we visualized multiple assessment values on the same graph and table so that all assessment
values could be grasped at once. Second, to achieve (ii), the average values over the entire
assessment period were described in a radar chart and in three levels of ABC, so that the
average characteristics of learning over the period could be captured. Achievement of these
two requirements is expected to lead to a comprehensive understanding of the characteristicsof learning activities.
In c-2, the three requirements were (iii) that multiple assessment values be visible
together, (iv) that the process of changing assessment values be visible, and (v) that it bepossible to see when the change occurred. To achieve (iii), multiple assessment values werevisualized on the same graph as in c-1. To achieve (iv), the assessment values were
calculated according to a time series so that the process of change in the assessment values
could be seen. Finally, to achieve (v), the assessment values were calculated weekly to showin which week the change occurred. Fulfillment of these three requirements is expected tolead to continuous assessments and measurement of the effectiveness of instruction.
In summary, this module has the functionality to perform a comprehensive andcontinuous assessment based on multiple assessment values. It is expected to contribute tothe observation of the characteristics and changes in learning activities and the effects of theirown interventions.

932

4.3 RQ3: Interview about the System
To investigate their impressions of the developed data-based assessment system, weinterviewed three teachers (one math, two English) from high schools that have implemented
the LEAF system.
4.3.1 Expectations and Concerns
Each teacher was asked about his expectations and concerns about the system compared to
traditional competency assessment. As expected, the English teachers responded that it waspossible to assess independent reading activities. The mathematics teacher responded that
there were activities that were not fully captured in the field and that the system could be used
to assess such activities. He also said that it would be good to be able to assess indicatorsthat can only be qualified by data (e.g., average solving time). One concern raised by teachers
in both subject areas was that not all students’ activities were conducted in the LEAF system,and even when they used it, they were not always performing the activities they were
supposed to.
This indicates that teachers expect two things from data-driven assessment: that it willallow them to understand what is beyond their reach, and to quantify students’ activities with
objective measures. We also found that they were concerned that the activities using thesystem would not be fully accomplished. Therefore, data from more sensors should be
converted into indicators to increase their diversity and promote the activities supported by
LEAF.
4.3.2 Experience and Observation
Next, we asked a mathematics teacher to use the system and observe its usage to elicit
reflections of teachers. This was conducted shortly after a demonstration of the system. Thus,
although he had received a brief explanation of the system's functions and purpose, he had
never directly used the system. In the same context as in Section 4.1.2, we asked him to selectindicators for assessing students’ SRSs in exercise activities in two patterns: with and without
the use of the function for sharing assessment cases. In both patterns, he selected indicatorsfrom a state in which nothing is selected. Figure 4 shows the results of the indicator selection
for each case. Indicators not selected for either pattern are not shown.
Without Sharing
0 1 2 3 4 5With Sharing
0 1 2 3 4 5

Browsing Time
Number of Page-Jumps
Number of Handwritten Notes
Number of Attempts of Quiz

Figure 4. Two Patterns of Indicator Selection by the Teacher
Also, while he silently selected indicators from start to finish when he did not use thesharing function, he commented on it as follows.
(u1) “It would be nice to see the title (author's note: name of the indicator), I would like toknow the purpose for which the indicator was created.”(u2) "I guess different subjects have different choices."(u3) "Just a quiz would be fine."(u4) "Even if the others’ selections of indicators are shared, the selection would not beaffected very much."
These differences provide some insight into the effect the feature had on him. First, instatement u3, while he considered quizzes important to assess SRSs, he gave less weight tothe number of quiz attempts than the other indicators without the sharing function. However,933

when using the sharing function, he gave the maximum weight to this indicator. Furthermore,in u4, while he said that the selection was not much affected, the actual selection changed
between the condition with sharing function and that without. These findings suggest that the
information about others’ choices shared by the sharing function provided him with someawareness of which indicators were actually important to him.
Contrarily, a weak point is that, as stated in u1, the name or purpose of the indicator was
not indicated. This implies that to refer to the shared information, it is important that its purpose
matches one’s own purpose. In the future, it is necessary to improve the sharing function andshare the purpose itself so that teachers can refer to the shared information.
Finally, concerning u2 and u3, some statements focused on the differences in subjects
and emphasized the importance of quiz activities that they often use. This suggests thatteachers may choose indicators based on their own context and teaching style. Therefore, wewill examine whether these factors actually influence teachers’ choice of indicators and also
investigate the possible influence of other factors in future research.

5. Conclusion and Future WorkTo achieve data-driven competency assessment by teachers, we created a framework thatallows teachers to freely select indicators and implemented a system that shares information
about other teachers’ assessments. The system differs from conventional data-driven
competency assessment systems in that teachers select their indicators by themselves. While
this has the advantage of allowing teachers to tailor their assessments to their context, it also
decreases the validity of the assessment. However, the accumulation of assessment cases is
expected to contribute to addressing this issue. In addition, the interviews with teachers
revealed that the system seemed to allow the capture of independent learning and detailed
efforts that could not have been captured before. However, it became clear that there were
problems outside the system, such as a lack of sufficient activities using the system.
Furthermore, when teachers were asked to use the system, it was found that the Sharing
Module helped teachers gain awareness regarding indicator selection and that their indicator
selection changed. In the future, we would like to increase the effectiveness of the shared
information by clarifying what teachers consider important in selecting indicators.
In summary, the proposed system and functionalities are expected to achieve a contextindependent and flexible data-driven assessment. Furthermore, it will be possible tocontinuously improve learning and teaching with activities using the system from multiple
perspectives. However, in the specific assessment cases presented in this study, only the logs
obtained from LEAF were used for assessment, and there was a limitation that the dataobtained from off-system activities and other tools were not included in the assessment target.
However, xAPI format-compliant data are available in the system. Therefore, if other tools arealso xAPI compliant, cross-data source assessment will be possible in the system.
Furthermore, if more sensors record off-system activities as data, the system can be extended
to off-system activities. In this way, we hope to make it possible to conduct highly valid
assessments in a variety of contexts.

Acknowledgments
This study is supported by NEDO JPNP20006, JSPS KAKENHI 22K20246, and 23H00505.

References
Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual,
theoretical, methodological, and analytical issues. Educational psychologist, 50(1), 84-94.
Dilmore, T. C., Moore, D. W., & Bjork, Z. (2013). Developing a competency-based educational
structure within clinical and translational science. Clinical and Translational Science, 6(2), 98–
102.
Dollinger, M., Liu, D., Arthars, N., & Lodge, J. M. (2019). Working together in learning analytics
towards the co-creation of value. Journal of Learning Analytics, 6(2), 10-26.

934

Greene, B. A. (2015). Measuring cognitive engagement with self-report scales: Reflections from over
20 years of research. Educational Psychologist, 50(1), 14-30.
Greene, J. A., & Azevedo, R. (2009). A macro-level analysis of SRL processes and their relations to
the acquisition of a sophisticated mental model of a complex system. Contemporary educational
psychology, 34(1), 18-29.
Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive andmetacognitive processes while using computer-based learning environments. Educational
Psychologist, 45(4), 203–209.
Guerrero-Roldán, A. E., & Noguera, I. (2018). A model for aligning assessment with competences andlearning activities in online courses. The Internet and Higher Education, 38, 36-46.
Henri, M., Johnson, M. D., & Nepal, B. (2017). A review of competency-based learning: Tools,
assessments, and recommendations. Journal of engineering education, 106(4), 607-638.
Holt, E. A., Young, C., Keetch, J., Larsen, S., & Mollner, B. (2015). The greatest learning return on
your pedagogical investment: Alignment, assessment or in-class instruction? PLoS One, 10(9).
MEXT. (2018, June 5). Human Resource Development for Society 5.0 ~Changes to Society, Changes
to Learning~.
https://www.mext.go.jp/component/a_menu/other/detail/__icsFiles/afieldfile/2019/09/27/1419637_15.pdf, (ref 2023-8-1).
MEXT. (2020, July 16). The image of the transformation of learning brought by “1 device for 1 studentwith a high-speed network”.
https://www.mext.go.jp/en/content/20200716-mxt_kokusai-000005414_04.pdf, (ref 2023-8-10).
Ogata, H., Majumdar, R., Akcapinar, G., Hasnine, M. N., & Flanagan, B. (2018). Beyond learning
analytics: Framework for technology-enhanced evidence-based education and learning. In 26th
International Conference on Computers in Education Workshop Proceedings (pp. 493-496).
Asia-Pacific Society for Computers in Education (APSCE).
Renninger, K. A., & Bachrach, J. E. (2015). Studying triggers for interest and engagement using
observational methods. Educational Psychologist, 50(1), 58-69.
Siadaty, M., Gasevic, D., & Hatala, M. (2016). Trace-based micro-analytic measurement of selfregulated learning processes. Journal of Learning Analytics, 3(1), 183-214.
Sung, Y. T., Chang, K. E., & Liu, T. C. (2016). The effects of integrating mobile devices with teaching
and learning on students' learning performance: A meta-analysis and research
synthesis. Computers & Education, 94, 252-275.
Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-basedlearning environment on students’ learning outcomes: A meta-analysis. Review of educational
research, 85(4), 475-511.
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In Handbook of selfregulation (pp. 531-566). Academic Press.

935

" !   !" " "   #" !
#" !"$  #" ! #"        


6/,0&#@;40 # $#=0:(10:&#086(20 %
(8(+;(:,$*/6636-5-684(:0*9?6:6&50<,890:?(7(5)*(+,40*,5:,8-68647;:05.(5+,+0($:;+0,9?6:6&50<,890:?(7(55(2(4;8(26/,0 89:2?6:6;(*17  %0170.?4:9>,=0>,4/?:48;=:A0?0,.30=>J;=:10>>4:9,7>6477>$@0>?4:99,4=0>
,9/ A4/0:=0.:=/492 80?3:/> =0<@4=0 ,/A,9.0/ ;=0;,=,?4:9 B34.3 4> :1?09
-@=/09>:80 1:= ?0,.30=> ,9/ >?@/09?> 9 =0.09? D0,=> =0>0,=.30=> 3,A0 ?@=90/ ?:
,9,7DE492?307:2/,?,:1'>D>?08>'34>80?3:/4>70>>/08,9/492,9/307;>.7,=41D
?30 0/@.,?4:9,7 ;=:.0>> 7>: B4?3 8@7?4;70 .7,>>=::8 >D>?08> B0 ;=:;:>0 ,
A4>@,74E,?4:9 80?3:/ ?: ?=,.6 ?0,.30=>J 49>?=@.?4:9,7 ;,??0=9> "@= 2:,7 4> ?: .7,=41D
/4110=09.0>49?0,.3492@>492,.?@,7?0,.30=7:2/,?,?B,>1:@9/?3,??3080?3:/:1
@>492 ?30 >D>?08 /4110=0/ /0;09/492 :9 0,.3 ?0,.30= '34> 80?3:/ ,77:B> @> ?:
48;=:A00/@.,?4:9<@,74?D-D1,.474?,?492?0,.30=>J=0170.?4:9>,9/1:>?0=492/4>.@>>4:9>
B4?3.:770,2@0> 9>?=@.?4:9,7;,??0=9C#?0,.3492,9,7D?4.> 
&0170.?4:9:9?0,.3492;=,.?4.0;7,D>,948;:=?,9?=:7049?0,.30=;=:10>>4:9,7/0A07:;809?
($$:>?3:78
?4>9:?:97D0110.?4A0?:=0170.?:9:90I>0/@.,?4:9,7;=,.?4.0>-@?,7>:
:9?3:>0:1:?30=?0,.30=>0/@.,?4:9,7;=,.?4.0>7049690.3?'.3904/0=:@64>
48:D4,994>
($09.:8;,>>0>A,=4:@>,;;=:,.30>-@?4?>@7?48,?0:-50.?4A04>?:
48;=:A0;=,.?4.0,9/1,.474?,?0>?@/09?/0A07:;809?A,7:>
(30=01:=04?4>90.0>>,=D?:2,?30=/,?,1=:8/,47D.7,>>=::8;=,.?4.092090=,7.7,>>;=,.?4.0/,?,.:770.?4:9:1?09.:770.?0/?480>7:?>,9/80?3:/>>@.3
,><@0>?4:99,4=0>,@/4:,9/A4/0:>30==492?:9 :A0=4/20
:B0A0=?34>;=:.0>>
.,9 -0 -@=/09>:80 1:= ?0,.30=> ,9/ .:>?7D !,=?G90E 0? ,7 (30=01:=0 ?30=0 4>
2=:B49249?0=0>?49@?474E492?0,.30=>I7:2/,?,.:770.?0/1=:8/,47D.7,>>=::8;=,.?4.0:D:>
*07F><@0E := 49>?,9.0 &02@0=,> 0? ,7  @>0/ /,>3-:,=/> ?: A4>@,74E0
?0,.30=>I @?474E,?4:9 :1 70,=9492 8,9,20809? >D>?08>  !' -,>0/ :9 7:2 /,?, ?: >@;;:=?
=0170.?4:9
? 4> 48;:=?,9? ?: =0.:294E0 ?3,? ?0,.30=> :1?09 @>0 8@7?4;70 ( ?::7> 1:= 0110.?4A0
49>?=@.?4:9!,92,=:>6,0?,7:9>0<@09?7D=07D492:9/,?,1=:8,.09?=,7;7,?1:=8>@.3 ,> ,9 !' 8,D 9:? ,..@=,?07D ?30 /D9,84.> :1 , =0,7 .7,>>=::8 C,849492 3:B
?0,.30=> 08;7:D 8@7?4;70 >D>?08> ,9/ ,9,7DE492 ?30 ?0,.3492 ;=:.0>> 4> .=@.4,7 ?: 2,49 ,
.:8;=0309>4A0 @9/0=>?,9/492 :1 ?0,.3492 ;=,.?4.0>  00@B09 0? ,7 := 49>?,9.0
",6,8@=, :=46:>34 ,9/ #2,?,  @?474E0/ C$ ?: ,9,7DE0 ?0,.3492 ;=,.?4.0 ,.=:>>
8@7?4;70(?::7>,9/A4>@,74E0?30;=:.0>>
(30?0,.3492;=,.?4.0,9/;=:.0>>49A:7A049>?=@.?4:9,7;,??0=9>B34.3,=0>?=,?024.
80?3:/>?0,.30=>@>0/-D?0,.30=>?:2@4/0>?@/09?70,=9492 490?,7
.:9/@.?0/,>?@/D :9 49>?=@.?4:9,7 ;,??0=9> 1:= ?0,.3492 ,9/ 70,=9492 ,=2@809?,?4A0 B=4?492 B30=0 ?30D
:->0=A0/ 3:B ?0,.30=> .:8-490 A,=4:@> 80?3:/> ?: ?0,.3 /4110=09? ,=2@809?,?4A0
.:8;:909?>+0,480/?:4/09?41D?3049>?=@.?4:9,7;,??0=9>:1,;,90>05@94:=3423>.3::7
?0,.30=>
9?34>>?@/DB0,48?:.7,=41D?30/4110=09.049?0,.3492-D@?474E4928@7?4;70>D>?08>
+0B47748;=:A0",6,8@=,0?,7I>80?3:/,9/A4>@,74E0?3049>?=@.?4:9,7;,??0=9>:1
?0,.30=>(30=0>0,=.3<@0>?4:9>2@4/492:@=>?@/D,=0,>1:77:B>
&% +3,?649/:1(?::7>/:?0,.30=>@>0?:?0,.3/,47D

936

&% :49>?=@.?4:9,7;,??0=9>/4110=/0;09/492:9?::7@>00C;0=409.0
&% >?30=0,/4110=09.04949>?=@.?4:9,7;,??0=9>@>492(?::7>,8:92?30>,80>@-50.?>

  )% %- % %#0+ *)$.&)" (30 >D>?0842@=0 B,>@>0/?:.:770.? :2/,?,1:=?34>>?@/D(30>D>?08.:8;=4>0>
?3=00>@->D>?08>?3070,=94928,9,20809?>D>?08 !'!::/70?300-::6=0,/0=
::6=:77 ,9/ ?30 70,=9492 ,9,7D>4> ?::7  :2;,70??0 #2,?, 0? ,7 (30>0>@->D>?08>,=049/0;09/09?>D>?08>49?0=.:990.?0/?3=:@23 0,=9492(::7>9?0=:;0=,-474?D ((307:2/,?,1=:80,.3>D>?08B0=0>?:=0/,>C$49?30 0,=9492&0.:=/'?:=0 &'(0,.30=>,=049,/424?,70/@.,?4:9,709A4=:9809?B30=0 .,9-0@>0//,47D-@?4?>@>0
4>,??30?0,.30=I>/4>.=0?4:9#A0=A40B:1?30 >D>?08

 )+  '%+*% &%+/+*(34>>?@/D@>0/:->0=A,?4:9,7/,?,1=:8?30 &',9/49A:7A0/ 5@94:=3423>.3::7?0,.30=>
(30/,?,B0=0.:770.?0/1=:8?30 >D>?08-0?B009;=47 >?,9/'0;?08-0= ?3);:9,9,7DE492/,?,1=:8 ?0,.30=>4?B,>:->0=A0/?3,?8:>??0,.30=>B3:@>0/8@7?4;70
>D>?08>/,47DB0=08,?3?0,.30=>
 +')* %%- *,# 1+ &%+% (,@=492 ?30 >?@/D ;0=4:/ :1  8:9?3> 49
 , ?:?,7 :1  ?0,.30=> 2090=,?0/    >?,?0809?>(:1:.@>:9?30@>0:1(?::7>B0147?0=0/7:2>B30=0?30>0?::7>B0=0@>0/,?
70,>? ?B4.0 (34> =0>@7?0/ 49 ?30 0C?=,.?4:9 :1   .7,>>0> -,>0/ :9 ?30 ?480?,-70
';0.414.,77DB309B00C?=,.?0/?30.7,>>0>:11:@=8,?308,?4.>?0,.30=>B01:@9/?3,??30=0
B0=0 .7,>>0>/@=492?30?,=20?;0=4:/=:8?30>0.7,>>0>B01@=?30=0C?=,.?0/.7,>>0>
?:>;0.414.,77D,9,7DE0?3049>?=@.?4:9,7;,??0=9>B4?349?30>,80>@-50.?>+0,9,7DE0/?30
C$>?,?0809?>,9/0C?=,.?0/491:=8,?4:9:9 ?30>D>?08@>0/ ?30>;0.414.?::7@>0/,9/ ?30?D;0:1:;0=,?4:9;0=1:=80/42@=0 (3=:@23?34>,9,7D>4>B04/09?4140/,?:?,7
:1 ?D;0>:1;=:.0>>0//,?,//4?4:9,77D-,>0/:9?301@9.?4:9>:10,.3>D>?08 !'937

-::6=0,/0=,9/ 0,=94929,7D>4>?::7>,>>3:B94942@=0 B0;=0;,=0/,9/,9,7DE0/?D;0>:1?0,.30=>I-03,A4:=>#& & %& #     
&"$+  ,%#&  

($+&&#&!!#" ($,"&+

 ()

 "
&&#%%& *"&"'###   !&"!+&*#&&#&!!#" &(&*&*#$%"'$! +!
  ,    

!""#
"'"###                  !"($!

                  


!("#$#'(30;=:.0/@=0:1/,?,;,=>492,9/?309@8-0=:1/,?,;:49?>.:770.?0/49?34>>?@/D      
     "  $%
#
  %"#"#$"
$)&'*+ +&'  %"#&'   % !$   %&'  "# %"&')'($#&  #%"&)&'
$&)"
   "%$ !&' #%## &' #%## "$ #%## "!     ,($$ "  "   "   "   "   " "  #" ($" %"  !" "" #" "&  "  "   $""##%%$&"( )##($%   !  " #$ !!" !%#   ##$(($# "  " $ "  "  $ "  "  "     '(& "  (&%% )##&' )## $($#$ " $ "!,%!#($&-!&##%% )###$" )*%

 $"+$& %% )##&$ ##$#"$" ! "$       


938




 

77 ?30 .7,>>4140/ ;=:.0>>0/ /,?, B0=0 ;:49?;=:.0>> /,?, ? 4> 48;:=?,9? ?: 0C;7,49
B34.3 ?::7> B0=0 @>0/ B309 ,9/ 1:= 3:B 7:92 ?: 2,49 , -0??0= @9/0=>?,9/492 :1 ?3049>?=@.?4:9,7 ;=:.0>> 42@=0 477@>?=,?0> ?34> -D =0;=0>09?492 .74.6 0A09?> ,> =0/ /:?> ,9/,//492?30?480-0?B0090A09?>?:/0;4.? ?::7@>,20?480//4?4:9,77D?:4/09?41D ?30 >D>?08
1=:8 B34.3 ?30 /,?, :=4249,?0/ !'=07,?0/ /,?, B0=0 ,>>4290/ ?30 .:7:= :=,920 0-::6
=07,?0//,?,B0=0;=0>09?0/49-7@0,9/,9,7D?4.>?::7=07,?0//,?,B0=0>3:B9492=009 ,.3
0A09?B,>,>>4290/, @94<@0.:7:=(30D,C4>=0;=0>09?>?30;=:.0>>0/,9/.7,>>4140//,?,
,>>3:B94942@=0 B3470?30C,C4>=0;=0>09?>?480 (30?4804>/4A4/0/49?:49?0=A,7> :1 
849 B4?3 0,.3 49?0=A,7 .:==0>;:9/492 ?: 3,71 :1 ?30 ?48492 1=:8 ?30 >?,=? ?: ?30 09/ :1 ?3070>>:949,77DB0A4>@,74E0/?30?0,.3492;=:.0>>,>>3:B94942@=0

    
      



 



  

  


  

 :B?:A4>@,74E0?0,.30=>I ,.?4A4?D1=:8 ;:49?;=:.0>>/,?,$=:;:>0/A4>@,74E,?4:980?3:/+ %#0* *+&

+0;0=1:=80/?3=00,9,7D>0>?:,9>B0=?30?3=00&%>B0>0? 5(3?909 >736805.:/,(03?&9,6-%%663905%,(*/05.%6(59=,8#" @=492,9,7D>4>  B049A0>?42,?0/?30>D>?08>@?474E0/-D?301:@=8,?308,?4.>?0,.30=>@>0/49?304=.7,>>=::8;=,.?4.0,9/.:8;,=0/?308 ?:?30@>,20;,??0=9 :1,77 ?0,.30=>>;,=?
:1?34>,9,7D>4>B0,22=02,?0/,9/,9,7DE0/?30.:8-49,?4:9:10,.3>D>?08,9/?309@8-0=
:1.7,>>0>307/  5(3?909  5(3?@05. 0--,8,5*,9 05 59:8;*:065(3 !(::,859 (9,+ 65 >7,80,5*,9%6(59=,8#" +0 :A0=7,4/ ?30 /4,2=,8> 2090=,?0/ @>492 ?30 ;=:;:>0/ 80?3:/ 42@=0  B4?3 48,20
?=,9>;,=09.D >0? ?:  (30=01:=0 ?30 :A0=7,;;0/ ;:=?4:9 ,;;0,=0/ 49 , /,=6 .:7:= D=010==492?:?30@>,2034>?:=D,9/@>,20:1(?::7>B0.7,=41D?30/4110=09.0>49?0,.30=>I
49>?=@.?4:9>,.=:>>8@7?4;70>D>?08> 5(3?909 %/,+0--,8,5*,905059:8;*:065(37(::,859;905.%:6639%6(59=,8#" 939

(: 49A0>?42,?0 /4110=09.0> 49 49>?=@.?4:9,7 ;,??0=9> B00C,8490/  .7,>>0> ?,@23? -D 1:@=
8,?308,?4.>?0,.30=>B4?349?30/0>429,?0/;0=4:/(30,9,7D>4>49A:7A0/@>492?30;=:;:>0/
A4>@,74E,?4:980?3:/ ?:4/09?41D,9/,9,7DE0A,=4,?4:9>4949>?=@.?4:9,7,;;=:,.30>

 (30:A0=7,D80?3:/1:=A4>@,74E49249>?=@.?4:9,7;,??0=9>  +" %& +&&#*&+)*,*+&+ #0  %#0* *
9,//=0>>492:@=14=>?=0>0,=.3<@0>?4:9B0>:=?0/ ?:2,49,/00;0=@9/0=>?,9/492:1?30
>;0.414.>:1(?::7@>,20(309B0 @9.:A0=0/ ?30;=010=09.0>,9/;=,.?4.0>:1?0,.30=>49
?304=/,47D.7,>>0>42@=0 >3:B>?30>D>?08>08;7:D0/49?0,.30=>I .7,>>0>1=:8;=47 >?
?:'0;?08-0= ?3
 9?30 ,9,7D>4>:1 .7,>>>4?@,?4:9>1:=?30 ?0,.30=> 42@=0 ,
B01:@9/?3,? !'B,>?308:>?1=0<@09?7D@>0/(?::749/0;09/09?7D"
(30>0.:9/8:>?.:88:9.:8-49,?4:9B,> !',9/-::6=0,/0="   
:9A0=>07D42@=0-=0A0,7>?30.:8-49,?4:9 :1 !',9/-::6=0,/0=B,> ?30
8:>? ;:;@7,= >D>?08 .3:4.0 ,8:92 ,77 1:@= 8,?3 ?0,.30=> "   (30 90C?
1=0<@09?7D @>0/ .7,>>0> B0=0 -::6 =0,/0= "    ,9/ .7,>>0> >:707D @?474E492 ,9!' "  (34> 49/4.,?0> ?3,? 8,?308,?4.> ?0,.30=> @>0/ ?30 >D>?08 /4110=09?7D
?3,9?0,.30=>49.3,=20:1:?30=>@-50.?>







 

(&9(.,)?(33:,(*/,89 05  *3(99,9940 

 

  )&9(.,)?-6;84(:/,4(:0*9:,(*/,8905*3(99,9(30@>,20:1 >@->D>?08>49/,47D.7,>>/@=492?30;0=4:/   & %*+),+ &%#'++)%* )'% %&%+&&#,*/') % C;,9/492@;:9:@=494?4,7149/492>B090C?0C;7:=0/B30?30=?0,.30=>I0C;0=409.0B4?3?30
>D>?084917@09.0/?304=49>?=@.?4:9,7;,??0=9>,9/80?3:/:7:240>+0/07A0/49?:?30
;=:1470>:11:@=8,?308,?4.>?0,.30=>,??34>5@94:=3423>.3::7(0,.30=3,A492-009B4?3
?30 49>?4?@?4:9 >49.0
 ,>>@80/ ?30 =:70 :1 ?0,.3492 =/2=,/0 8,?308,?4.> 49!0,9B3470 (0,.30=  5:490/ ?30 49>?4?@?4:9 ,9/ ?,@23? 9/2=,/0 8,?308,?4.> 49(0,.30=3,>-009,;,=?:1?3049>?4?@?4:9>49.0
B3470(0,.30==0.09?7D5:490/?30
49>?4?@?4:949
-:?3?0,.3492 >?2=,/08,?308,?4.>49&02,=/492?304=;=:14.409.D
B4?3 ?30  >D>?08 (0,.30=  >?,9/> :@? B4?3 1:@= D0,=> :1 0C;0=409.0 1:77:B0/ -D
(0,.30=B4?3?B:D0,=>(0,.30=3,>D0,=>I0C;0=409.0B3470(0,.30=4>=07,?4A07D90B
B4?3 70>> ?3,9 , D0,= :1 092,20809?  .:8;=0309>4A0 :A0=A40B :1 ?30>0 ?0,.30=>I
-,.62=:@9/>4>;=0>09?0/49(,-70      

7,>>
$:>4?4:9
>>429809?;0=4:/
@>00C;0=409.0

(0,.30==/2=,/0
>094:=?0,.30=
1=:8D0,=>

(0,.30=
9/2=,/0
(0,.30=
1=:8D0,=

(0,.30=>?2=,/0
?0,.30=
1=:8D0,=>

(0,.30=>?2=,/0
?0,.30=
1=:8
0>>?3,9 D0,=
+0,480/?:A4>@,74E0?0,.30=>I49>?=@.?4:9,7;,??0=9>1=:8/,47D?0,.34927:2/,?,?:
@9/0=>?,9/ ?304= 0C;0=409.0 ,9/ 1,8474,=4?D B4?3 ?30  >D>?08 +0 @>0/ ?30 :A0=7,D
80?3:/;=:;:>0/4942@=042@=0>3:B>,A4>@,74E,?4:9:1,77.7,>>0>.:9/@.?0//@=492
?30>;0.4140/;0=4:/9?34>.,>0(0,.30=B3:?0,.30> =/2=,/0@>0/70,=9492,9,7D>4>
?::7>1:=,7:92?480,9/?09/0/?:@>0?30849?3014=>?3,71:1?30.7,>>42@=0,!:=0:A0=
B309 .:8;,=0/ ?: 42@=0 - . ,9/ / 4? 4> ,;;,=09? ?3,? (0,.30=  @>0/ 8@7?4;70
1@9.?4:9>>@.3,>1:=@8>,9//4>.@>>4:9>/@=492?34>;0=4:/(30>0=0>@7?>>@220>??3,??30
@>0:1(?::7>.3,920>,>?30>.3::7D0,=;=:2=0>>0>=0>@7?49249?3049.7@>4:9:1/4A0=>0
49>?=@.?4:9,780?3:/>941

,(0,.30=-(0,.30=.(0,.30=/(0,.30=#A0=7,D0/49>?=@.?4:9,7;,??0=9>:10,.3?0,.30=49,77.7,>>0>  *+) )% % %*+),+ &%#'++)%*,* % +&&#*$&%+*$*,!+* 9:@=,9,7D>4>:1?0,.30=>@?474E492A,=4:@>(?::7>49.7,>>0>B0,9,7DE0/?30>;0.414.
?::7> ?30D >;0.414.,77D @>0/ (30 -=0,6/:B9 :1 ?30>0  .7,>>0> 4> ,> 1:77:B> (0,.30= .:9/@.?0/ .7,>>0>B3470(0,.30=.7,>>0>.:9/@.?0/ .7,>>0>//4?4:9,77D?30=0B0=0
.7,>>0>?,@23?-D?0,.30=,9/?0,.30=.:8-490/(,-70 
:77:B492?30,9,7D>4>:1.7,>>0>B04/09?4140/?B:/4>?49.?.,?02:=40>49?0,.30=>I
49>?=@.?4:9,7;,??0=9>4=>?7D?30=0B0=0.7,>>0>?3,?>:707D@?474E0/0-::6>"   '0.:9/7D?30=0B0=0.7,>>0>@>492 !',9/0-::6>" +0?309,9,7DE0/?30
/4110=09.0>4949>?=@.?4:9>@>492?30;=:;:>0/A4>@,74E,?4:980?3:/        
(0,.30= "@8-0=:1.7,>>0>
0,=9492)94?>
@=,?4:9
'3,;0>,9/'4847,=4?D (0,.30=>
(0C?-::6;;  
9A0>?42,?0'3,;0>
(0,.30=(0C?-::6;;  $=:;:=?4:9,9/'3,;0> (0,.30=(0C?-::6;;  $=:;:=?4:9,9/'3,;0> (0,.30=(0C?-::6;;  

!4/;=47!4/!,D

!4/!,D!4/@90,=7D@90,=7D'0;?08-0=,=7D@90,=7D'0;?08-0=


42@=0;=0>09?>,.,>0>?@/D:1,.7,>>?3,?0C.7@>4A07D@>0/0-::6>(30(0,.30=>I
@>,20:10-::6.,9-0.7,>>4140/49?:?B:.,?02:=40>-,>0/:9 ?30?48492:1?0C?-::6
@>0(3014=>?;,??0=9@>0/?30?::7,77?30?480/@=492?30.7,>>" ,>>3:B949
42@=0,,9/?30:?30=@>0/?30?::73,71?30?480"   ,>>3:B94942@=0-

942





            

42@=07,>>0>@>0:97D0-::6>" .7,>>0>-D1:@=?0,.30=>
9.:9?=,>?42@=0 477@>?=,?0>?30.,?02:=4E,?4:9:1.7,>>0>?3,?@?474E0/-:?3 !',9/-::6>(30,9,7D>4>=0A0,7>?3,??308,5:=4?D:1.7,>>0>;=0/:849,9?7D=0740/:9?B:
?::7>?0C?-::6,9/.:@=>0A40B0/" ,>/08:9>?=,?0/4942@=0 ,
:B0A0= , >8,770= >@->0? ,> ;=0>09?0/ 49 42@=0 - 49.:=;:=,?0/ ,9 ,//4?4:9,7 ?::7
99:?,?4:9=0>@7?49249?30@>0:1?3=00/4>?49.??::7>49?:?,7" ,?B:?D;0>:1-03,A4:= ,9/ 

-?3=00?D;0>:1-03,A4:=  ,9/

42@=0 7,>>0>@>492 !',9/0-::6>?:20?30=".7,>>0>-D1:@=?0,.30=>
 ,9/49/4.,?0?309,80>:1?0,.30=>I-03,A4:=>>3:B94942@=0  

(30>?@/D,480/?:.7,=41D?30/4110=09.0>49.7,>>49>?=@.?4:9@>4928@7?4;70>D>?08>?:>@;;:=?
?0,.30=>I=0170.?4:9>:9?0,.3492;=,.?4.0>1:=($(3=:@23:@=,9,7D>4>B04/09?4140/?3,?
?30@>,20:1(?::7>/4110=>/0;09/492:9?30>@-50.?:B0A0=?30=0B,>,.:88:9,74?D
49?::7@>,20B4?349?30>,80>@-50.?@=?30=8:=0B0:->0=A0/?3,??30@>,20,9/?48492:1
(?::7>/4110=0/-,>0/:9?300C;0=409.0,9/=:70:1?30?0,.30=
&0170.?4:9 1=:8 0C;0=409.0 ,9/ ;=,.?4.0 4> 48;:=?,9? ?: ;=:8:?0 -03,A4:=,7 .3,920
,9//0.4>4:98,6492,9/48;=:A0?0,.30=>I;=:10>>4:9,7 >6477>$=40?:0?,7:B0A0=
49?30.:9?0C?:1,9,7DE492?0,.30=>I,.?4A4?40>?30D1:.@>3,>-009:98:94?:=492?30>?@/09?>
=,?30= ?3,9 .7:>07D :->0=A492 ?0,.30=>I ,.?4A4?40> "/@6B0  ,9407 (30=01:=0
?0,.30=> 3,A0 ?=,/4?4:9,77D =0740/ :9 <@0>?4:99,4=0> ,9/ A4/0:> ?: =0170.? :9 ?304= ;=,.?4.0
:B0A0=?3080?3:/;=:;:>0/49?34>=0>0,=.3:110=>,9:A07,;;=:,.3?3,?,77:B>?0,.30=>
?:=0170.?:9?304=.7,>>0>B4?3:@?=0<@4=492,9D>;0.4,7;=0;,=,?4:9@>492(?::7>/,47D(34>
80?3:/ ,7>: 1,.474?,?0> ?30 >3,=492 ,9/ /4>.@>>4:9 :1 ?0,.3492 ;=,.?4.0> B4?3 :?30=>

943

>>09?4,77D ?34> =0>0,=.3 >@;;:=?> ?0,.30=>I =0170.?4:9> B4?3:@? ?30 900/ 1:= 0C?09>4A0
;=0;,=,?4:9(3=0060D149/492>.:80:@?:1?34>>?@/D  #" '/(:205+6-%:6639+6:,(*/,89;9,:6:,(*/+(03?
#@=149/492>@9/0=>.:=0?30A,7@0:1/,47D7:2/,?,49@9/0=>?,9/492?0,.3492;=,.?4.0>+3470
:@=>?@/D/4/9:?/07A049?:>;0.414.?::7>4?34237423?0/?30A,=4,9.049?::7@>,20;,??0=9>
-,>0/ :9 ?0,.30=>I -,.62=:@9/> ? -0.,80 .70,= ?3,? /4110=09? ?0,.30=> @>0/ ?30 >,80
>D>?08/4110=09?7D
  #" 6059:8;*:065(37(::,859+0--,8+,7,5+05.65:663;9,,>7,80,5*,
&0>0,=.3<@0>?4:9 1:.@>0/:91:@=8,?308,?4.>?0,.30=>,9/,480/?:4/09?41D;,??0=9>49
3:B?0,.30=>@>0/(?::7>-,>0/:9?304=?09@=0,9/=:70(:,..:8;74>3?34>B008;7:D0/?30:A0=7,D80?3:/;=:;:>0/49?30>?@/D(30=0>@7?>49/4.,?0/?3,?>094:=0/@.,?:=>;0=3,;>
>0,>:90/-D?304=;=:7:920/49?0=,.?4:9B4?3?30>D>?08>008?:0C34-4?,-=:,/0==0;0=?:4=0
49?304=?0,.3492,;;=:,.3!0,9B347090B0=09?=,9?>,;;0,=0/?:-0:9,70,=9492.@=A0 #" 9:/,8,(+0--,8,5*,05059:8;*:065(37(::,859;905.%:6639(465.:/,9(4,9;)1,*:99 =0>0,=.3 <@0>?4:9  B0 .:9/@.?0/ ,9 ,9,7D>4> :1 ?30 @>,20 :1 >;0.414. ?::7> 49 
8,?308,?4.>.7,>>0>#@==0>0,=.3149/492>49/4.,?0?3,??30=0,=0A,=4,?4:9>49?0,.3492>?D70>
B4?3?30>,80>@-50.?+0.,90C;7:=0/4110=09.0>49?0,.3492>?D70>1=:80C?09>4A0/,47D7:2
/,?,-D@>492?34>A4>@,74E,?4:9D4/09?41D492?30/4110=09.0>49?0,.3492>?D70>/0;09/492:9
?30>@-50.??0,.30=>.,91:.@>:948;:=?,9?/4110=09.0>49;=,.?4.0>,9/=0170.?:9?308
(34> >?@/D 3,> .0=?,49 7484?,?4:9> ?3,? >3:@7/ -0 ,.69:B70/20/ 4=>?7D ?304/09?414.,?4:9:1.7,>>0>B,>-,>0/:9=@70>/,?0>,9/?480?,-70>0C?=,.?0/1=:87:2/,?,
//4?4:9,77D?300>?48,?4:9:149>?=@.?4:9?D;0>B,>,7>:-,>0/:9?307:2/,?,!:=0:A0=?30
=0>@7?>:1?34>>?@/D,=07484?0/?:,;,90>05@94:=3423>.3::7?0,.30=>#9?30:?30=3,9/
B03,A0;=0>09?0/,80?3:/1:=A4>@,74E492C$7:2/,?,(30=01:=0?34>80?3:/3,>9:?D0?
-0090A,7@,?0/-D?0,.30=>9?301@?@=0B0;7,9?:49A:7A0?0,.30=>49?300A,7@,?4:9:1?30;=:;:>0/80?3:/
9 ?30 1@?@=0 B0 49?09/ ?: 0C?09/ ?30 ;=:;:>0/ 80?3:/ ?: 49.7@/0 :?30= ?0,.3492
8,?0=4,7>,9/>@-50.?>(0,.30=>@?474E0A,=4:@>?::7>>@.3,>(?:/074A0=?30.:9?09?:1
?304= >@-50.?> (30=01:=0 ?30 @>0 :1 ( ?::7> 8,D A,=D /0;09/492 :9 ?30 >;0.414. .:9?09?
-0492?,@23?(30;=:;:>0/80?3:/3,>?30;:?09?4,7?:0C?=,.??0,.30=>I49>?=@.?4:9,7;,??0=9>
B4?349?30>,80>@-50.?>D@>492(?::7>4?-0.:80>;:>>4-70?:4/09?41D0110.?4A0?0,.3492
80?3:/>?3=:@237:2/,?,,9,7D>4> 
9?34>>?@/DB0093,9.0/",6,8@=,0?,7I>
80?3:/?:>@;;:=??0,.30=>I=0170.?4:9>
:9 .7,>> ;=,.?4.0 ,9/ .7,=41D ?30 /4110=09.0> 49 .7,>> 49>?=@.?4:9 @>492 8@7?4;70 >D>?08> D
,;;7D492?34>80?3:/?:?30C$7:2/,?,:1,;,90>05@94:=3423>.3::7?0,.30=>B0B0=0
,-70 ?: A4>@,74E0 ?30 ?::7> @?474E0/ -D ?30 ?0,.30=> ,.=:>> 8@7?4;70 >D>?08> :B0A0= ?30$=:;:>0/A4>@,74E,?4:980?3:/B,>9:?0A,7@,?0/>:B07,.649>423?49?:3:B=0,7?0,.30=>
8423?@>0?30A4>@,74E,?4:99?301@?@=04?4>90.0>>,=D?:;=0>09??30=0>@7?>A4>@,74E0/-D
?34> ;=:;:>0/ 80?3:/ ?: ,.?@,7 ?0,.30=> ,9/ ,>>0>> ?304= 0110.?4A090>> D 08;7:D492 ?30;=:;:>0/80?3:/?0,.30=>.,9=0170.?:9?304=/,47D@>0:1(?::7>-,>0/:9?307:220//,?,B34.33,>?30;:?09?4,7?:48;=:A0?304=;=:10>>4:9,7>6477> 

(34>>?@/DB,>>@;;:=?0/-D"#$"$$   '$'"$


944



,9/

A,7:>  '0,.30=#=:10>>4:9,70A07:;809?49'0,.3492,9/'0,.30=/@.,?4:9:A0='09
+0,=>'0,.3492,9/'0,.30=/@.,?4:9   I 
30==492?:9&:A0=4/20  (>492)4/0:?:#=:8:?0,=7D347/3::/'0,.30=>J'3496492,9/
%0170.?4:9'0,.3492,9/'0,.30=/@.,?4:9  I 
:D:>     )07,><@0E      '0,.3492 9,7D?4.> @==09? 3,770920> ,9/ @?@=00A07:;809?%'   I
7049690.3?   &.3904/0=     *3,? /: ?0,.30=> ?3496 ,9/ 1007 B309 ,9,7DE492 A4/0:> :1
?308>07A0>,9/:?30=?0,.30=>?0,.3492'0,.3492,9/'0,.30=/@.,?4:9   !,6,8@=,:=46:>34,9/"2,?,  '0,.34929,7D?4.>,.=:>> @7?4;70&D>?08>,>0
&?@/D ,? , @94:= 423 &.3::7 49 ,;,9 #=:.00/492> :1 ?3 9?0=9,?4:9,7 :910=09.0 :9
:8;@?0=>49/@.,?4:9  >4,#,.414.&:.40?D1:=:8;@?0=>49/@.,?4:9#&
):7 :@64> !  48:D4,994>     ""& 1:= '0,.30= #=:10>>4:9,7 0A07:;809? C;7:=492
'0,.30=>J #0=.0;?4:9> ,9/ .340A0809?> 9?0=,.?4A0 '0.39:7:2D ,9/ &8,=? /@.,?4:9   I 
49 ' !,2;,7  ),90=04/0  , & +  !0B077     9>?=@.?4:9,7 ;,??0=9> 1:= ?30
?0,.3492,9/70,=9492:1,=2@809?,?4A0B=4?492493423>.3::79274>37,92@,20,=?>.7,>>=::8>
9?0=,.?4A0'0.39:7:2D,9/&8,=?/@.,?4:9  I  
,92,=:>6,)0>49:>?,6:>)=@>47:A>6D#4,99,6:>   =.34?0.?492,9,7D?4.>
,.=:>>8@7?4;70070,=9492>D>?08>?:093,9.070,=9492/0>429'=,9>,.?4:9>:90,=9492
'0.39:7:240>    I 
,=?G90E   7:>0=  &=494A,>,9  &?0.30=   /078,9     0A07:;492 &4?@,?0/
0,>@=0> :1 &.409.0 9>?=@.?4:9 '3=:@23 ,9 99:A,?4A0 70.?=:94. #:=?1:74: ;; 1:= :-470
0A4.0>%074,-474?D),74/4?D,9/0,>4-474?D/@.,?4:9,7,9/#>D.3:7:24.,7 0,>@=0809? 
I  !/@6B0,9407  '0,.3492,9,7D?4.>A,7@0,9/?::7>1:=?0,.30=/,?,74?0=,.D>D>?08,?4. ,9/ ?=4;,=?4?0 ,;;=:,.3 9?0=9,?4:9,7 :@=9,7 :1 /@.,?4:9,7 '0.39:7:2D 49 4230=
/@.,?4:9  "2,?,  ,5@8/,= % 6F,;K9,=  ,>9490 !  7,9,2,9     0D:9/ 0,=9492
9,7D?4.> =,80B:=6 1:= '0.39:7:2D93,9.0/ A4/09.0,>0/ /@.,?4:9 ,9/ 0,=9492 49
#=:.00/492>:1?30 ?39?0=9,?4:9,7:910=09.0:9:8;@?0=>49/@.,?4:9  #:>?3:78  '0,.30=>/0A07:;492;=,.?4.0%0170.?4:9,>60D,.?4A4?D'0,.3492,9/'0,.30=
/@.,?4:9     #=40?:# ,29@>:9#47709-:@=2#&,,=   %0170.?4:91:=,.?4:90>429492?::7>?:>@;;:=??0,.30==0170.?4:9:90A0=D/,D0A4/09.0'0.39:7:2D#0/,2:2D,9//@.,?4:9  I  %02@0=,> 0>@>)0=/@ /0,>?=:  =@70-,>0/0C;0=?>D>?081:=?0,.30=>J
.0=?414.,?4:9 49 ?30 @>0 :1 70,=9492 8,9,20809? >D>?08> 9?0=9,?4:9,7 5:@=9,7 :1 49?0=,.?4A0
8@7?480/4,,9/,=?414.4,749?0774209.0  A,900@B09%@8807!A,9:2'  *3,?491:=8,?4:9>3:@7/.>.7?0,.30=/,>3-:,=/>
;=:A4/0 ?: 307; ?0,.30=> 49?0=;=0? .>.7 >4?@,?4:9> 9?0=9,?4:9,7 :@=9,7 :1 :8;@?0=&@;;:=?0/
:77,-:=,?4A00,=9492     945

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31 st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Sustaining students’ interest in an
instructional system design course byleveraging interest-driven creator theory
Vishwas BADHEa*, Sunita RASTEa, Sahana MURTHYa, and Sridhar IYERaa
Indian Institute of Technology, Bombay, India*vishwasbadhe@iitb.ac.in
Abstract: In semester-long courses, the trajectory of students' interest hits multiple
highs and lows as the course progresses. In general, sustaining students' interest is achallenge, which becomes even more important when there is a real-world projectwith an expected deliverable. This study emphasizes investigating how instructors’
strategies help in sustaining students’ interest in instructional system design (ISD)
course. The course participants developed a real-world project involving training
resources for sensitization about the needs and challenges of persons withdisabilities (PwD) in an educational institute. We have analyzed interview data offifteen students and one instructor, survey data of the same students, and In-classobservations by researchers. We have identified instructors’ strategies which were
implemented thoughtfully and supported by interest-driven creator (IDC) theory. We
have mapped indicators from student responses to the components of the interest
cycle i.e. triggering, immersing, and extending. The findings of the study show thatcourse strategies supported by IDC theory played an important role in sustaining thestudents’ interest throughout the ISD course while ensuring both conceptual learningand project execution.
Keywords: Interest, Instructional systems design (ISD), Interest-driven creator (IDC)
theory, project-based learning, persons with disabilities (PwD)

1.

Introduction

Design is a process of engaging in a disciplined inquiry for creating something new withpractical utility. Instructional design (ID) is a subset of design, directed toward the practical
purpose of learning (Rowland, 1993). A typical instructional system design course covers aconceptual understanding of instructional design (ID) models, application of the models,understanding of different aspects of an ID project execution, etc. In instructional design, it’s
important that students not only understand the concepts but should also be able to follow
and carry out the processes outlined in a model (for example, analysis, design,development, implementation, and evaluation in the ADDIE cycle). To achieve this, many
instructional system design (ISD) courses follow a project-based learning approach wherestudents get an opportunity to apply and test their conceptual understanding. In projectbased learning, students engage with the problem, learn by doing, discuss, and apply ideas
to solve the problem given to them. This increases students’ engagement and helps them todevelop a deeper understanding of important ideas by facilitating opportunities for problemsolving, decision-making, and explaining their ideas (Krajcik & Shin, 2014). However, in asemester-long project students’ interest keeps varying due to various reasons, and possibly
students might end up losing interest. Sustaining students’ interest in a semester-longproject becomes a key challenge. The blind emphasis given to exams or tests acts as an
impediment to the learning process where students miss the opportunity to develop aninterest in the subject material and don't prepare for ill-structured real-world problems (Chanet al, 2018).
In spring 2023, the ISD course instructor from a premium educational institute inIndia proposed a real-world problem of designing a training website for the awareness andsensitization of the institute community towards the challenges and needs of persons with946

disability (PwD). Rohm et al (2021) suggest that the authentic real-world problems selectedin project-based learning give more relevant learning opportunities to the students. The PwD
project was highly relevant for ISD course students and involved stakeholders within theimmediate social spheres of students in the institute. The real-time usability of a project by
the institute community and the opportunity to make an impact at scale could catalyze
students' interest in the course and project. The PwD project was real-world in nature anddemanded a high level of interest and time to execute tasks and achieve a certain level oflearning. Thus, the interest-driven creator (IDC) theory was chosen as a suitable basis forthe project execution in the ISD course.
Interest is an important component of learning and it is conceptualized as an affective
state which can arise from situational triggers (Ainley, 2006). Improved learners’ interest in
the topics can make learning easier, more effective, and more enjoyable. According to Chanet al (2018), learning as a process entails interest, creation, and habit, thus learners whodevelop the habits of creating out of interest will go on to become interest-driven creators.
The IDC theory is useful to prepare learners for ill-structured and unpredictable real-world
problems (Chan et al, 2018). The interest loop of IDC theory contains three components thatare important in sustaining interest. The first component is "triggering interest," which isfacilitating a task that sparks initial interest in learning, the second component is “immersing”
learners in learning activities that capture students' attention and put them in a state of flow,
and the third component of the interest loop is “extending” which involve activities to extend
learners' interest refers to integration of prior and new knowledge in learning activities (Wonget al., 2020). The IDC theory further talks about the create loop that includes imitating
existing knowledge, combining different concepts to generate new ideas, and staging thework for getting feedback from peers (Chan et al., 2018). The third anchored concept in IDC
is the habit loop consisting of a cuing environment to automate behavior, the routine ofrepetitive behavioral patterns, and harmony as a result of habit activation (Wong et al.,2020). However, the scope of this paper is limited to students’ interest loop hence we will befocusing on how the course-level instructors’ strategies helped in sustaining students’interest in ISD course.
The research study was set in the premise of the ISD course where research andteaching assistants observed the creation and development of the PwD project (website) as
an outcome of students' interest throughout the semester. The research question (RQ)
under investigation is-How did the course level instructors’ strategies help in sustaining
students’ interest? To strengthen and validate our observations, we identified the indicatorsof students' sustained interest through interviews (with students and instructor), self-reportsurveys, and In-class observations by researchers.

2.

Related Work and Theoretical Basis

To prepare students for the future challenges, the what, why, and how of learning needs tochange. An inevitable and permanent change is taking place in learning as it moves fromteacher-centered to student-centered learning. Learning becomes enjoyable and effective
when students learn with interest, creation further makes learning productive and successful,and instilling the habits of creating with interest can lead to students becoming lifelong
interest-driven creators (Chan et al., 2018). Thus, IDC theory focuses on three anchoredconcepts: interest, creation, and habit. In The Merriam–Webster's Dictionary “interest” isdefined as “a feeling of wanting to learn more about something or to be involved insomething,” It has been argued that interest is crucial in guiding behavior and attention andthat it promotes learning and inspires effort (Dewey, J. 1913).
The four-phase model for interest development specifies the following phase:triggered situational interest, maintained situational interest, emerging individual interest, andwell-developed individual interest (Hidi & Renninger, 2006). Loosely mapped to thesephases, the IDC theory proposed a three-component “interest loop” to guide thedevelopment of a coherent learning process: triggering interest, immersing interest, andextending interest (Chan et al., 2018). “Curiosity” for triggering interest, “flow” for immersing
interest, and “meaningfulness” for extending interest were proposed as suitable indicators by947

Wong et al, 2020 with a goal of explaining design factors to be taken into account for earlyinterest development. The action words—triggering, immersing, and extending, call for
educators to use them as springboards for student-centered learning that results in deeper
immersion in students' interests (Roschelle & Burke, 2019).
IDC theory has been implemented in different regions throughout Asia. In Malaysia,
IDC was implemented in educational technology courses for instilling innovativeness among
learners (Khambari, 2019) and ignite students' interest in educational technology (Wong etal, 2020). In Taiwan, IDC has been implemented in an experimental school with subjects
reading, writing, math, science, English language, and social studies (Wong & Wong, 2019).
Kong (2006) used IDC theory to develop computational thinking for K-12 learners. IDC
provides a theory of learning design for Asia, hypothesizing that learning activities that are
created using the IDC approach will help students become more interested in learning,
immerse in the creation process, and reinforce their habit of creating (Chan et al., 2018).
This brings us to the fit of IDC to our own context, where the goal was to create training
resources for the institute community to improve sensitization and awareness about theneeds and challenges of PwD as a real-world ID project. We have used IDC theory forsustaining students' interest in a semester-long course on instructional system design (ISD).

3.

Course Design and Orchestration

Instructional system design (ISD) is a post-graduate level course in the curriculum of theeducational technology master and doctoral program. This 6-credit, semester-long course
typically covers instructional design (ID) models, tools, and technologies for creating
instructional materials, processes followed in ID projects for developing instructional
materials, etc. The learning outcome of this course is to enable students to apply given
instructional design models. The latest offering of the ISD course was in January 2023
having a total of 22 students registered. Students were categorized into 3 categories- credit,
audit and sit through based on their prior experience with the ISD course and registration
status. Freshman students attending the course for the first time and to be graded werecalled credit students. Senior students attending the ISD course for the first time and fulfilling
all the course and project requirements except formal grading were called audit students.
Students who have attended the course previously, but volunteered to participate in thecourse and project out of interest were given the role of mentors and called the sit-throughstudents.
The course was delivered in physical mode with a hybrid set-up for smooth executionof the multi-cohort setting. As the course was run in a hybrid setup, the Zoom videoconferencing platform was used for synchronous class sessions. Total of 14 classes of 3hours each were conducted over a period of sixteen weeks. Weekly 3 hours of class time
were allotted for both conceptual learning and project work. For the conceptual classes, oneclassroom was occupied by all students for 3 hours. For the group work activities under the
PwD project, 4 different rooms were set up for each group to collaborate with groupmembers. One dedicated research assistant was assigned in each class to record classobservations. The physical group work was video recorded and an online meeting was also
recorded on the Zoom platform. The recording of data and observations were managed by
the research and teaching assistants throughout the course.
Various break-out activities were designed by the instructor for facilitating group
discussions during class hours. In each class, 20-30 minutes were reserved forpresentations and updates by groups followed by feedback and suggestions by the instructor
and students. The instructor has organized three feedback/interaction sessions with subject
matter experts (SME), and PwD persons from the institute. The instructor and course teamhas also organized a 3-hour long workshop by leading organizations in the field of PwD
rehabilitation in order to facilitate a deeper understanding of the needs and challenges of
PwDs. The class was divided into 4 groups each working on a different type of disability to
be covered under the PwD project (website). Figure 1 depicts the different topics covered
under each module of the PwD website and the corresponding group composition.

948

Figure 1. Types of disabilities covered under the PwD website and group composition
For adding instructional material under each topic, a template that emerged as aresult of the class discussion was followed. Each group has used suitable elements forpresenting a certain type of content, for example using a poster to depict challenges andsupport strategies. Combining the inputs and elements proposed by each group, a common
template was proposed by the course team for maintaining structural uniformity across the
topics. The template divided the website into four sections with the first section covering an
introductory video of the topic e.g. “What is attention deficit hyperactivity disorder (ADHD)
and learning disability (LD)?”. The second section includes a graphics interchange format(GIF) for depicting the point of view (POV) of a person with a disability. The third section
would cover the challenges faced by persons with disability, in the form of a poster, and thefourth section covering support strategies for PwD persons in the institute. The project was
situated in an educational institute hence, the key requirement was to contextualize thetraining resources keeping the institute set-up in mind.

4.

Method

4.1 Participants and Data Collection
A total of 22 students registered for the course, of which, 10 students registered for credit, 5
students for audit, and 7 senior students attended the course in the “sit through'' mode andcontributed towards the PwD project. In the ISD course, stakeholders and end-users wereclosely connected with students through the PwD website project. Figure 2 gives an
overview of the timeline and sources of data collected in the longitudinal study. The research
assistants noted in-class observations throughout the course, from the IDC theory
implementation perspective. To know the instructors' strategies to sustain interest in the ISD
course, we conducted one semi-structured interview with the course instructor at the end of
the course.

Figure 2: Timeline of course, project execution, and data collection949

We have administered an Interest survey to understand students' interest in thecourse. Students' interest survey questionnaire, designed by a senior post-doctoral
researcher who was also a course team member, was administered post-mid-sem exam.
This interest survey consists of a total of 7 questions with 4 open-ended and 3 Likert scalequestions. For the Likert scale questions, students were asked to respond on a scale of 1-5
corresponding to strongly agree to strongly disagree. We also conducted semi-structured
interviews (after the end of the semester) with credit and audit students to understand theirinterest in the course and PwD project.

4.2 Data Analysis
The PwD project details and subtasks were priorly discussed with all the students in theclass. All the ISD students were divided into four teams and each team had 4-6 members.
Each team consisted of credit, audit, and sit-through students. For this study, we havesampled credit and audit students. To Investigate the students’ sustained interest, we haveanalyzed responses from semi-structured interviews and one survey. The grounded theory
approach in content analysis (Mayring, 2015, p. 16) was followed to analyze Interview andsurvey data. We have done deductive coding of the interview and survey responses. Threecomponents of the interest cycle i.e. triggering, immersing, and extending were used ascodes and we have mapped the students’ responses to these codes. Extending the
grounded theory approach in content analysis, we have analyzed instructors’ interview
responses to investigate the instructor's strategies. To gather additional insights about
instructors’ strategy implementation in the classroom and its impact on students’ interest, wehave analyzed observational data by researchers following the observational protocol
approach (Mayring, 2015, p. 48). The analysis of observational data was exploratory innature aiming to find more micro-level evidence from the classroom.

5.

Findings

This section contains findings on how the course activities and instructors’ strategies help insustaining students’ interest. Section 5.1 contains an episode showing students' interaction
during a class discussion, mapped to different components of the interest loop. In section
5.2, Table 1 further shows course-level strategies applied by the instructor in the ISD course
to sustain the student’s interest in the course. In section 5.3, we show evidence of students’sustained interest, based on a self-reporting survey questionnaire and student interviews.

5.1 Students’ Interest
The episode below depicts a classroom discussion between the students, where theinstructor played the role of a facilitator. This is an example of a carefully designed activity
where the instructor triggered students' interest in the task. In this case, it was the creation oftraining resources for the PwD website. The goal of this discussion was to attain clarity about
the task students are about to undertake for the next few weeks. While doing so, theinstructor also gave room for students to express their opinions and reach a common
understanding for working towards the project goal. Figure 3 shows evidence fromclassroom discussion mapped to the components of the interest loop in IDC theory.
The instructor triggered a discussion by posing the following question prompt: “Whatis a training resource? Can a brochure or a poster or FAQ be considered a training
resource? If yes, under what conditions, and if no, why not? Okay, I don't have an answer.
So that's why I'm throwing it open. So that we can collectively come up with an answer towhat we are going to call training material. Okay, so this discussion has implications for what
we create as material for the PWD cell.”While putting forward the project requirements, the instructor also reiterated thebroader learning goal beyond the project, which is learning ISD concepts through the
project.

950

Figure 3: Components of interest cycle observed during a class discussion.
Above was the anchor question for the discussion and students started participatingbased on their prior knowledge. The goal of this discussion was to bring out students’thoughts and views on “What is a training resource?” and make them understand thecontextual nature of training resources. Students were seen immersed in the discussion andhere are some excerpts from the students’ discourse:
S16: Training through the brochure or poster cannot be considered as a training resource
but it depends on what is the title or theme of the poster. If it is a “how to” question then it
can become a training resource.
S12: ..But a poster or brochure can’t be a training module. It comes under awareness. So if
it is for spreading awareness about PwD for people other than PwD then it can beconsidered as a training resource.
S18: I think a poster can be considered a training module depending on what you are trying
to learn. For example: “How to use a printer”- a poster is sufficient training for that.
S5: I would like to counter that, yes, the instructions with any equipment or anything can be
can be treated as a resource but I can’t be considered training itself. Whatever comes alongwith the printer or the laptop, it’s a set of instructions or FAQ, It’s not training actually.
S19: I think it depends on the context. For example, during COVID situations, a lot of
information was given in the form of instruction through the brochure, and they were kept inpublic places. Whereas, if the same thing is made in a different context, for example, “Youdo this so that you will learn how to ride a bicycle” So the instructions given are still
instructions but can’t be considered training. So it depends on the context.
Here, the instructor intervened and tried to steer the discussion by extending thepoint of context. Instructor suggested: “Okay, so let's pause here, for how to ride a bicycle. A
poster is not a good training resource. Why? Because the training requires a hands-on andthere is no hands-on. That is why it's not an appropriate training resource. Right. So, that isthe point that we have to keep in mind, that it depends upon the purpose. What you have to
do is design activities. Okay, so the point of this discussion is that, while we are creating
material for the PwD cell, we have to keep asking ourselves this question, what is the
purpose of this material? (The implications of this discussion were seen further, during thecreation of training resources under the PwD project.)951

5.2 Course Level Strategies and Their Effect on Students’ Interest
We present a set of important strategies that were applied by the instructor during the ISD
course, along with its goal and evidence that these strategies were useful for sustaining
students' interest.
Table 1: course level strategies applied by the instructor to sustain the students’ interest
Instructors’ strategy
and a targetedcomponent of interest
loop

Goal of the strategy

How this strategy helped insustaining the interest

Real-world project:
Selecting real-world PwD
project with a high social
value for the students
and institute community.
Component: Trigger

To involve the students ina semester-long PwD
project

Having a real-world project with
implications for stakeholders within
the institute, helped students to see
the value associated with the
project.

Interactions withexperts & stakeholders:
Facilitating Interaction
with subject matter
experts (SMEs), PwD
stakeholders, andexperts from the PwD
rehabilitation
organization
Component: Immerse

Students should
understand the needs andexpectations ofstakeholders. Get
feedback on work
progress, and incorporate
the feedback for content
refinement.

Students expressed that the realworld nature of the PwD project in
the ISD class has played an
important role in keeping theirinterest at a high level.
These interactions were perceived
as ‘great motivations’ by thestudents.

Group presentations:
Arrange grouppresentations for peer
and instructor feedback
post each group activity.
Component: Trigger &ImmerseAll groups should beaware of the progresswith respect to the bigger
picture (the whole PwD
website) and ideas should
be shared to get moreclarity

Many valuable suggestions cameout of these presentations anddiscussions. Groups used to
discuss challenges in front of theirpeers used to share potential
solutions for the problem.
Outcomes from those presentations
contributed to the PwD website
project. subtasks completion gavethem a feeling of confidence andboosted their interest.

Flexibility in project
scope & activities:
Increasing or decreasing
the scope of activities
dynamically and deciding
deadlines with mutual
consensus
Component: Immerse &ExtendStudents should not be
overwhelmed by thevolume and/or complexityof work, so putting thetask within their zone ofproximal development toensure students don’t lose
interest.

The PwD project required extra
hours of work other than class time.
Along with the extra time, it
required some content
development skills which students
had to learn. By involving students
in decision-making anddiscussions, the instructor ensured
that the project progress is led anddriven by students. This instilled a
sense of responsibility andownership, which helped insustaining students' interest.

952

Homework 0:
Asking for students’reflection after class in
andaddressing/discussing
those points in the next
class
Component: Trigger,
Immerse & Extend

Giving an opportunity toexternalize and put forth
the points/opinions.
Making students aware ofothers’ reflections andopinions. Facilitatingreflection opportunities tostudents in asynchronousmode so the learningspace remains dynamic.

Students shared that HW0 was likea knowledge-sharing platform, aninteresting and encouraging activity
they enjoyed. They could readothers’ views and were able tocomment, hence the reflection
activity facilitated the opportunity forthe students to know others’ views.

Clear goals:
Mentioning the clear goaland deliverables
Component: Trigger

To bring clarity among
students about the natureof the PwD project,deliverables, roles andresponsibilities, and timecommitments.

Being aware of the goal from the
beginning helped students tounderstand the expected time andeffort for PwD project. Having
clarity about why and how I should
contribute to the course projects
and what I will get out of it, helped
the students maintain a high levelof interest.

Productive
To connect other relevant
digressions: Using
topics with current
ongoing topics.
digressions as a tool todiscuss and connect
interrelated knowledge
components
Component: Immerse &ExtendStudents expressed that
digressions were fruitful for them as
it was an opportunity to connect theinterrelated topics and construct
new knowledge on the basis ofprior knowledge.

When asked about the rationale and thought process behind using these strategies,
the instructor was quoted saying: “The predominant approach here is to draw upon theknowledge that is present in the class already. So, any orchestration is centered aroundmaking people bring out their knowledge and realize that they already know something andalso to listen to others and learn from them rather than the approach of the instructor
imparting some knowledge.”
While explaining how these strategies were utilized for maintaining students' interest,
the instructor mentioned: “So, maintaining the interest of students is again a fairly
challenging task. So, in the beginning, the interest is high and that is automatic, there issome intrinsic interest from the student side. But as the course progresses, a lot of detailingwork has to be done. Because of that, it is inevitable that student interest will drop as thecourse progresses. So, the challenge there was to keep some variety so that the student
interest is maintained. So for example, instead of doing all the stakeholder interviews in one
shot, we kept looping in that you do something, analyze, reflect, and then do anotherinterview so that each time even though it's the same project, there is some newness when
they are talking to different people at different points of time in the course right. So that was
one strategy to maintain interest.”

5.3 Indicators of Students’ Sustained Interest
In this section, we have mapped the evidence of students’ interest in the course to thespecific components of the interest loop. This evidence is drawn from students’ responses
to the interest questionnaire (administered after mid-sem) and semi-structured interviews(conducted after end-sem). Given below is the evidence of students' interest viz triggering,
immersing, and extending.

953

Triggering: Three major points were observed to be triggering the initial interest of students
in the ISD course. These include, the course content being a valuable addition to students’knowledge as educational technology researchers, the PwD project having real-life value forthe institute student community, and instructors teaching methodology. When asked about
their motivation for taking the course, students responded:
S17: The motivation for taking this course was to contribute to the project as the project
outcome had huge implications on someone's academic life.
S11: The reason for having good interest was the project and the stake of the project
because it was very much interested in how it's going to be implemented.
Immersing: For the question: “I enjoy the activities of the class, being unaware of thepassage of time”, student responses were seen distributed as strongly agree-7, agree- 4,
neutral-4, disagree-0 and, strongly disagree-0. Similarly, for the question, “ I find the classactivities rewarding in terms of experience”, student responses were distributed as strongly
agree-10, agree-5, neutral-0, disagree-0, and strongly disagree-0. Additionally, here aresome responses to the open-ended question such as “I look forward to attending every ISD
class because___”. Responses indicate students' immersion in the course:
S4: “In every class, the engagement is different. We will not just sit and look at the slides
prepared by the instructor.”
S5: “I will get to learn more about designing various training materials for PwD cell which isaligning with my interest to join this course”.
Extending: For the question: “I find the Pwd project an interesting extension as anapplication of the learning that is happening in the course”, student responses were seen
distributed as strongly agree-11, agree-3, neutral-1, disagree-0, and strongly disagree-0. An
example of students extending their understanding is given below:
S13: “PwD project has a non-academic context. We can relate the theories which we arelearning, being applied to this project as well. This broadening of the horizon of application ofwhat we are learning beyond the typical academics-related projects gives us a newperspective.”
S11: I had learned during that time that we are not helping them (PwD), it's not like giving
them something that they can't do, they are able to do it themselves, we are just giving
maybe some sort of scaffold. So this change in perspective and mindset also occurredbecause of that interaction with the students.

6.

Discussion and Conclusion

In this study, we investigated how instructor strategies have helped in sustaining students’interest in the semester-long ISD course. We have identified the course-level strategies
implemented by the instructor and mapped the indicators of triggering, immersing, andextending the interest with the help of the strategies. The findings of this study are in linewith Wong et al (2020) and suggest that the thoughtful implementation of IDC theory in thesemester-long course can trigger students' interest in the project activities, immerse them byattaining a state of flow and make them extend their learning in a meaningful way. The
findings regarding real-world projects are in line with Rohm et al (2021) and show that thereal-world context of the PwD website project has played a bigger role in sustaining theinterest. Evidence has shown that having a diverse group composition with students fromdifferent levels of expertise gives an opportunity for better collaboration and teamwork.
participants expressed that the social value of the PwD website project and its potential
impact has helped them to maintain an interest and moral responsibility at a high level.
Each team has put in hours of work while brainstorming ideas, curating and creating
training resources, presenting work in front of the whole class, and adopting the changes
and feedback given by different stakeholders. Feedback at different time points from thestakeholders, end-users, and experts gave an authentic test bed for the product (PwD
website) developed by the students. Working on a relevant problem, acquiring required954

skills, managing the task deadlines, and delivering desired product deliverables has provided
the ISD students with experience in managing a real-world instructional design project. For
students, this experience has been valuable from multiple perspectives, including thelearning of ISD concepts, expanding the horizons of understanding the challenges of PwD,
and their role as individuals contributing towards a shared goal i.e. PwD website.
For an instructor, understanding students’ interest and aligning it with course projects
is an important task. Though it requires intensive efforts, It has helped instructor design
effective learning strategies and sustain the students’ interest throughout the course.
Sustained students’ interest seems to be a key to sculpting interest-driven creators.

Acknowledgment
We would like to thank the anonymous reviewers, and our study participants for theirvaluable participation and colleagues for their support throughout the ISD course.

References
Ainley, M. (2006). Connecting with learning: Motivation, affect and cognition in interest processes.
Educational Psychology Review, 18, 391-405.
Chan, TW., Looi, CK., Chen, W. et al. Interest-driven creator theory: towards a theory of learning
design for Asia in the twenty-first century. J. Comput. Educ. 5, 435–461 (2018).
https://doi.org/10.1007/s40692-018-0122-0
Dewey, J. (1913). Interest and effort in education. Forgotten Books.
Hidi, S., & Renninger, K. A. (2006). The four-phase model of interest development. Educational
Psychologist, 41(2), 111–127.
Kong, S. (2016). A framework of curriculum design for computational thinking development in K-12
education. J. Comput. Educ. 3, 377–394. https://doi.org/10.1007/s40692-016-0076-z
Krajcik, J. S., & Shin, N. (2014). Project-based learning. In R. K. Sawyer (Ed.), The Cambridge
handbook of the learning sciences (2nd ed.) (pp. 275–297). New York, NY: Cambridge University
Press.
Mayring, P. (2015). Qualitative content analysis: Theoretical background and procedures. Approaches
to qualitative research in mathematics education: Examples of methodology and methods, 365380.
Md. Khambari, M. Instilling innovativeness, building character, and enforcing camaraderie throughinterest-driven challenge-based learning approach. RPTEL 14, 19 (2019).
https://doi.org/10.1186/s41039-019-0115-2
Rohm, A. J., Stefl, M., & Ward, N. (2021). Future proof and real-world ready: the role of live projectbased learning in students’ skill development. Journal of Marketing Education, 43(2), 204-215.
Roschelle, J., Burke, Q. Commentary on Interest-Driven Creator theory: a US perspective on fosteringinterest, creativity, and habit in school. RPTEL 14, 13 (2019). https://doi.org/10.1186/s41039-0190107-2
Rowland, G. (1993). Designing and instructional design. Educational technology research anddevelopment, 41(1), 79-91.
Wong, LH., Chan, TW., Chen, W., et al. IDC theory: interest and the interest loop. RPTEL 15, 3(2020). https://doi.org/10.1186/s41039-020-0123-2
Wong, S. F., & Wong, S. L. (2019). Relationship between interest and mathematics performance in atechnology-enhanced learning context in Malaysia. Research and Practice in Technology
Enhanced Learning, 14(1). https://doi.org/10.1186/s41039-019-0114-3
Wong, S. L., Khambari, M. N. M., Wong, S. L., Voon, X. P., & Wong, L. H. (2020). Igniting student
interest towards educational technology through interest-driven creator theory: A case study at
Universiti Putra Malaysia.

955

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Supporting science teachers to selectquality edtech learning solutions fortheir context
Shruti JAIN, Sheeja VASUDEVAN*, Madhuri MUVINKURVE & Sahana MURTHY
Educational Technology, Indian Institute of Technology Bombay, India*isheeja@iitb.ac.in
Abstract: There has been a surge in the development and use of education technology(edtech) learning solutions in the past decade. Amidst the pandemic, teachers werecompelled to incorporate edtech into their instructional approaches. While there areseveral frameworks, standards, or guidelines available for implementing these edtech
solutions, few address the problem of selecting good quality edtech solutions. The ones
that address the problem of quality edtech solutions fail to cover all criteria that areneeded for effective edtech solutions. In addition, they are not easily usable by theteachers for the selection of an edtech solution. For an edtech solution to be effective
it is crucial that it aligns with the context of the learner such as their grade range, theirsubject of interest, language comprehensibility, etc. Also we found that there are nostandards that are available specifically for science teachers in the Indian context. In
this paper, we present a validated, reliable, and research-based, Edtech Tulna Index
designed to support science teachers in India to select quality edtech learning solutions
for their classrooms. We conducted a study to understand the challenges that teachers
face and to test the usability of the tulna index and its usefulness to mitigate teachers’
challenges pertaining to the selection of edtech. We found that teachers found the tulna
index easy to use. This paper contributes to the research of evaluation of edtech
solutions by listing the criteria required for science teachers to select the edtech
product, while it also provides the practice community with a usable tool that can beused to select effective learning solutions and potentially enhance their teachinglearning.
Keywords: EdTech, TULNA, guideline, evaluation, useability, perception

1. Introduction
The availability and variety of edtech learning solutions are growing exponentially worldwide.
Post-Covid the demand for these edtech solutions is shown to be on the rise and so is theinflux of learning solutions into the market (Burns, 2021). The variety in the solutions isbrought about not only by the numerous subjects offered, the language of instruction, andage groups but also by different settings in which instruction takes place. Under suchcircumstances, it is challenging for users (students/ parents/ teachers/ institutions) to choose
an effective learning solution (Patel et. al., 2021), i.e. one that contributes to a meaningfullearning experience. What is required is a set of guidelines that is easily usable by teachers
for identifying and selecting relevant and good-quality edtech solutions. In the absence ofsuch standards or guidelines, the selection will depend on individual teachers’ understanding
and perspective of ‘What is a good quality learning solution for science?’ For some, it could
be good graphics or 3D animation, while others might feel experimentation or activities are
more relevant. This leads to variability and lack of a common understanding of what
constitutes ‘good quality’. Often users rely on reviews by previous users and market reviews
to guide them in the selection process. Such ad hoc decision-making will lead to selection of
an ineffective learning solution that might not best suit their requirement.
Globally there exist some standards and frameworks to evaluate the quality of digitallearning solutions. However, these cannot be adopted universally as learning is known to becontext-dependent (Bransford et. al., 2000). This paper reports the design and development956

of a research-based, easily usable, context-specific, “Edtech Tulna” index to support
teachers in the selection of a good quality edtech learning solution for their use in thescience classroom. A study was undertaken with science teachers currently teaching grades
6-10 in India, to explore their perceptions of the index's usefulness and usability.

2. Literature Review
There are several frameworks and standards that provide guidelines and structures toteachers for effective teaching with ICT. Among these ISTE provides guiding principles forthe selection of edtech (ISTE, 2000). CENTA standards (CENTA, 2023) are used for training
teachers to use the ICT tools to communicate, create, disseminate, store, and manage
information and are aligned with National Professional Standards for Teachers under India’s
National Education Policy 2020 (NEP, 2020) guidelines. The Central Institute of Educational
Technology (CIET, 2023) provides an e-content evaluation tool and an assessment tool for
digital media. Similarly, the recently released Pragyata guidelines for adopting ICT anddigital education (Pragyata, 2023) support infrastructure assessment, teacher training, cybersafety and privacy, teacher professional development, physical and mental well-being ofstudents, and a few pedagogical strategies for using ICT. Many of these frameworks focus
on effective implementation of ICT and those that focus on the quality of edtech learning
solutions do not contain detailed guidelines. Hence the onus is on the teacher to apply thecriteria to their subject and context.
During the Covid-19 pandemic teachers were compelled to change their instructional
approaches to include digital resources or edtech. There are theoretical frameworks, like
UTAUT2(Venkatesh et. al., 2012), that are developed to understand different aspects of the
adoption of technology-based products. In Indian classrooms, the integration of edtech ismainly in the hands of the teacher. According to Bharat Survey for Edtech (2023) Report,
about 1 in 3 of 9867 school children received teacher-directed materials through various
digital mediums. The context of the learner, the teaching style or method of teaching
adopted by the teacher for a particular grade range, the subject being taught, and language
comprehensibility are important points to be considered while adopting edtech solutions(Soundararaj et.al., 2022). A study for developing a conceptual framework for evaluating
web-based learning resources (WBLR) for school education found the influence of culture on
the choice of WBLR among students (Hadjerrouit, 2010). Such studies underscore the needfor a framework that considers the context in which the learning is taking place.
Few discipline-specific standards for science such as Next Generation Science
Standards (NGSS, 2023) and Common Core Science Standards (2010) are available,
however, they are mostly developed in the United States and fail to include the context oflearners from other countries. To overcome this problem, countries such as Australia and the
UK have come up with their own standards to be incorporated into their curriculum. Sinceteachers have a good understanding of the context of their learners, it is important to equip
teachers with a framework that they can adopt for finding good quality edtech learning
solutions for their context. Our research aims at providing this to the teachers in the form of
an index called Edtech Tulna (2023) that contains detailed guidelines for teachers in India for
selecting science edtech learning solutions in their context.

3. Design of Edtech Tulna Index
We created the Edtech Tulna index as a comprehensive and user-friendly tool forsystematically identifying high-quality science edtech learning solutions. Our index isinformed by (i) literature from learning theories, multimedia learning, Human-Computerinteraction, existing edtech evaluation models, and disciplinary teaching-learning practices inthe science domain, (ii) the Indian govt. policies and educational standards and (iii) insightsfrom the product landscape in the Indian edtech ecosystem (Figure 1). The Tulna index is a957

rubric covering three constructs: Content Quality, Pedagogical Alignment, and Technology
and Design (Edtech Tulna, 2023).

Figure 1. Basis of Edtech Tulna Science Evaluation Index.

3.1 Theoretical Basis for Criteria
Seminal papers with theories related to teaching and learning in the science domain andmultimedia learning were used to finalize the twelve criteria in the Tulna Index. The mostimportant criterion for evaluating the content quality of any educational material is theaccuracy of the content. Also in science, it is important that the content is up-to-date withrecent happenings (eg. pandemic, etc.) and scientific advancements (NGSS Lead States,2013). The voice-over and captions used in the content have to be comprehensible by thetargeted group of learners for ease of learning (Mayer, 2014; Soundararaj et.al., 2022).
Alignment with national standards like the National Council of Educational Research andTraining(NCERT)) and coverage of skills recommended by the government guidelines such
as NCF(2005) and NEP(2020), in an Indian context, is also crucial. When designingeducational content, it's crucial to reflect the learner's local context and ensure inclusive
representation in terms of race, gender, religion, and socio-economic backgrounds. This
must be achieved while avoiding stereotypes, allowing learners to identify with the content(Dore, 2022).
Table 1. Tulna criteria under each construct
Construct (no. of criteria)Content Quality (4)Pedagogical
Alignment (6)Technology and Design (2)Criteria Name
Content accuracy & clarity, Language comprehensibility,
Alignment to national standards, Inclusivity
Content in Context, Learner Scaffolding. Cognitive Engagement,
Logical chunking and connectedness, Feedback Quality,
Teacher Support
Interface Design. Universal Design

Pedagogy for science must include authentic and meaningful scenarios from the local
surroundings like the learner’s home, neighborhood, community, and/or culture, that reflect
the practice of science (NGSS Lead States, 2013). Support and assistance should be
provided to bridge the gap between a learner’s current abilities (actual development) andtheir potential abilities (zone of proximal development) to accomplish tasks more complex
than they could do alone (Quintana et.al., 2004; Vygotsky & Cole, 1978). To promote
engagement the content should be conversational and visually guided (Mayer, 2014; Van
Gog, 2014). Breaking the content into small meaningful segments arranged in logicalsequence can enhance learning (Mayer & Pilegard, 2014). Good feedback strengthens thestudents’ capacity to self-regulate their own performance (Nicol & MacfarlaneǦDick, 2006). In
addition, instructors need support to integrate Edtech learning solutions into their teachingeffectively (Banerjee & Murthy, 2012).

958

Any technology-based solution should have an easy-to-use and intuitive interface(Davis, 1989). Furthermore, the learning solutions should be designed to be accessible todiverse learners in order to provide equal access and equal opportunity to people withdiverse abilities (Caldwell et. al., 2008). Based on the literature review and national
standards, the final criteria included in the Tulna index are summarized in Table 1.

3.2 Structure of the Index
Each criterion in the Tulna Index has easy-to-understand yet detailed reviewer guidelines tohelp teachers score on a 3 pt-scale of Not-at-all Incorporated, Partially Incorporated, andFully Incorporated. The reviewer guidelines were developed based on literature and an
examination of multiple science edtech solutions for grades 6-8 and 9-10 in the Indian
edtech landscape. Insights from exploring existing solutions were utilized to create authentic
examples for each criterion. Table 2 gives a glimpse of the index using one criterion fromeach of the three constructs.
Table 2. Sample criteria and reviewer guidelines from the Tulna Index
Criteria

Reviewer Guidelines

Is the language appropriate for the intended learners?
- The vocabulary and accent used is familiar to the learners of the particular grade
range.
Language - The sentences spoken or those appearing on-screen are short and easy to follow.
Compreh - The introduction of scientific terms has to be done in a grade-appropriate manner.
-ensibility e.g. Use of s,p,d,f orbitals to teach covalent bonds in grade 10 is not recommended as
the orbitals are not introduced until that grade.
- There is no use of slangs or informal language [e.g., gonna, wanna, gotcha] that thelearners find difficult to follow
Does the content incorporate appropriate real-life context?
- The context provided is relevant to the context of the learner wherever required- The context provided is sufficient for the learner to master the specific skill- The context does not contain scenarios involving violence, combat, gambling or
stereotyping
Content in- The product provides opportunities to see the application of the skill in the lives of theContext
learners
e.g. to explain the concept of magnetism, showing that a magnet attracts iron nails isrelevant but not sufficient, whereas showing that a magnet can be used to separate
recyclable iron particles from other wastes in a junkyard is a relevant and sufficient
context.

Universal
DesignIs the product accessible to diverse learners?
There are multiple ways to make the product accessible to learners with diverse needs.
- Audio-only or video-only media provides a transcript and/ or on screen text/ caption.
- Text-only media has a voice-over matching with the text.
- Provide captions for all prerecorded / live audio (with or without video) content.
- Possibility of recording an answer and hearing assessment questions as voice-overs.
- Make it easier for users to see and hear content (e.g., an option to increase font size)

For a particular learning solution if the teacher finds that all the indicators mentioned in thereviewer guidelines are met the learning solution “fully incorporates” the criteria. The learning
solution “partially incorporates” the criteria, if only some of the indicators are met or if all theindicators are met but are present inconsistently in the learning solution. And if none or very
few of the indicators are met then the learning solution “not-at-all incorporates” the criteria.

3.3 Validity and Reliability of Tulna Index
The Face and Content Validity of the Tulna index was established through a rigorous
process involving expert review. Individual assessments from the experts comprising a959

senior faculty, an edtech expert, and SME were incorporated to arrive at the final index. Theinterrater reliability was established in 2 stages: (i) during the development of the index and,(ii) after completion of the index. During development, the interrater was conducted amongthe 3 researchers who were involved in the design of the index. Each researcher assessed 2
learning solutions, yielding an average Cohen’s kappa of 0.335 across the 12 criteria. After
this, the researchers discussed their rationale for the assigned scores. Based on thediscussion, some modifications to the wording were done and a few more explanations andexamples were added to remove subjectivity. After revision, the same team evaluated 2 newlearning solutions and the average value of the kappa statistic improved to 0.704, which isconsidered to be a “fair” agreement (Jonsson & Svingby, 2007). In the second stage, twonovice evaluators were trained on the index and they along with a researcher evaluated 4
learning solutions individually. The average value of the kappa statistic between the novice
evaluators and the researcher was 0.708, replicating the result observed with the researchteam thereby establishing the reliability of our index.

4. Study Method
To understand the perceived usefulness, usability, and relevance of the Tulna index we
conducted a study with science teachers. The research questions for the study were as
follows:
1. Is the Tulna index useful for teachers for selecting edtech learning solutions? How
can the teachers adopt it for their use?
2. Do teachers find the Tulna index usable?

4.1 Participants
For purposive sampling, we approached science teachers through social media forums ofschool teachers, publicizing our study as Edtech Tulna- Sensitization Workshop to aid
selection of good quality science edtech learning solutions for their classrooms. The
teachers that registered for the workshop were from either government or government-aided
schools catering to medium to low-income groups in urban and semi-urban areas in India.
Most of the teachers had some experience in using edtech while the number of years ofexperience varied. A few teachers from non-science domains also registered but their data
was not considered for the analysis.

4.2 Procedure
Post registration, the teachers attended a 3hr workshop where they were introduced to the
Tulna index, and the 12 criteria were explained in detail with appropriate examples. The
workshop was made interactive with several interspersed activities to invite participationfrom the attendees. Figure 2 shows one of the activities from the workshop.

4.3 Data collection and analysis
To answer the RQs of the study, we employed several data collection instruments andanalyzed them using mixed methods (Creswell & Poth, 2016). To get insights into teachers’
backgrounds and perceptions of using edtech in the classroom, we asked the teachers to fillout an online questionnaire during registration. The questionnaire included questions like
“According to you what features contribute to good quality content in an edtech learning
solution?”, etc.
To understand the perceived usefulness of our index, we conducted semi-structured
interviews with interested participants. Perceived usefulness is known to be influenced byculture (Straub et. al., 1997) and it is highly unlikely to be captured through surveys like TAM(Davis, 1989). A few of the questions asked in the interview were, “Did your idea about960

evaluating the quality of edtech learning solutions change in any way after the workshop?”and “How does Tulna help you overcome the challenges you face while selecting edtech
learning solutions?”. The interviews were recorded and transcribed, which were later
analyzed using the principles of content analysis (Mayring, 2015).

Figure 2. Sample activity from the workshop on choosing an edtech learning solution
To gauge the Tulna index's usability, we had teachers complete the System Usability Scale(Brooke, 1996) post-workshop. SUS is a robust and versatile tool that can be applied to awide range of ‘products’ (Bangor et. al., 2008). The participants answered the SUS survey
containing 10 items on a 5-point Likert scale ranging from strongly disagree to strongly
agree. The survey includes both positive and negative questions like - “I thought the Tulnaindex was easy to use.” and “I think that I would need the support of a technical person to beable to use the Tulna index.”.

5. Findings
5.1 Demographics and background of teachers
A total of 27 educators enrolled in the workshop, with teaching experience spanning from 1
to 30 years. The majority of these instructors were involved in teaching science to students
in grades 6 to 8, as well as grades 9 to 10. About 74% (20 teachers) had some experience
using edtech learning solutions, such as Audio-Video content, assessments, simulations,and experiments. All the teachers agreed that using Edtech learning solutions was beneficial
for the students. When asked to articulate the benefits of using edtech in the classroom, themajority of teachers mentioned that edtech helps in visualization and increases the efficiencyof teaching-learning.
Based on responses provided regarding the challenges in selecting edtech learning
solutions it was observed that teachers encounter difficulty in finding content for theireducational context. A few responses were “Language that is used by majority of the videos
is quite not in sync with the demographics that we support”; “The explanation style orpresentation style in the available AV are different from my style, that the children are used
to for several years.”. In addition, the complexity of selecting a single learning resource
amidst an array of options was a concern. The teachers mentioned “I can't choose thesource for a particular topic”, and “Selection is not easy as it is not between A or B. Youtube
has a lot of videos on the same topic, several channels, etc.”. Spontaneous edtech selection
was evident from responses such as “it is time-consuming to check all the content, maybe
the first 5 minutes is best but later it is difficult.” and “I select by searching with Google orcollecting from friends”.

961

Table 3. Frequency analysis of quality indicators of edtech (reporting indicators mentioned
by at least 4 teachers)
Quality indicators

freq.

Quality indicators

freq.

assessment questions

14

integration with other tools

7

activity and experimentation based

10

gamification

6

daily-life context

10

user friendly

6visualization

9

interesting4

easy to understand8

scalable4

build on previous learning

7

curriculum

4

Table 3 shows the frequency analysis of the teachers' responses to “According to you, whatfeatures contribute to a good quality edtech learning solution?”. Given that the teachers
provided textual answers, their responses were categorized into distinct "quality indicators”
in edtech solutions. It is noteworthy that almost no teacher (n<4) mentioned indicators like
supporting learning materials, teacher support, short videos, and up-to-date scientificinformation.

5.2 Usefulness
To gain insight into the usefulness of our Tulna index we conducted semi-structured
interviews with 8 out of 16 teachers who attended the workshop. In response to whether the
Tulna index aids in selecting appropriate edtech learning solutions, 5 teachers strongly
agreed, and 3 agreed on a 5-point Likert scale. One of the teachers said that “Yes, using
Tulna criteria gives me faith that I will select the right solutions because when I am reviewing
to choose a product it is my perspective or maybe 1-2 teachers but with Tulna tool is madeaddress to a particular solution.” Additionally, teachers were asked about the significance ofeach of the 12 criteria in selecting edtech learning solutions. Their response was recorded
using a 5-point Likert scale ranging from Strongly Disagree to Strongly Agree. The outcomesare consolidated in Table 4.
One of the questions that were asked to teachers was “How would you use the Tulnaindex?”, we found the responses to be very insightful. One of the teachers said that “byunderstanding criteria such as logical chunking and connectedness, content in context andalignment to national standards, I found the process to make my teaching effective”(translated from Hindi). We also observed a change in the teachers' perception regarding thequality of edtech learning solutions. When asked the question “After the workshop, has youridea about the quality of edtech learning solutions changed in any way?”, most agreed, andfour of the teachers elaborated. One of the teachers said “I can choose the content for my
demographics that fits Tulna criteria, focused attention to the particular parameters, and be
more mindful while I do my own review.” another teacher said “Yes, I will remember to checkfor the criterias discussed. Also, I will use the same criteria for video creation. Content
accuracy and clarity and proper language are main for any content, logical connections andlength of the content to be only 5 to 10 mins for discussing a single concept is very useful. It
is useful. Yes, Tulna is useful”. One teacher said “If I follow Tulna criteria then we will get agood outcome… can expect better responses from students. Pedagogy changes with grade
but children might get tired of my teaching style so videos will be useful to use in teaching”,
while another teacher said “Content if not related to what is not being taught can harmful forchildren…in point of view of examination the content has to be aligned to national standards.
The examples should be given in the context of the child’s learning environment… supportshould be given in the context of child’s mental ability, long videos should be avoided…”
When asked if “any criteria that you did not check before. But now, after theworkshop, you realize that there's something more that you should look for?”, one of theteachers mentioned that “I will check for content accuracy throughout the entire duration,
earlier we used to look for initial few mins… full video must be watched”. Another teacher

962

said that "There's a difference between the content and the context. So how I can, you know,
this is what will be content in the context, making them before making any lesson plan."
Table 4. Avg. score of relevance for each criterion on 5-pt Likert Scale (n=8).
Construct

Content Quality

Pedagogical Alignment

Technology and Design

Criteria

Avg. score

Content accuracy & clarity

4.625

Language comprehensibility

4.875

Alignment to national standards

4.625

Inclusivity

4.125

Content in Context

4.750

Learner Scaffolding

4.500

Cognitive Engagement

4.250

Logical chunking and connectedness

4.500

Feedback Quality

4.500

Teacher Support

4.625

Interface Design

4.625

Universal Design

4.250

5.2 Usability
The Tulna index's usability was confirmed by 8 participant teachers who completed the
System Usability Survey during post-workshop interviews.The SUS score of the Tulna index
was found to be 71.563, which corresponds to a “good” usability score (Brooke, 1996). The
SUS score can be decomposed into usability and learnability (Lewis & Sauro, 2009). The
Tulna index scored high on usability (92.969), while it scored poorly on learnability (48.438).
Items 4 and 10 measure learnability with statements: “I think I would need the support of atechnical person to be able to use this system.” and “I needed to learn a lot of things before I
could get going with this system.”. Low scores on these two items emphasize the need for
Tulna index training.

6. Discussion and Conclusion
To address RQ1, we conclude that teachers found the Tulna index useful, as evidenced bytheir responses to the Likert scale questions as well as open-ended questions. Teachers
stated that they will likely use Tulna in their future practice to identify an edtech productsuitable for their needs, thereby making their practice effective. This aligns with priorresearch which has emphasized the need and importance of contextualization in edtech(Specht, 2006). Teachers also repurposed the Tulna criteria and stated that they will use
them as quality criteria during their video creation process. Teachers’ responses were highlyreflective and illustrated examples of reflection-on-action (Schon, 1983). We noticed a shift
in teachers’ decision-making process from an ad hoc selection approach to a morestructured and nuanced one, wherein they applied specific research-based criteria to make a
judgment. Teachers also reflected on changes in their own practice, such as viewing content
more deeply before making selection decisions. With regard to RQ2, teachers found the
Tulna index highly usable, which indicates a high potential for actual use in practice (Pan et.
al., 2020). Overall, teachers perceived the value of a common set of quality standards foredtech selection rather than varied individual perceptions.
The Covid -19 pandemic has forced educators to adopt and integrate edtech learning
solutions into their teaching practices. The lack of appropriate scaffolds complicates the
adoption of good quality edtech solutions. Our study introduces a research-based andcontext-specific Edtech Tulna index, offering teachers a reliable and user-friendly tool to963

choose high-quality edtech learning solutions for their classrooms. This study involving
science teachers also investigates the shift in their perception of edtech learning solutions
following awareness about the Tulna index. Additionally, the study evaluates the Tulnaindex's usefulness and usability.
Research has established a positive correlation between teachers' perceptions of thebeneficial impact of technology and the actual utilization of technology in classrooms(Domingo & Garganté, 2016) but what is missing is context-specific guidelines that would
help the teachers to choose relevant quality content for their teaching practices. Available
frameworks and standards address the evaluation or implementation of edtech solutions, anindex to solve the selection problem of edtech solutions specifically in science education is
missing. The Tulna index bridges this gap by providing selection guidelines.
The limitations of our study include the sampling process wherein only the teachers
who were already interested in using edtech volunteered and hence our findings could not
capture the changes in the perception of teachers who are not inclined to use edtech.
Another limitation was that the teachers were mostly from urban and semi-urban schools. It
would be beneficial to scale up the study and understand the perspective of teachers fromrural India.
The science-specific Tulna index with detailed guidelines on what indicators to checkin a learning solution is not only a valuable contribution to addressing the selection challenge
faced by teachers but will also give a starting point to the researchers to further explore thisarea of providing teachers with support for edtech adoption.

Acknowledgments
The authors would like to thank Kavya Alse, Meera Pawar, and Leena Bhattacharya for theircontribution to designing the Tulna index. We would like to thank the Centre Square
Foundation for the financial support for the project.

References
Banerjee, G. & Murthy, S. (2012). Effect of Instructors' Pedagogy and TPACK on integration ofcomputer based visualizations. Workshop Proceedings of ICCE, Singapore.
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability
scale. Intl. Journal of Human–Computer Interaction, 24(6), 574-594.
Bharat Survey for EdTech (BaSE). (2023). https://www.edtechbase.centralsquarefoundation.org/,
retrieved 21 May 2023.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.) (2000). How People Learn: Brain, Mind,
Experience, and School. National Academies Press.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189, 4-7.
Burns, M. (2021). Technology in Education. Paper commissioned for the 2023 Global Education
Monitoring Report, Technology and education.
Caldwell, B., Cooper, M., Reid, L. G., Vanderheiden, G., Chisholm, W., Slatin, J., & White, J. (2008).
Web content accessibility guidelines (WCAG) 2.0. WWW Consortium (W3C), 290, 1-34.
CENTA (2021). Retrieved from https://centa.org/ on 21 May 2023.
CIET (2023). Retrieved from https://ciet.nic.in/upload/GuidelinesforeContent3.pdf on 21 May 2023.
Common Core State Standards (2010). National Governors Association Center for Best Practices &Council of Chief State School Officers. Washington, DC.
Creswell, J. W., & Poth, C. N. (2016). Qualitative inquiry and research design: Choosing among five
approaches. Sage publications.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of
information technology. MIS quarterly, 319-340.
Domingo, M. G., & Garganté, A. B. (2016). Exploring the use of educational technology in primaryeducation: Teachers' perception of mobile technology learning impacts and applications' use in
the classroom. Computers in Human Behavior, 56, 21-28.
Dore, R. A. (2022). The effect of character similarity on children’s learning from fictional stories: The
roles of race and gender. Journal of Experimental Child Psychology, 214, 105310.

964

EdTech Tulna (2023). Retrieved from https://edtechtulna.org/ on 21 May 2023.
Hadjerrouit, S. (2010). A conceptual framework for using and evaluating web-based learning
resources in school education. Journal of Information Technology Education: Research, 9(1),53-79.
International Society for Technology in Education. (2000). ISTE national educational technology
standards (NETS). Eugene, OR: International Society for Technology in Education.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational
consequences. Educational research review, 2(2), 130-144.
Lewis, J. R., & Sauro, J. (2009). The factor structure of the system usability scale. In Human
Centered Design: First International Conference, HCD 2009, Held as Part of HCI International
2009, San Diego, CA, USA, July 19-24, 2009 Proceedings 1 (pp. 94-103). Springer Berlin
Heidelberg.
Mayer, R. (2014). Principles Based on Social Cues in Multimedia Learning: Personalization, Voice,
Image, and Embodiment Principles. In R. Mayer (Ed.), The Cambridge Handbook of Multimedia
Learning (Cambridge Handbooks in Psychology, pp. 345-368). Cambridge: Cambridge
University Press.
Mayer, R. E., & Pilegard, C. (2014). Principles for managing essential processing in multimedia
learning: segmenting, pre-training, and modality principles. In R. E. Mayer (Ed.), The
Cambridge Handbook of multimedia learning (2nd ed., pp. 316–344). Cambridge: Cambridge
University Press.
Mayring, P. (2015). Qualitative content analysis: Theoretical background and procedures. In
Approaches to qualitative research in mathematics education, Springer, pp. 365–380.
NEP (2020). Policy document released by Government of India.
National Curriculum Framework. (2005). National Council of Educational Research and Training.
NGSS Lead States. (2013). Next Generation Science Standards: For states, by states. Washington,
DC: The National Academies Press.
Nicol, D. J., & MacfarlaneǦDick, D. (2006). Formative assessment and selfǦregulated learning: A
model and seven principles of good feedback practice. Studies in higher education, 31(2), 199218.
Pan, S. L., & Pee, L. G. (2020). Usable, inǦuse, and useful research: A 3U framework fordemonstrating practice impact. Information Systems Journal, 30(2), 403-426.
Patel, A., Dasgupta, C., Murthy, S., & Dhanani, R. (2021). Co-Designing for a Healthy Edtech
Ecosystem: Lessons from the Tulna Research-Practice Partnership in India. Rodrigo, MMT etal.(Eds.).
Pragyata (2023). Retrieved from https://www.education.gov.in/sites/upload_files/mhrd/files/pragyataguidelines_0.pdf on 21 May 2023.
Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G., Kyza, E., Edelson, D.,& Soloway, E. (2004). A scaffolding design framework for software to support science inquiry.
The journal of the learning sciences, 13(3), 337-386.
Schon, D. A. (1983). The reflective practitioner: How professionals think in action (p. 1983). New
York: Basic Books.
Soundararaj, G., Badhe, V., Ishika., Pawar, M., Dasgupta, C., & Murthy, S. (2022). Unpacking
contextual parameters influencing the quality of Personalized Adaptive Learning EdTechapplications. Proceedings of ICCE 2022, Kuala Lumpur, Malaysia.
Specht, M. (2006). Contextualized Learning: Supporting Learning in Context. In Advances in WebBased Education: Personalized Learning Environments. pp. 22. IGI Global.
Straub, D., Keil, M., & Brenner, W. (1997). Testing the technology acceptance model acrosscultures: A three country study. Information & management, 33(1), 1-11.
Van Gog, T. (2014). The signalling (or cueing) principle in multimedia learning. In R. E. Mayer (Ed.),
The Cambridge handbook of multimedia learning (2nd ed., pp. 263e278)., NY University Press.
Venkatesh, V., Thong, J. Y., & Xu, X. (2012). Consumer acceptance and use of information
technology: extending the unified theory of acceptance and use of technology. MIS quarterly,
157-178.
Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes.
Harvard university press.
Zhang, X., De Pablos, P. O., Wang, X., Wang, W., Sun, Y., & She, J. (2014). Understanding theusers’ continuous adoption of 3D social virtual world in China: A comparative case study.
Computers in Human Behavior, 35, 578-585.

965

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

A Preliminary Study: Exploring Teachers’
Perspectives on the Role of Gathering
Information in Supporting Teachers’ Digital
Learning Agility
Kamilah ABDULLAHa, Mas Nida MD KHAMBARIa*, Nur Dania MOHD ROSLIa, Su Luan
WONGa, Noor Syamilah ZAKARIAa, Siti Raba’ah HAMZAHa, Priscilla MOSESba, Universiti Putra Malaysia, Malaysiab, Universiti Tunku Abdul Rahman, Malaysia*khamasnida@upm.edu.my
Abstract: Gathering information is essential to enhancing teachers' digital learning
agility. By gathering data, teachers can stay abreast of the latest digital technologies
and tools, acquire new skills, and improve their teaching practices. This preliminary
study explores the role of gathering information in supporting teachers’ digital learning
agility. The preliminary study employed a qualitative research approach, using semistructured interviews to collect data from three teachers. The findings reveal thatteachers gather information related to student learning and digital technologies toimprove their teaching practices and support their students' digital learning. The study
highlights the importance of information gathering in enhancing teachers' digitallearning agility and provides insights for the development of policies and initiatives
aimed at facilitating the improvement of teachers' digital learning agility.
Keywords: teachers, gathering information, digital learning agility, educational
technology

1. Introduction
In today's digital environment, teachers must have digital learning agility to swiftly andeffectively acquire new digital tools and technologies. Chrismastianto and Wibawanta (2023)
claim that many teachers lack the knowledge and resources to use digital technology to teachand learn. Getting information is crucial to improving teachers' digital learning skills. Teachers
may develop new skills, keep up with modern tools, and enhance their teaching throughgathering information (Van Petegem et al., 2021).
The literature highlights organisational learning as a major consequence ofdigitalisation (Kuusisto, 2017). This means that organisations, including schools, must
constantly adopt new digital tools and technology to stay competitive. Teachers are key to thisprocess as they deploy digital technology and classroom aids. Teachers must gather data tokeep current on digital technologies and tools and help their businesses adapt to digitalization.
Additionally, teachers must learn about digital technologies and tools and create digital
literacy (Hanan et al., 2022). This encompasses digital information access, evaluation, andorganisation. By enhancing their digital literacy, teachers may assist pupils master digital skills
for the 21st-century workforce. Digital literacy training and tools may help teachers improvetheir abilities and promote student digital learning.
Thus, this pilot study interviewed teachers who utilised digital technology in uniqueconditions (such the COVID-19 pandemic and flooding) to learn about digital pedagogy. The
results of this basic study might be utilised to launch a more in-depth study to help
policymakers and educators enhance teachers' digital abilities and teaching efficacy.

966

2. Literature Review
Since digital technology is essential to education, teachers must be proficient in its use.
Information collecting in education is well-known, but how it may assist Malaysian educators
become digitally agile is not. Insufficient data address the existing study on Malaysianteachers' digital technology information-seeking practises. However, several studies haveexamined Malaysian educators' information-seeking. Anuar et al. (2019) found that teachers'
attitudes, subjective norms, and perceived behavioural control all affected their intention touse digital technology in teaching. Veeramuthu (2023), on the other hand examined Malaysian
literature students' IR4.0 learning practices and challenges. The study found that students
struggled with digital literacy and resource shortages while shifting to digital learning. This
study found that teachers need digital abilities to help pupils learn in the digital age.
Additionally, Zainal and Zainuddin (2020) examined how national Information andCommunication Technology (ICT) regulations affect Malaysia's educational sector, focusing
on school technology use. Policy measures improved technological integration in educational
institutions, according to studies. Poor infrastructure and teacher training have hindered the
optimal use of digital technologies in education. As a result, digital technology has become acrucial tool in education, and teachers must be proficient at using it. There is little information
on how Malaysian educators may improve their ICT skills. Several studies have examined
Malaysian teachers' digital technology information-gathering practices, underlining therelevance of digital skills in education. These studies suggest that teachers must master digitalskills to help kids learn in the digital age. Poor infrastructure and teacher training also hinder
the proper implementation of digital technology in education, according to the findings.
Clearly, information acquisition will grow in importance. The role of informationcollection in shaping instructors' digital learning agility is misunderstood. Thus, to fill thisknowledge vacuum, this pilot study answers the following research question: How does
gathering information support teachers’ digital learning agility?

3. Methodology
This study examined how Malaysian primary school teachers collected data on digitalteaching methods amid unprecedented COVID-19 and flood occurrences. The researcher
initially contacted prospective instructors for the exploratory inquiry using a prepared
information document that explained the study's goals and expected outcomes. Teachers
who wanted to participate were screened to make sure they met the researcher's criteria,
which included (1) having experience teaching during unsettling times (i.e., COVID-19, theflood, etc.), (2) having taught for at least five years to ensure they had ample experience
before, during, and after COVID-19, and (3) teaching at a school with digital facilities. Three
Selangor primary school teachers were included in the preliminary research.
After ethical permission, data gathering began. Participants have to sign a form forapproval and the Google Meet semi-structured interviews averaged 45 minutes. The
researcher copied the interview tapes to her laptop to transcribe and preserve them. The
transcriptions were returned to instructors to enhance the interviews and assure accuracy(Mero-Jaffe, 2011).
The data was thematically analysed using the six stages proposed by Braun and
Clarke (2006) because it is appropriate for the research as it reports experiences, meanings,
and the reality of participants (Braun & Clarke, 2006) in order to uncover teachers' realexperiences and gather information about digital teaching.

4. Findings and Discussion
Based on the data taken during the interviews, the findings are grouped in line with thecategories of information obtained by the teachers. Notably, when questioned about acquiring
information, teachers mentioned two types: (1) getting information and data connected to
student learning, and (2) gathering more digital teaching-related information.

967

Based on the biographical backgrounds gathered, Table 1 depicts the age distributionof three teachers. Their educational backgrounds were as follows: two had bachelor's degrees
and one had a master's degree. According to teaching periods, all three teachers had an
average of 12 years of teaching experience and had taught before, during, and after COVID19.
Table 1. Age Distribution of Teachers
Age
Count

30-34
1

35-39
1

40-44
1

4.1 Gathering Information and Data Related to Students’ Learning
Regarding the first variety of information and data pertaining to student learning
collected by teachers, one major theme has been derived from the data collected as presented
in Table 2.
Table 2. Teachers’ Motivational Factors for Gathering Information Related to Students’
LearningxxStudents’ Participation
Identifying the time and day
students are most active
Engage with students

Students’ Proficiency Level
Plan the teacher’s next
teaching strategy
x Improvise teaching methodsx

The teachers were motivated to gather information related to digital teaching because
it increases students’ participation in online classrooms. One of the types of information thatthe teachers gather is the student’s attendance and the teacher explains “…by knowing therate of attendance on certain days or certain times, we’ll know that, maybe on Wednesday
noon, there will be more students.” (T1). T1 also states that once they have identified a suitable
time and day that the students are most likely to participate, “…that is the time where you
share more input, and for other sessions, maybe you can share something that is not really orvery important during that (session)…” They also expressed that by gathering information,
they would engage with the students better. T2 explains “…Yes. I keep a record (because) I
don’t know the children, so I ask whenever I start to do a reading activity and ask questions or
ask what’s your name? I got a list, and then I write down, I record it there because I don’t know
the children…” This conclusion is consistent with Fullan's (2002) study, in which data collecting
assists teachers in identifying students' strengths and shortcomings, allowing them to give
focused assistance and interventions.
The second motivational factor was the ability to identify students’ proficiency levels
which can be done by gathering information related to digital teaching. T2 states that theiractions of gathering information after they did online activities helped them to know thestudents’ proficiency level “…from there, I know their level of proficiency, so then (I know) how
to improvise or what shall I do, I mean, my enrichment or remedial activities for the children.”
T1 also echoes the same sentiment when s/he states “…from there it actually helps me toplan for my next steps to teach the kids.” This conclusion is corroborated by research done by
Basuki and Hidayati (2019), who said that data gathering assists teachers in monitoring
student development and identifying students who may be struggling and require further
support. They also claimed that data collecting enables teachers to assess the efficacy of
educational technology tools and resources (Basuki & Hidayati, 2019).
The researcher found the data insufficient as no participants addressed their difficulties
in collecting the material, despite its usefulness in understanding teachers' motivations. The
literature has highlighted the challenges teachers face, including (1) time constraints to gather
and analyse data; (2) a lack of training or expertise; (3) a lack of technology or assessment
tools; and (4) concerns about students' privacy, especially sensitive information. While968

gathering data on student learning is crucial for good teaching, the researcher emphasises
the importance of addressing teachers' challenges.

4.2 Gathering Additional Digital Teaching-Related Information
Regarding the second variety of information gathered by teachers, namely additional
digital teaching-related information, the researcher has identified a central theme as presented
in Table 3.
Table 3. Factors Influencing the Acquisition of Additional Digital Teaching-Related Information
by Teachers.
Keeping up with the most recent digitalteaching trendsxx

Enhance teachers’ own knowledge
and abilities regarding digitalteaching tools and informationx Self-satisfaction
x Upgradeown
digitaltechnology skills

Digital technology constantly
changing
Many new digital teaching
platforms

Based on the data extracted from the preliminary interviews, it is evident that there areseveral factors influencing teachers’ acquisition of additional digital teaching-related
information. P1 and P2 both agreed that there are constant changes when it comes totechnology that are teaching-related. P1 explains that "It is important to do this (gather
information) because it's (digital technology) constantly changing and there are also, there arealways new things that we need to know that we will learn, or we can learn." P2 in the interview
highlights that “…yes of course. Everything is new. digital itself is very new…" According to
Perifanou et al. (2021), by keeping up with digital teaching trends, teachers may meet their
daily teaching requirements and prepare for long-term digital education progress. Furthermore,
new digital teaching platforms also expose teachers to several digital technologies that might
help them boost students' learning (Hadianti & Rohmah, 2021).
The next influencing factor is to enhance teachers’ own knowledge and abilities
regarding digital teaching tools and information. P3 strongly asserts that “We learn digital tools
for our own self-satisfaction. We want to know how to use it. It’s very motivating when we feel
happy. We like to explore and explore.” When further asked whether P3 always keeps up todate with new digital technology, P3 further explains that “(I) always do. When we discuss
innovation, devices, and tools, we typically refer to apps. I seek digital technology-related
information through the website. Numerous websites Occasionally through TikTok. TikTok isnt
just about people dancing. There is a video tutorial about design—how to take an angle (whenphotographing)—so they have instructional videos. Therefore, we update our knowledge of all
aspects, not just teaching." It is important that teachers should follow the latest digital teaching
trends to increase their digital literacy and abilities, which can boost student learning (Liza &Andriyanti, 2020; Rahmi & Fajrussalam, 2022). Additionally, teachers who are competentin using technology for lesson preparation, classroom instruction, assessment and feedback,and contact with students and families are more motivated to learn and use digitaltechnologies (Beardsley et al., 2021).
Using the latest digital teaching methods can have pros and cons. Some teachers lackdigital literacy abilities to use technology successfully in the classroom (Liza & Andriyanti, 2020)
which may induce frustration and insecurity while utilising digital technology, lowering teaching
and learning quality (Shen et al., 2017). A lack of school digital technology infrastructure may
also hinder digital teaching (Chrismastianto & Wibawanta, 2023). Some teachers may feelleft behind in their digital education developments (Tolks et al., 2020). Finally, teachers may
become too dependent on digital technology and neglect other important components ofeducation like building relationships with students and delivering individualised feedback (Linet al., 2017).

969

4.3 Teachers’ Preferred Method of Gathering Information
Final data from preliminary interviews are participants' favourite ways of acquiring
information. All three individuals preferred to get information online. P1 mentions that the
information gathered is “mostly online videos and webinars” while P2 explains that s/hegathered information through “… (I use) YouTube, Google…” P3 on the other hand further
expands that “…it’s a good thing (to be able to gather information) because it’s all at yourfingertips. By phone. Everything is on the phone, through the Internet.” The discussion with all
three preliminary participants highlighted the ease with which knowledge can be gotten in thisday and age because everything is available online. It is worth noting that teachers can use
the Internet to obtain knowledge on digital teaching (König et al., 2020; Southerton & Lee,
2021; Cahyaningsih, 2021). However, it is critical for teachers to have the appropriate digital
literacy abilities to browse and utilise the internet for teaching purposes (Hanan et al., 2022),
since this can increase the quality of students' teaching and learning experiences (Liza &Andriyanti, 2020).
The preliminary study shows that acquiring information supports teachers' digitallearning agility in numerous ways. First, it keeps teachers up to date on the latest digital tools
and technology to better their teaching (Tour, 2016). Second, it helps students find andaccumulate learning resources (Eraku et al., 2021). Thirdly, it motivates teachers to evaluate
their instruction and improve their digital literacy (Tour, 2016). Finally, collecting information
helps teachers socialise and build personal learning networks for continuous digital learningassistance (Tour, 2016). Thus, information collecting is essential to teachers' digital learning
agility because it allows them to build digital literacy skills and stay current on the latest digitaltechnologies and tools that might improve teaching.

5. Conclusion
Finally, this pilot research showed how information collecting improves teachers' digitallearning agility. The literature highlighted the relevance of digital skills in education and thechallenges to efficient digital technology use in education. Methods detailed this study's
qualitative research style, which comprised semi-structured interviews with Malaysian primaryschool teachers.
The findings show that teachers use student learning and digital teaching information
to improve their teaching. The study also identified factors that affect teachers' student
learning data collection. The survey found that acquiring information is an important part of ateacher's job and may help them become more technologically agile. Online platforms were
the favoured way.
This study expanded teachers’ information collecting and digital learning agility
literature. This exploratory study may prompt a more thorough investigation that might inform
policies and activities to improve teachers' digital dexterity and pedagogical efficacy. Future
study should examine teachers' obstacles in obtaining digital instructional data and how tosolve them. Future research should also examine how different informationgathering methods might increase instructors' digital learning agility.

Acknowledgements
This study was funded under Fundamental Research Grant Scheme (FRGS) by Ministry ofHigher Education Malaysia. We would like to thank all the teachers for their valuable input tothis study.

References
Anuar, N. B., Yusoff, R. C. M., & Yusoff, M. Z. M. (2019). Digital education behaviour among Malaysian
school instructors: An extended theory of planned behaviour. Education and Information
Technologies, 24(6), 3475-3491.
Basuki, Y., & Hidayati, N. (2019). The effectiveness of Quizizz in vocabulary learning. Journal of English
Language Teaching and Linguistics, 4(2), 1-12.

970

Beardsley, M., Albó, L., Aragón, P., & Hernández-Leo, D. (2021). Emergency education effects onteacher abilities and motivation to use digital technologies. British Journal of Educational
Technology, 52(4), 1455-1477.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77-101.
Cahyaningsih, N. L. G. D. P. (2021). Digital tools used in bahasa indonesia for non-native speakers(BIPA) online learning in Canggu community school. In 2nd International Conference on Education,
Language, Literature, and Arts, ICELLA 2021,114-124. Atlantis Press.
Chrismastianto, I. A. W., & Wibawanta, B. (2023). Teacher’s competencies profile in teaching andlearning using digital technology in postmodern society. In Proceedings of the 4th International
Conference on Law, Social Sciences, and Education, ICLSSE 2022, 28 October 2022, Singaraja,
Bali, Indonesia.
Dahlqvist, C. (2021). Information-seeking behaviours of teacher students: A systematic review ofquantitative methods literature. Education for Information, 37(3), 259-285.
Eraku, S. S., Baruadi, M. K., Anantadjaya, S. P., Fadjarajani, S., Supriatna, U., & Arifin, A. (2022).
Digital literacy and educators of islamic education. Edukasi Islami: Jurnal Pendidikan Islam, 10(01),569-576.
Fullan, M. (2002). The change leader. Educational Leadership, 59(8), 16-20.
Hadianti, S., & Rohmah, D. W. M. (2021). English teachers’ perception on using digital tools in theclassroom. Exposure: Jurnal Pendidikan Bahasa Inggris, 10(2), 234-241.
Hanan, A., Firman, E., & Putri, A. N. (2022). Investigating English teachers' digital literacy in the context
of online learning. Jurnal Ilmu Sosial dan Pendidikan (JISIP), 6(3).
König, J., Jäger-Biela, D. J., & Glutsch, N. (2020). Adapting to online teaching during COVID-19 school
closure: teacher education and teacher competence effects among early career teachers inGermany. European Journal of Teacher Education, 43(4), 608-622.
Kuusisto, M. (2017). Organizational effects of digitalization: A literature review. International Journal Of
Organization Theory And Behavior, 20(03), 341-362.
Lin, M. H., Chen, H. C., & Liu, K. S. (2017). A study of the effects of digital learning on learning motivationand learning outcome. Eurasia Journal of Mathematics, Science and Technology Education, 13(7),3553-3564.
Liza, K., & Andriyanti, E. (2020). Digital literacy scale of English pre-service teachers and their perceived
readiness toward the application of digital technologies. Journal of Education and Learning(EduLearn), 14(1), 74-79.
Merro-Jaffe, I. (2011). “Is That What I Said?” Interview transcript approval by participants: An aspect of
ethics in qualitative research. International Journal of Qualitative Methods, 10(3), 231-247.
Perifanou, M., Economides, A. A., & Tzafilkou, K. (2021). Teachers’ digital skills readiness during
COVID-19 pandemic. International Journal of Emerging Technologies in Learning, 16(08), 238251.
Rahmi, Y. N., & Fajrussalam, H. (2022, November). Digital media: Strategy Islamic religious for children
in modern era. In Proceeding of Saizu International Conference on Transdisciplinary Religious
Studies (pp. 39-47).
Shen, X. H., Li, L., Zhang, Y. F., & Ji, X. H. (2017). Building multi-level practice teaching system for
digital media major. International Conference on Education Innovation and Economic
Management.
Southerton, E., & Lee, V. R. (2021). Current approaches in teacher learning on digital social platforms.
Handbook of Research on Transforming Teachers’ Online Pedagogical Reasoning for Engaging
K-12 Students in Virtual Learning, 624-641.
Sree P.N.U. & Das V.T. (2021, February 5). Impact of intentional change theory and intrinsic motivation
on agile teachers. EPRA International Journal of Economic and Business Review, 10–22.
Tolks, D., Kuhn, S., & Kaap-Fröhlich, S. (2020). Teaching in times of COVID-19. Challenges andopportunities for digital teaching. GMS Journal for Medical Education, 37(7).
Tour, E. (2017). Teachers’ self-initiated professional learning through personal learning networks.
Technology, Pedagogy and Education, 26(2), 179-192.
Van Petegem, W., Bosman, J. P., De Klerk, M., & Strydom, S. (2021). Evolving as a digital scholar:
Teaching and Researching in a Digital World (p. 180). Leuven University Press.
Veeramuthu, V. (2023). Learning behaviours and challenges of Malaysian literature students in thecontext of IR4.0. Education and Information Technologies, 28(1), 1-18.
Zainal, N. H., & Zainuddin, Z. (2020). The impact of national ICT in education policy initiatives in
Malaysia on school technology adoption. Education and Information Technologies, 25(6), 52655283.

971

Rosli, N.D. M. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Role of Flexibility in Shaping
Teachers’ Digital Learning Agility: A
Preliminary Study
Nur Dania MOHD ROSLIa, Kamilah ABDULLAHb, Mas Nida MD. KHAMBARIc*, Su
Luan WONGd, Noor Syamilah ZAKARIAe, Priscilla MOSESf & Siti Raba’ah HAMZAHg
a,b,c,d,e,g
Universiti Putra Malaysia, MalaysiafUniversiti Tunku Abdul Rahman, Malaysia*khamasnida@upm.edu.my
Abstract: Digitally agile teachers are capable to deal with new experience flexibly andrapidly by trying new behaviors and making quick adjustment so that learning can takeplace even they have constraints especially in using technologies. Flexible teachers
are capable to adjust their plan on the fly and make changes to suit current demand.
This preliminary study explores the role of flexibility in supporting teachers’ digitallearning agility. The study employed qualitative approach with three in-service teachers
through semi-structured interviews. Findings suggest that flexibility helped teachers to
become digitally agile by developing more effective and strategic behaviors to
overcome difficulties in online learning. This study emphasizes the value of flexibility inenhancing teachers' digital learning agility and offers suggestions for personal andprofessional development.
Keywords: Teachers, flexibility, digital learning agility, online learning

1. Introduction
The term "flexibility" describes individuals who are able to adapt or adjust to changes incircumstances or expectations, enabling them to remain open-minded and responsive to new
ideas, situations, or challenges (DeRue et al., 2012; Burke, 2018; Murphy, 2021). Flexibility isa crucial trait for teachers, enabling them to adapt to changing circumstances and expectations.
It allows them to remain open-minded and responsive to new ideas, situations, and challenges.
Corpus et al. (2022) found that teachers who can utilize ICT and adjust their teaching methods
to suit their students' needs improve learning quality. This paper explores the flexibility of
Malaysian teachers as a key factor in supporting digital learning agility and enhancing theirperformance in the digital age.
Learning agility is a crucial aspect of organizational leadership (Lombardo & Eichinger,
2000; DeMeuse et al., 2010; Bedford, 2011; DeMeuse, 2017a; Burke & Smith, 2018), but it
has received less attention in educational settings. Warkentien's (2016) study highlights
teachers as strategic classroom leaders, emphasizing teachers are also capable of serving as
leaders and can benefit from developing their agility. the importance of flexibility and speed inlearning. DeRue et al. (2012) defined learning agility with a primary focus on flexibility andspeed. Regarding flexibility, they emphasized the crucial nature of being able to change one's
viewpoint to better comprehend how information is related and how it is used to describe agiven circumstance. Technology ability is a challenge for teachers in Malaysia, especiallyduring the COVID-19 pandemic. However, a quick response to unexpected situations can help
curb learning delays and loss. Thus, exploring teachers' flexibility using qualitative approaches
can help develop digital learning agility, enabling them to cope with steep curves and adapt tochanging environments.

2. Literature Review

972Flexibility is a sign of a better cognitive skills that the capacity to visualize links andmove between concepts with ease (DeRue et al., 2012). Flexibility plays a crucial role inenabling teachers to be agile in today's rapidly changing world, allowing them to grasp evolving
educational pedagogies and technologies and adjust accordingly for effective teaching.
DeRue et al. (2012) defined learning agility based on cognitive and behavioral processes, witha primary focus on flexibility and speed. In contrast, Burke (2016) presented a different
perspective on learning agility, defining it through behavioral components and identifying ninedimensions, including flexibility, speed, experimenting, performance risk-taking, interpersonalrisk-taking, collaborating, information gathering, feedback seeking, and reflecting. Despite
their divergent approaches, flexibility remains a vital component of learning agility.
Maintaining consistent high performance requires a willingness to learn from mistakes
and a positive outlook toward change and failure. These traits underscore the importance of
flexibility for agility, as it encourages adaptability in response to situational demands. Allen(2016) examined the value and distinctiveness of learning agility in the workplace and founda significant correlation between flexibility and learning agility. Therefore, it is essential to
explore how flexibility shapes teachers' learning agility, particularly in utilizing digital tools tocope with the ongoing technological advancements in education, to ensure teachers reach
their maximum potential in the classroom.
Learning agility is a crucial predictor of leader success (De Meuse, 2017), with only
10% of current leaders having high levels as noted by Joiner and Josephs (2007). This isbecause, under pressure to complete tasks quickly, people are more likely to make false
decisions. Teachers, as classroom leaders responsible for students' cognitive and social
development (Lieberman & Miller, 2005), face the risk of making poor judgments too.
Developing learning agility is crucial for motivation, leadership, and engagement (Juhdi et al.,2012). Teachers' digital learning agility is characterized by speed and flexibility in responding
to uncertain circumstances. Strong technology understanding allows them to confidently use
various methods (Csordás, 2020; Seale et al., 2010), including modifying, adapting, or
personalizing technology to meet current teaching and learning needs. Identifying
characteristics that support teachers in becoming digitally agile is crucial.
Research on learning agility in the business sector is extensive, but few studies have
explored it in the educational sector. Schools too face dynamic changes due to factors like
immigration flows, demographic shifts, and emerging technology (Brzychcy, 2019). The
Malaysian Teacher Standard (MTS) 2.0 framework emphasizes four competencies for soulful
teachers: knowledge orientation, instructional, community involvement, and personal quality.
However, technology expertise is not specifically addressed in this area.To address thesegaps and identify the characteristics of digitally agile teachers, the following research question
has been posed: How does flexibility shape teachers' digital learning agility?

3. Methodology
It is worth emphasizing that the actual research focuses on the development of a digitallearning agility framework for Malaysian teachers. To meet the study's aims, a larger set ofdata was gathered. Nevertheless, a limited scope of the study was extracted for this article,
and the objective of this study was to examine teachers' flexibility in shaping their digitallearning agility by examining their experiences with digital tools for online teaching andlearning during the school closure caused by the COVID-19 pandemic.
Three Malaysian public-school teachers aged 30 to 44 were selected using purposive
sampling, with a minimum of five years of experience or involvement in online teaching during
the COVID-19 pandemic. The findings included annotations of the responses from the threeteachers, identified as P1, P2, and P3. Semi-structured interviews were conducted to gather
data, providing a deeper understanding of the overall experience compared to structured
components alone (Moustakas, 1994). End-of-term holiday closures led to interviews
conducted via online meeting rooms via Google Meet. Data collection began after ethicalapprovals from institutions and the Ministry of Education. Participants provided a plain
language statement, completed consent forms, and provided audio recordings. Interviews
lasted 80 minutes, and audio recordings were transcribed and stored securely.

973

The interview process involved gathering demographic information, exploring teachers'
experiences with technology in the classroom, and their attitude towards being digitally agile
teachers. The transcribed data were then structured by using thematic analysis (Braun &Clarke, 2006) as a ‘coding frame’. Data analysis involved recognizing and classifying themes
connected to flexibility characteristics that support teachers in becoming digitally agile. Data
collected from the participants were coded based on themes and analyzed based on them.
Firstly, the phrases and sentences from the data were highlighted and coded according to themeaning of words and sentence structure. The coded data were then consolidated and themes
were developed.

4. Findings and Discussion
4.1 Adapting and Thriving in the Digital Classroom
The teachers ranked their digital teaching and learning knowledge as proficient, withmost using digital technologies during online learning. They found it helpful, easier, andenhanced their performance. However, their utilization of digital tools decreased after theschools reopened for physical classes. While two teachers continued to employ digitaltechnologies, one teacher has stopped using them. From the thematic analysis, this theme
encapsulates the essence of digital agility and flexibility, encompassing the underlying themesof (1) students’ engagement, (2) professional growth, and (3) instructional effectiveness. The
participants emphasized the necessity for them to adapt their teaching practices to navigate
the digital landscape, creating dynamic classroom environments, fostering professional
growth, and enhancing instructional effectiveness by adopting an adaptable mindset andleveraging technology for engaging learning experiences.
This section discusses students’ engagement, the first theme of the findings. The
teachers said that they have used several web-based applications online during class. “While
doing the Google Meet, after the session I use Quizziz and usually I'll ask them to give answers
in the chat box.” (P2) and “I used an app on the internet called The Wheel of Fortune. I used
the wheel of fortune and made a lucky draw. And I used the medium at the time, Telegram; I
used Telegram using video chat, so I used both Telegram and Wheel of Fortune.” (P3). They
explained that by trying several platforms, it increases students’ participation and helps
teachers’ ability to control and manage the class. P2 explains that "… maybe like doing somequiz and then the chat box so they get excited.” P3 in the interview highlights that “When aname was selected, the students got excited. … They’d spread the news, telling their friends
that I do lucky draws in my class.” While P1 stated that “...I don't really know how to switch
between devices but after several classes it becomes smoother.” In other words, teachers
who are flexible in finding technology integrated strategies in teaching can increase students’
engagement. These findings are in line with a study conducted by Park and Park (2020) who
found that flexibility helps individual to solve problems, achieve goals and perform effectively.
Professional growth served as the second findings. According to P2, their ability to beadaptable allowed them to gradually overcome unforeseen circumstances “…we can adapt tothe situation slowly, but we can adapt. Like myself, so, like we start from zero but everything
we learned by ourselves (and) we managed to overcome the situation …” This findings
supported by a study by Barak and Levenberg (2016) which they highlighted that the readinessto participate in unknown settings and enables a learner to efficiently interact with difficult
situations. P3 states that online learning requires teachers’ technological proficiency tonavigate online tools and thus, by being flexible in responding to this situation, it benefits
teachers to expand their digital literacy skills “At that time, most people were more familiar with
Google Meet because this feature was very new to Telegram. But after I got the hang of it, it
was easier for me.” It is advantageous for an individual's flexibility to successfully cope with
online learning to provide them the technological skills essential to handle unforeseen changes(Răducu & Stănculescu, 2021), which supported the finding.
The third findings focuses on the instructional effectiveness which can be done byteachers who are flexible related online teaching and learning. They went through several
digital platforms and chose platforms that could be more effective to teach students digitally974

“…try a few apps that is east to be integrated during lessons. So, in the end, I used WhatsApp
and then in order to check the answers as well I use another app…” (P1). P1 explained thatfrom the chosen platforms, s/he found out it ease students’ evaluation “…so that it’s easier tolink the checked worksheet or the products that the kids did to Google Classroom.” P3 statedthat online lessons is intimidating but s/he was able to optimally prepared class materials using
digital tools “…the preparation was paperless and actually faster.” According to Lübke et al.
(2021), flexibility play a significant role in explaining teachers’ intention to use strategies forstudents’ achievements and thus, it is evident that flexibility shape teachers’ attitude towards
the effective teaching
Despite being able to understand teachers’ motivational factors of being flexible related
to teaching online in this study, the researcher noticed that the data inadequate considering
only two teachers teach using digital technology after class resumed physically. It is important
to note that students currently attending schools that they are characterized as experts inunderstanding technology (Cruz & Díaz, 2016). Poláková and Klímová (2019) suggest thatmobile devices enhance students' knowledge, prompting teachers to update traditional
teaching methods and create a modern learning environment. However, challenges in utilizing
technology during physical classes should not be overlooked.

4.2 Teachers’ Adaptability to Online Teaching
In this study, the teachers expressed various ways of flexibility specially to navigate
the digital teaching environment through their adaptability in response to a new andchallenging of online lessons. The teachers’ ways of adaptability were divided into two main
themes: (1) Teachers as learners, and (2) Supportive network contribution.
All teachers strongly agreed that they had to come out with solutions while teaching
online during the pandemic. P1 says that s/he need to find ways to conduct his/her class “…I
have to think how to incorporate digital and hardcopy resources…” while P2 says “I have tothink something creative to attract the children and focus.” P3 also echoes the samestatements when s/he states “…I was looking for a way to get this student to come to myclass.” They expressed that through learning, they were able to utilize new digital technologies
in online teaching. P1 explains that “I prepared my materials online, but I upload them in thegroup or Google Classroom … I borrowed my husband’s tablet so that I can use two screensat once to show solutions in doing Math questions … I attend Google Training, webinars,
watching videos, that were sent by PPD officers and school.” P2 states that h/she were
anxious of using online meeting but managed to conduct class well after several trial anderrors “…I did Google Meet first time …I was panicking because it was like trial-and-error
session. The, I managed. Actually, I was learning at that time.” It is crucial for teachers to stay
current with recent digital teaching needs, by learning, it enables the best adaptability to online
learning because the learner and the meaning are prioritized over the content and the activity(Răducu & Stănculescu, 2021).
In addition, the teachers’ adaptability to online teaching were contributed by theirsupportive network. A teacher commented that his/her colleague help to get knowledges on
how to use digital technology to conduct online lesson “Sometimes my friends will give me
ideas like why don’t we use this even they said we can play Kahoot games and everything’s
right so I learned from friends.” (P2). P3 concurred that using digital technology in partnershipwith other educators increases their desire in doing so over doing it independently “Like Excel,
there are some teachers who I can talk about the internet, the materials they found. When theteam is driven, we feel excited, when the team is one-man show, that’s when things become
difficult.” It also helps teachers when involves time constraints “…it’s quite helpful because
sometimes we don’t really have the time to do everything something you don’t have time toeven think for that thing so that’s when collaborating actually helps.” (P2). The findings fromall three teachers highlighted the ease with which a supportive network truly contributes to theusage of utilizing modern technology throughout the pandemic. It is notable that teachers who
work in supportive environments can increase the quality of their teaching in a variety of ways,
including through teacher cooperation (Podolsky & Darling-Hammond, 2019).

975

Adaptability in teachers is important to keep up with constant evolving learning
environment especially in mastering technological skills. However, teachers who find thisintimidating need support to boost their confident and competence, even post-COVID-19,
school might have possibility to close again (Guttentag,2020). Lieberman (2020) emphasized
five strategies to support online adaptability; (1) Providing collegial guidance, (2) Evaluating
technology tools collaboratively, (3) collecting data on effective baseline use, (4) Considering
online learning applications beyond mandatory, (5) planning intentional technology use toaccommodate potential situations.
In summary, based on the findings and discussion of the preliminary study, it is evident
that flexibility does support teachers to be digitally learning agile. Flexibility support teachers
in many ways. First, being flexible allow teachers to embrace and adopt new technologies by
exploring and integrating various digital tools and online platforms (Ulanday et al., 2021).
Second, it helps educators to adapt to any changes by staying updated on latest
advancements and learn new digital tools (Mardiana, 2020). Third, flexible teachers are able
to facilitate collaborative and interactive learning between other teachers or students(Kariippanon et al., 2019). Thus, by embracing flexibility, teachers can become digitally agile,
adapt to new technologies, exploring digital sources, and customizing instructional delivery inthe online teaching environment. This adaptability enables teachers to effectively utilize digital
tools and platforms to facilitate meaningful learning.

5. Conclusions
In conclusion, this preliminary study has added to the corpus of knowledge on flexibility
and how it affects teachers’ digital learning agility. The literature review has emphasized thevalue of flexibility in current educational environments and how it might help teachers be
nimble. The qualitative research methodology used in this study, which employed semistructured interviews with public school teachers in Malaysia, has been detailed in themethodology section. The research has shown what motivates teachers to be adaptable when
they teach online. The study also revealed how teachers prepared for online instruction andfound methods to adjust to unanticipated changes.
Overall, this study has contributed to existing literatures on flexibility and digitallearning agility among teachers. The findings of this preliminary study may serve as the basis
for the extensive investigation that could provide best practices to be shared with educators,
contributing to their professional development and growth. Besides, this study can bebeneficial to inform policy makers for policy development and decision-making at various
levels. Prior to this study, emphasis was mostly placed on instructors' experience in supporting
their ability to be flexible. Future research is advised to focus on larger concerns, such as thedifficulties teachers may have in adapting to the use of digital technology in physical classes
and how these difficulties can influence current teaching methods.

Acknowledgements
This study is supported financially by the Fundamental Research Grant Scheme (FRGS) of
the Malaysian Ministry of Education (FRGS/1/2022/SSI07/UPM/02/1). The assistance of the
Research Management Center (RMC) of Universiti Putra Malaysia in coordinating anddistributing funds for this research is greatly appreciated.

References
Allen, J. (2016). Conceptualizing learning agility and investigating its nomological network. [Doctoral
dissertation, Florida International University]. FIU Electronic Theses and Dissertations.
Barak, M., & Levenberg, A. (2016). Flexible thinking in learning: An individual differences measure forlearning in technology-enhanced environments. Computers & Education, 99, 39–52.
Bedford, C. L. (2011). The role of learning agility in workplace performance and career advancement.
[Doctoral dissertation, University of Minnesota]. University of Minnesota Digital Conservancy.
https://hdl.handle.net/11299/109995.

976

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in
Psychology, 3(2), 77–101.
Brzychcy, K. (2019). The school institution viewed as an organisation and its environment. WSB
Journal of Business and Finance, 53(1), 107-117.
Burke, D. W., & Smith, D. D. (2018). Burke learning agility technical report v3. 3. A guide for learning
about learning agility. https://easiconsult.com/wp-content/uploads/2018/10/burke-learningagility-inventory-technical-report.pdf
Corpus, R. A., Dizon, R. R., & Hernandez, A. C., (2022). Teachers’ flexibility and ict-adeptness in the
now normal in education. European Online Journal of Natural and Social Sciences:
Proceedings, 11(1 (s)), pp-212.
Cruz, F. J. F., & Díaz, M. J. F. (2016). Generation z's teachers and their digital skills. Comunicar.
Media Education Research Journal, 24(1).
Csordás, A. (2020). Diversifying effect of digital competence. AGRIS on-Line Papers in Economics
and Informatics, 12(1), 3-13.
De Meuse, K. P. (2017). Learning agility: Its evolution as a psychological construct and its empirical
relationship to leader success. Consulting Psychology Journal: Practice and Research, 69(4),DeRue, D. S., Ashford, S. J. & Myers, C. G. (2012). Learning agility, in search of conceptual clarity
and theoretical grounding. Industrial and Organizational Psychology, 5(3) 258-279.
Joiner, B., & Josephs, S. (2007). Developing agile leaders. Industrial and commercial training, 39(1),35-42.
Juhdi, N., Pa'wan, F., & Hansaram, R. M. S. K. (2012). Examining characteristics of high potential
employees from employees' perspective. International Journal of Arts & Sciences, 5(7), 175.
Kariippanon, K. E., Cliff, D. P., Lancaster, S. J., Okely, A. D., & Parrish, A.-M. (2019). Flexible learningspaces facilitate interaction, collaboration and behavioural engagement in secondary
school. PLOS ONE, 14(10), e0223607.
Lieberman, A., & Miller, L. (2005). Teachers as leaders. In The educational forum (Vol. 69, No. 2, pp.
151-162). Taylor & Francis Group.
Lieberman, M. (2020). Like it or not, K-12 schools are doing a digital leapfrog during COVID19. Education Week, 39(34), 13.
Lombardo, M. M., & Eichinger, R. W. (2000). High potentials as high learners. Human Resource
Management, 39(4), 321-329.
Lübke, L., Pinquart, M., & Schwinger, M. (2021). the role of flexibility in the realization of inclusive
education. Sustainability, 13(8), 4452.
Mardiana, H. (2020). Lecturers’ adaptability to technological change and its impact on the teachingprocess. JPI (Jurnal Pendidikan Indonesia), 9(2), 275.
Mitchinson, A. & Morris, R. (2014). Learning about learning agility. White Paper, Center for Creative
Leadership, 1-18.
Moustakas, C. E. (1994). Phenomenological research methods. Sage Publications, Inc.
Murphy, S. M. (2021). Learning agility and its applicability to higher education. [Doctoral dissertation,
Columbia University]. Columbia Academic Commons.
Podolsky, A., Kini, T., & Darling-Hammond, L. (2019). Does teaching experience increase teacher
effectiveness? A review of US research. Journal of Professional Capital and Community, 4(4).
Poláková, P., & Klímová, B. (2019). Mobile Technology and generation z in the english language
classroom – A preliminary study. Education Sciences, 9(3), 203.
Răducu, C.-M., & Stănculescu, E. (2021). Adaptability to online teaching during covid-19 pandemic: amultiple mediation analysis based on kolb’s theory. International Journal of Environmental
Research and Public Health, 18(15), 8032.
Seale, J., Draffan, E. A., & Wald, M. (2010). Digital agility and digital decision-making: conceptualising
digital inclusion in the context of disabled learners in higher education. Studies in higher
education, 35(4), 445-461.
Ulanday, M. L., Centeno, Z. J., Bayla, M. C., & Callanta, J. (2021). Flexible learning adaptabilities in
the new normal: E-learning resources, digital meeting platforms, online Learning Systems andLearning Engagement. Asian Journal of Distance Education, 16(2).
Varsney, D. & Varshney, N.K. (2020). Workforce agility and its links to emotional intelligence andworkforce performance: A study of small entrepreneural firms in India, Wiley Periodicals, Inc.
GBOE. 2020;39(5):1-11. wileyonlinelibrary.com/journal/Joe.
Warkentien, M. (2016). Teachers as Strategic Classroom Leaders: The Relationship of Their
Cognitive and Behavioral Agility to Student Outcomes and Performance Evaluations [Doctoral
dissertation, Florida Atlantic University]. ProQuest Dissertations and Theses Global.

977" !   !" " "   #" !
#" !"$  #" ! #"  
   


" "$$%&&!  &!  $%&! !
! #' !&&
   
  
      
  
40(6@<,3052 *</2 ,+< /2  4?@D9>7B?2?D93C1C1@5417?7931<D??<9C3?==?><I?2C5BF549>C38??<
21C54@B?7B1=CB5<1D54D?54E31D9?>8?G5F5BC31>DB5C51B3881C255>3?>4E3D54
?> D51385BC @5B359F54 381<<5>75C D85B59> *89C CDE4I D8B?E78 9>D5BF95G9>7 ! 
D51385BC> 19=54D?@B?259>D?D859B@5B359F5421BB95BC1>45>12<5BC1CG5<<1C
D853?BB5C@?>49>79=@13DC9>D85C9DE1D9?>?681B>5CC9>7B?2?D93C9>54E31D9?>*89C
CDE4IB5F51<54D81D1@1BD6B?=D85@B5C5>35?6C?=53?==?>5HD5B>1<1>49>D5B>1<
21BB95BC  5>12<5BC 1 >5G 613D?B >1=5<I MD538>?<?7IN 3?>35B>9>7 D85 AE1<9DI 1>4
6E>3D9?>C ?6 D538>?<?7I 5=5B754 *89C G?B; @B?F945C D85 695<4 G9D8 9>C978DC 9>D?
45C97>9>71>49=@<5=5>D9>7D51385B@B?65CC9?>1<45F5<?@=5>DDB19>9>7?>54E31D9?>
9>! 3?>D5HDC 4E31D9?>(?2?D93C!

*51385BC1BB95BC>12<5BC 

$05*,;/,),.05505.6-;/0:+,*(+,(9;0-0*0(305;,330.,5*,,+<*(;065/(:),,59,.(9+,+(:(.36)(33@04769;(5;050;0(;0=,(*96::,3,4,5;(9@;6/0./,9,+<*(;065, . /0<,;(3 (0,;(3 0,;(3  "5;676-;/(;;/,:0.50-0*(5;:6*0,;(3047(*;6-*(33:-69;/,
05*69769(;0656-9,3(;,+256>3,+.,05,(930,9,+<*(;065(3:;(.,: #,*,5;3@;/,9,/(:),,5
(56;()3,7<:/;6+,=,367*<990*<3(05 ,+<*(;065>693+>0+,>0;/96)6;0*:,4,9.05.(:(70=6;(3;663-69,5/(5*05.3,(9505.(5+;,(*/05.'<,,;(3  6>,=,9:*(5;(;;,5;065/(:),,5.0=,5;6;/,*/(33,5.,:;,(*/,9:-(*,(40+;/,:<9.,6-,+<*(;06505;/, +64(057(9;0*<3(93@9,.(9+05.;/,<:,6-96)6;0*: &/03,79,=06<:9,:,(9*//(:05=,:;0.(;,+;/,)(990,9:9,3(;,+;605;,.9(;05.5-694(;065(5+644<50*(;065:%,*/5636.@%05;6,+<*(;0659<5056:;96A(3040;,+9,:,(9*//(:-6*<:,+65;/,05;,.9(;0656-96)6;0*:,:7,*0(33@>0;/05*6<9:,: %/,(046-;/,79,:,5;:;<+@0:;6,?7369,;/,)(990,9:
(5+76;,5;0(3,5()3,9:7,9;(0505.;696)6;0*::<7769;,+3,::65:7,9*,0=,+)@ ;,(*/,9:
(:>,33(:/6>;/,:,-(*;69:40./;05-3<,5*,;,(*/,9:E67769;<50;0,:-69;,(*/05.>0;/96)6;0*: 76>,9,+ ,+<*(;065(3 96)6;0*: *(5 ), (5 ,--,*;0=, ;663 05 ;,(*/05.   :;<+,5;: 
256>3,+.,  ,=0*,: :<*/ (: 30*6; #6)6 (:;,9 (5+ " 6--,9 =(906<: 79,+,:0.5,+
46+<3,: ;/(; :<7769; +0--,9,5;  -<5*;065: 05*3<+05. -(*0(3 9,*6.50;065 ,46;065: :6<5+:
6)1,*;: *6369: (5+ .,:;<9, +,;,*;065  %,(*/,9: *(5 <:, 96)6;0*: ;6 05*9,(:, :;<+,5;:E
,5.(.,4,5; (5+ 46;0=(;065 $/(5. ,; (3  '(5. ,; (3  -6:;,9 *647<;(;065(3;/05205. /05.  :< $/(5. ,; (3  (5+ ,5/(5*, ;/, <5+,9:;(5+05. 6-
796.9(4405. *65*,7;: !(@(2 ,; (3  ,:70;, ;/, ()6=, ),5,-0;: ;,(*/,9: 4(@
,5*6<5;,9 =(906<: */(33,5.,: 05 05;,.9(;05. 96)6;0*: 05;6 ;,(*/05. 79(*;0*,: +<, ;6 +0=,9:,
9,(:65:/05.:<



978

(*;69:(--,*;05.;,*/5636.@05;,.9(;065*(5),*(;,.690A,+05;6,?;,95(3(5+05;,95(3 ?;,95(3-(*;69:05*3<+,(**,::;6*647<;,9:69+,=0*,:05;,95,;*655,*;0=0;@+0:;90*;7630*0,:;,(*/,9;9(0505. ;,*/50*(3 :<7769; 7,,9 :<7769; (5+ ;04,  5;,95(3 -(*;69: ,5*647(:: 7,9:65(3),30,-::,3-,--0*(*@79,=06<:,?7,90,5*,:=0:065:6-;,*/5636.@05;,.9(;065(5+*65-0+,5*,
%65+,<9,;(3  64465)(990,9:4,5;065,+05*3<+,05:<--0*0,5;/(9+>(9,69:6-;>(9,05(+,8<(;, ;,*/50*(3 :<7769; (5+ 9,:0:;(5*, ;6 ;,*/5636.@ <:,  5()3,9: *(5 05*3<+,
*6479,/,5:0=, 796-,::065(3 ;9(0505. 9,:6<9*,: (5+ -05(5*0(3 :<7769;  %/,9, 0: 3040;,+
30;,9(;<9,;/(;,?7369,:;/,)(990,9:(5+,5()3,9:6-05;,.9(;05.96)6;0*:05;63,::65:  

3,=,5/05,:, ;,(*/,9:3(),3,+;/96<./>/6(9,7065,,9:05:*/663,+<*(;065
7(9;0*07(;,+05:,40:;9<*;<9,+05;,9=0,>:;6,?7369,;/,)(990,9:(5+,5()3,9:6-05;,.9(;05.
96)6;0*: 05;6  3,::65:  %/, 05;,9=0,>: >,9, (5(3@A,+ <:05. ( ;>6:;,7 *6+05. (7796(*/$(3+(B( %/,-09:;7/(:,*65+,5:,++(;(05;6+,:*907;0=,(5+05-,9,5;0(3*6+,: %/,
:,*65+ 7/(:, ,4736@,+ 7(;;,95 (5+ -6*<:,+ *6+05. ;6 .,5,9(;, ;/,4,:  65:;(5;
*647(90:654,;/6+:>,9,<:,+;6/0./30./;;/,*64465)(990,9:(5+,5()3,9:   

%>6 7904(9@ ,?;,95(3 )(990,9: >,9, 0+,5;0-0,+ 3040;,+ 96)6;0*: (=(03()030;@ (5+ 05:<--0*0,5;
796-,::065(3;9(0505. ::;(;,+)@;/,;,(*/,9C;E:;66,?7,5:0=,;6796=0+,,=,9@:;<+,5;
>0;/96)6;0*:D;/,*6:;6-796*<905.96)6;0*:-69,(*/:;<+,5;0:796/0)0;0=,3@/0./047(*;05.
;/,093,(9505.,?7,90,5*, %,(*/,9:(3:6,?79,::,+*65*,95:()6<;;/,05:<--0*0,5;;9(0505.;6<:, 96)6;0*: 05 ;/,09 ;,(*/05.  "5, 9,4(92,+ C&0;/6<; ,?7,9; .<0+(5*,  9,:69; ;6 65305,
9,:6<9*,:>/0*/4(2,:4,<5*,9;(05()6<;;/,(**<9(*@  D76::0)3,9,(:65*6<3+),;/,40569:;(;<:(5+76:0;06505.6-(:(+0:*07305, *6<9:,:(9,6-;,5,4),++,+>0;/05;/, 05-694(;065 (5+ ;,*/5636.@ % :<)1,*; >/0*/ 0: =0,>,+ (: ( 40569 :<)1,*;  %/,@+,:*90),+ % ;,(*/,9: (: C&, (9, ;/, 4(9.05(30A,+ ;,(*/,9: >/6 ;,(*/ ;/, 4(9.05(30A,+
:<)1,*;:6:*/66305=,:;4,5;056<9+0:*07305,0::,=,9,3@3040;,+ D0./;;,(*/,9:9,769;,+;/(;(3(*26-*65-0+,5*,0505;,.9(;05.96)6;0*:0:;/,7904(9@05;,95(3)(990,9 "5,9,4(92,+
C @ (*(+,40* )(*2.96<5+ 0: 56; 05  (5+  (4 56; -(4030(9 >0;/ 96)6;0*:  5;,.9(;065 6-
96)6;0*:-,,3:90:2@;64, D5-(*;0;0:*64465-69;,(*/,9:>/6;,(*/%69,=,505
;656;/(=,4(169,+05*647<;,9:*0,5*, %/,3(*26-*65;,5;256>3,+.,(5+96)6;0*:
796-0*0,5*@*65;90)<;,:;6;/0:)(990,9 &,0+,5;0-0,+(5,>*(;,.69@6-)(990,9;/,;,*/5636.@-(*;69>/0*/>(:4,5;065,+)@(33
7(9;0*07(5;: 0--,9,5;-964;@70*(3,?;,95(3)(990,9::<*/(:(3(*26-/(9+>(9,96)6;0*:;/,
;,*/5636.@-(*;69*65*,95:;/,8<(30;@(5+-<5*;065:6-;/,96)6;0*: "5,;,(*/,99,769;,+
C$;<+,5;:5,,+65,96)6;0*;/(;*(5*(99@(33*65*,7;: 4,(5:;<+,5;:*(53,(95=(906<:
(:7,*;:6-;/96<./05;,9(*;065>0;/(:05.3,+,=0*, D6>,=,9;,(*/,9:6-;,55,,+;6
<;030A,+0--,9,5;205+:6-96)6;0*:;6;,(*/*699,:765+05.256>3,+., 7(9;-964;/,+,4(5+
6-*655,*;05.;6+0--,9,5;*65;,5;256>3,+.,;,(*/,9:,?7,*;96)6;0*:;6*9,(;,(*65;,?;
-69:;<+,5;:;6:63=,(<;/,5;0*796)3,4:(:>,33 :65,*644,5;,+C-96)6;0*:*(556;/,37
4,;6*65;,?;<(30A,(796)3,4;6),:63=,+>033),3,::05*305,+;6<:,0;054@*3(::9664:
 D %,(*/,9: (3:6 ,47/(:0A,+ ;/, ()030;@ 6- 96)6;0*: ;6 *(7;<9, :;<+,5;:E 05;,9,:;: C%/,
7904(9@ (04 6-  ,+<*(;065 -69   :;<+,5;: 0: ;6 -6:;,9 ;/,09 05;,9,:;: 05 ;/, 76>,9,+
>693+ 5;,.9(;05.+<33(5+<505;,330.,5;96)6;0*:054@*3(::,:0:*/(33,5.05.),*(<:,;/,@-(03;6,5.(.,:;<+,5;:E(;;,5;06565;,*/5636.@ D699,:765+05.3@>,(3:60+,5;0-0,+,5()3,9:<5+,9,?;,95(305;,95(3(5+;,*/5636.@-(*;69: !05,;,(*/,9:(*256>3,+.,+;/,,--,*;0=,5,::6-96)6;0**647,;0;065:(:,?;,95(3,5()3,9: "5,:(0+C#6)6;0**647,;0;065:73(@(=0;(3963,0546;0=(;05.:*/663:(5+4,;605*69769(;,
96)6;0*:05;6;/,3,(9505.,5=09654,5; $;<+,5;:5,,+(73(;-694;6,?(405,;/,093,(9505.
6<;*64,:(5+*647,;0;065:796=0+,(5,?*,33,5;67769;<50;@;6,=(3<(;,:;<+,5;:E3,(9505.

979

7,9-694(5*,  D #,.(9+05. 05;,95(3 ,5()3,9: ;,(*/,9: =(3<,+ 76:0;0=, (;;0;<+,: ;6>(9+:;,*/5636.@ 4,5;065:(5+*65:;9<*;0=0:;),30,-:4,5;065: "5,*644,5;,+C-0943@),30,=,;/(;:;<+,5;:*65:;9<*;;/,09<5+,9:;(5+05.6-;/96<./3,(9505./(5+:65;(:2: @<:05.96)6;0*::;<+,5;:*(5.,;(505+,7;/<5+,9:;(5+05.6-;/,796)3,4(5++,(3>0;/;/,:,;(:2:,--,*;0=,3@ D05(33@(33;,(*/,9:(.9,,+;/(;<:,9-90,5+3@96)6;0*:0:(*90;0*(3,5()3,9<5+,9;/,;,*/5636.@-(*;69 %/,@,?7,*;,+96)6;0*:;6),,(:@;6<:,+,=0*,:;/(;
405040A,*64730*(;065:-69)6;/;,(*/,9:(5+:;<+,5;: 
%/, 79,30405(9@ -05+05.: /0./30./; 2,@ )(990,9: (5+ 76;,5;0(3 ,5()3,9: 7,9*,0=,+ )@  ;,(*/,9:>/,505;,.9(;05.96)6;0*:05;6*6<9:,:>/0*/.,5,9(33@(9,(30.5,+>0;/79,=06<:
9,:,(9*/65:;<+@05.,?;,95(3(5+05;,95(3-(*;69:6-;,*/5636.@05;,.9(;065 ,:0+,:(<508<,
-(*;69;/,C;,*/5636.@-(*;69:D,4,9.,+-964;/,9,:<3;:6-;/,*<99,5;:;<+@05;/,*65;,?;6- ,+<*(;065  %,(*/,9: (9, *65*,95,+ >0;/ ;/, -<5*;065: (5+ 8<(30;@ 6- ;,*/5636.@ 0 , 96)6;0*:05;/0:*65;,?;0;:,3- %/,;,*/5636.@-(*;6905;,9:,*;:>0;/)6;/,?;,95(3(5+05;,95(3-(*;69: 6905:;(5*,;,(*/,9:),30,=,+;/(;0-96)6;0*:+0+56;76::,::;/,,?7,*;,+-<5*;065:
9,.(9+3,::6-;/,(=(03()030;@6-:<--0*0,5;9,:6<9*,:(5+:<7769;(5,?;,95(3-(*;6969;/,09
>0305,::;6<:,;/,4(505;,95(3-(*;690;>6<3+:;03379,:,5;()(990,9;6;,(*/05.256>3,+., 5 (++0;065 ;,(*/,9: 0+,5;0-0,+ 469, 05;,95(3 ,5()3,9: *647(9,+ ;6 ,?;,95(3 ,5()3,9: ;/(;
-(*030;(;, ;/, 05;,.9(;065 6- 96)6;0*: 05  3,::65:  %/, +(;( 9,=,(3 ( :;965. 05;,9,:; (5+46;0=(;065 (465. ;,(*/,9: ;6>(9+: 05*69769(;05. 96)6;0*: 05  *<990*<3( +,:70;, (5@
*65:;9(05;:;/,@40./;,?7,90,5*,+<,;6(3(*26-,?;,95(3:<7769; <;<9,9,:,(9*/:/6<3+(04(;+,:0.505.(5+0473,4,5;05.;,(*/,9796-,::065(3;9(0505.796.9(4:;6(++9,::;/,:,
)(990,9:,--,*;0=,3@ ++0;065(33@;/,047(*;6-;/,;,*/5636.0*(3-(*;6965;,(*/,9:+,*0:065:
9,.(9+05.;/,05;,.9(;0656-96)6;0*:>(99(5;:-<9;/,9,?7369(;065 

BE>#9>?CDB?J1 
"51B>9>7D?253?=51D51385B9>D85 CD35>DEBI*9>D57B1D9?>
9>>9D91<*51385B4E31D9?>9>89<5$*'#!$*) $#!#$!$,$ ),   
89>7/CE/
4E31D9?>1<B?2?D93C6?B45F5<?@9>73?=@ED1D9?>1<D89>;9>79>I?E>7
<51B>5BCCICD5=1D93B5F95G'#(  
89E*!#5>7819)!9>7-?>7)/5E>7/
B51D9?>1>45F1<E1D9?>?6
1@B5D5BD91BIBD969391<>D5<<975>353EBB93E<E='#() $#($#*) $#    
19/"9E'9> E?/ ?>7#)/819)"9>0
?<<12?B1D9F53?>CDBE3D9?>
?61BD969391<9>D5<<975>353EBB93E<E=9>@B9=1BIC38??<C$*'#!$# #' #*) $#  L
 
"9. 91>7#/ ?>7#)/081>7.819)
+>45BCD1>49>7=54931<CDE45>DCO
@5B35@D9?>C?61>42581F9?B1<9>D5>D9?>CD?G1B4<51B>9>71BD969391<9>D5<<975>35#)'#) $#!$*'#!
$#+ '$#"#)!('*! !)   $1I1;  !51>5 * "9>45> *  #?<>1B   *51389>7 8978 C38??< CDE45>DC 1BD969391<
9>D5<<975>35 2I @B?7B1==9>7 81D2?DC  # $ # #  ($$!(  (' #
%%! ) $#@@   )@B9>75B
)1<41K1 $ #"#*!$'&*! )) +'(''()175
)81>7. 91>70891>7!081>7/08E
6653DC?6B?2?D93C)*#31=@C?>
BEB1< 5<5=5>D1BI CDE45>DCO C5<65669313I 1>4 3?=@ED1D9?>1< D89>;9>7*) $#! #$!$,('#+!$%"#) L  
*?>45EB  ,1> B11;  BD=5B &   %DD5>2B59D"56DG938   +>45BCD1>49>7 D85
B5<1D9?>C89@25DG55>D51385BCO@5417?7931<25<956C1>4D538>?<?7IEC59>54E31D9?>CICD5=1D93
B5F95G?6AE1<9D1D9F55F945>35*) $#!#$!$,('#+!$%"#) 
/1>7%"19#-1>7/-6653D?61E7=5>D54B51<9DI21C54F9BDE1<54E31D9?>1<
B?2?D93C ?> @B?7B1==9>7 CDE45>DCO 5>:?I=5>D ?6 <51B>9>7 3?=@ED1D9?>1< D89>;9>7 C;9<<C 1>413145=9313895F5=5>D $"%*)'(*) $#   /E5# ?>7#)/19/
&5417?7931<45C97>?6! 1BD969391<9>D5<<975>3554E31D9?>
CICD5=1D93B5F95G*() # ! ),   980

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Collaborative design of a simulation-based
math classroom: Contradictions andsolutions between teaching and researchW濙濢濬濝濢澔澻濉濃a*, Bian WUb* & Dong LIc
aEast China Normal University, Chinab
East China Normal University, Chinac
The School Attached to Shanghai Caoyang No.2 High School, China* 51214108035@stu.ecnu.edu.cn
Abstract: Although co-design between teacher and researcher as a means could
effectively promote teachers’ professional development, little is known about theprocess of developing a simulation-based math classroom for compulsory education
by a co-design team between teachers and researchers. From a dual theoreticalperspective involving activity theory and boundary crossing, we investigate thecontradictions that emerged between teaching and research activity systems, and theway to deal with the contradictions. The results showed three main contradictions: (1)theoretical ideas versus practical knowledge, (2) simulation-based versus traditional
teaching design, and (3) generative interactions versus structured classroom culture.
The contradictions were resolved to varying degrees through the efforts of participants
who acted as brokers, mainly in terms of the process of perspective making and taking.
These findings not only guide how to maximize learning opportunities for teachers in
the co-design process with researchers but also enrich teacher education and activity
theory.
Keywords: Activity system, contradiction, boundary crossing, math classroom design,mathematics education research

1. Introduction
The increase in digital technologies and teaching tools has placed higher requirements onteachers’ professional capabilities to integrate technology into classroom practice. The
collaborative design between teachers and researchers has the potential to improveteachers’ teaching expertise and refine instructional practice by combining the uniqueexpertise of both teachers and researchers (Cai et al., 2017). Due to the different cultural
backgrounds of the co-design members, it is common for contradictions to arise incooperation (Qi et al., 2022). Properly handled contradictions could cause teachers andresearchers to critique existing practices or experiences and make an effort to explore new
solutions, which in turn can lead to innovation (Yan & Yang, 2019).
In traditional K-12 math classroom teaching, students tend to be polarized in theirunderstanding of mathematical relationships. Some students may struggle with graspingthese fundamental principles. Digital classrooms that include interactive simulations are
particularly valuable in math teaching and learning (Moeller et al., 2015), especially forinternalizing the understanding of complex mathematical concepts and relationships.
Currently, research focuses on the results or effects of computer simulations rather than on
the enactment process of simulations in actual math teaching during co-design. Therefore, inthis study, we aimed to identify the contradictions in the co-design process of a simulationbased math classroom as well as possible solutions to ameliorate them.

981

2. Literature Review
2.1 Computer Simulations for Math Teaching
Today, technology has become part of students’ exploration of knowledge. Computer
simulations, one of information and communications technology (ICT) applications, integrate
learning areas including cognition, affect, and psychomotility as one and allows students toimplement open-ended exploration and systematic experimentation (Ross & Bruce, 2012;
Rutten et al., 2015). The specific characteristic of simulation represents complex real-world
situations that can be changed by manipulating different parameters (Hillmayr et al., 2020).
Through simulations, students could visualize their topics and interact with the technology indoing activities (Garcia, 2020).
Math is most widely known to be challenging for some K-12 students, especially theunderstanding of abstract concepts and mathematical relationships. Interactive simulation
makes those sophisticated math concepts much easier to internalize for almost all students
as they observe the direct consequences of the changes they make (Buckley et al., 2004).
Most research focuses on the results or effects of computer simulations rather than on theprocess of teacher enactment of simulations in actual teaching. For example, Rutten et al.
(2012) reviewed that numerous researchers have investigated the efficacy of computer
simulations without measuring teacher influence or optimal instructional support.
Incorporating interactive digital tools into the math classroom requires theidentification of innovative teaching and learning approaches that are suitable for such tools.
This process involves identifying course components, making decisions regarding lesson
structuring, sequencing, and pacing, as well as instructional strategies for monitoring andresponding to student progress (Biggers et al., 2013; Remillard & Heck, 2014). Teachers’
beliefs about the value of classroom teaching and the role of technology in the classroom
significantly impact the enactment of these decisions and strategies. In cases that lackalignment, teachers may resist innovations or make substantial adaptations (Bates &Usiskin, 2016; Hermans et al., 2008). The collaborative design between teachers andresearchers could be a promising way to facilitate the development of teachers’ adaptive
expertise and address theoretical as well as practical issues of teaching and learning (Ko etal., 2022). There are at least two key players who have influence in a teacher-researcher codesign community for the use of computer simulations in math teaching: teachers andeducational researchers. In the co-design process, researchers provide educational
principles and strategies to facilitate the innovation of teaching approaches and learningtechniques. Teachers devise instructional activities to achieve the balance between
pedagogical efficiency and innovative teaching practices (Goodyear & Casey, 2015).

2.2 Contradictions and Boundaries in Activity Systems
Teachers and researchers have been guided by their own cognitive paradigms andbackgrounds for a long time, each pursuing distinct cognitions, understandings, andobjectives in teaching. It is not surprising that certain contradictions may arise during mutual
co-design, and the parties involved play a dynamic game in the contradiction (Goodyear &Casey, 2015). Cultural-History Activity Theory (CHAT) views human activity as inherently
social and culturally embedded, emphasizing the interplay between individuals and theirenvironment. From the perspective of CHAT, contradictions could be seen as historicallyaccumulating structural tensions within and between activity systems (Ko et al., 2022).
Collectives in the same activity system with a common goal orientation communicate,interact, and influence each other. Each activity system is composed of six core elements,
including subject, object, community, mediating tools, rules, and division (Engeström, 2001).
Based on CHAT framework, Qi et al. (2022) examined the contradictions and their plausible
strategies between the mathematics teaching system and university research system,
founding some typical contradictions such as tensions between school regulations and theobject to promote teachers’ professional development, as well as traditional teaching design
versus research-informed teaching design.

982

The presence of contradictions is not always an obstacle to progress. Instead, they
may trigger new attempts and changes by the subjects in the activity systems if identified
and resolved properly (Yan & Yang, 2019). Potari et al. (2019) exposed the contradictions
across teaching, research, and policy activity systems and their solutions, indicating
identification and coordination as two main ways in the teaching and research activity toformulate a collectively meaningful object. In the middle of activity systems, the boundary
represents the cultural difference and the potential difficulty of action and interaction acrossthese systems but also represents the potential value of establishing communication andcollaboration (Akkerman & Bakker, 2011). When collaborating across the boundaries ofactivity systems, teachers or researchers may need to enter into unfamiliar territory andcombine ingredients from disparate sites to achieve hybrid situations (Engeström et al.,1995). This is commonly referred to as boundary crossing. It could prompt the subjects toquestion and critique existing practices or experiences, leading them to explore new
solutions and ultimately generate innovation.
In our study, the CHAT and boundary crossing would be used to investigate thecontradictions that emerge and the way to deal with them during a co-design process of asimulation-based math classroom across two key activity systems: teachers and educational
researchers. Two research questions should be addressed:
RQ1: What were the emerging contradictions in the co-design process of asimulation-based math classroom?
RQ2: How did the participants deal with these contradictions between the teachingactivity system and the research activity system?

3. Method
3.1 The systemic context and participants
Our co-design team was composed of two groups, including three researchers (oneuniversity professor, one teaching and research staff, and one graduate student) and fiveteachers (three in primary school and two principals). We use the acronyms R (researcher),T (teacher), or their combinations (RT) to refer to the activity systems to which participants
belong. All the researchers had extensive research experience in educational technology,
which could be seen as full members of the research activity system. These threeresearchers were represented by the abbreviations R1, R2, and R3, respectively. Teachers
could be viewed as full members of the teaching activity system. They were all exposed toperiodic teaching research activities with the eagerness to learn about new reform ideas toimprove their teaching. Additionally, one of the principals had partial membership in the
research activity system as he held a Master’s degree in education and also played a keyrole in numerous educational research projects focused on classroom teaching supported byinteractive simulations. This principal was represented by TR1, and the others wererepresented by T1, T2, T3, and T4.
The PhET is a site of research-based interactive computer simulation in teaching andlearning physics, math, and other sciences (Garcia, 2020). Students engage in exploratory
learning through the use of a simulator called 'Number Line: Integers' from PhET, which
provides animated, interactive, and game-like environments. The whole team worked
collaboratively for about two months (March 2023-April 2023) and prepared three versions ofsimulation-based mathematics instructional design for trial teaching in three different fifthgrade classes, 35 students in each. After each trial teaching, teachers and researchers
attended a seminar meeting to reflect on and improve the current instructional designcollaboratively.

3.2 Data collection and analysis
The collaboration took place through three post-lesson meetings (M1, M2, and M3) andnumerous meetings in subgroups. After attending the final post-lesson meeting, the teacher

983

who teaching this mathematics course with the support of PhET was invited to beinterviewed. In the following, this interview was represented by the abbreviation I1. Theinterview questions include four main aspects: the process of using PhET simulation tool in amathematics classroom; the experience of co-design and improving such a digitalclassroom; the perceptions of different opinions of teachers and researchers; the meaning oftechnology innovation and its relation to research or teaching. The data analyzed included
the following: (1) video recordings of the three post-lesson meetings, (2) audio recordings ofthe interview, and (3) three versions of simulation-based mathematics instructional design.
To answer our first research question, the CHAT framework was used to identify andtrack the emerging contradictions between the teaching activity system and the research
activity system in the co-design context. The contradictions were characterized by theircontent, the involved activity systems, and the elements of Engeström’s (2001)
interconnected triangles (see Figure 1).

Figure 1. Interacting activity systems
To address the second research question, the construct of boundary crossing wasused to examine how the management of the contradictions by the team members
contributed to the design process. Akkerman and Bakker (2011) discerned and summarized
four types of learning potential of boundary crossing: (a) Identification involves recognizing
boundaries, acknowledging diverse perspectives, and bringing a renewed sense of different
practices and related identities. (b) Coordination refers to cooperating efficiently in
distributed work and dialogue between diverse partners to maintain the flow of work even in
the absence of consensus. (c) Reflection involves recognizing and articulating distinctions
between practices, thereby making explicit one’s understanding and knowledge of aparticular issue (perspective making), as well as looking at oneself through the perspectivesof other worlds (perspective taking). (d) Transformation refers to the engagement ofparticipants from diverse systems in constructive activities that result in significant changes
to existing practices and even create novel cultural forms. People who cross boundaries
serve as conduits for the introduction of elements from one practice into another, such as
brokers and boundary crossers (Star & Griesemer, 1989). The identified contradictions
indicated the boundary with the boundary brokers or objects. The process of dealing withthis boundary was coded by such four types of learning at the boundaries.

4. Results
We structure the results based on the main contradictions that emerged while preparing andrevising simulation-based mathematics instructional design. Three main contradictions wereidentified and coded, including theoretical ideas versus practical knowledge, simulationbased versus traditional teaching design, and generative interactions versus structured
classroom culture. In each of the three subsequent subsections, we present throughillustrative examples the identified contradictions, elements involved in activity systems, andhow these contractions were managed through boundary crossing to jointly address
research questions one and two.

4.1 Theoretical ideas versus practical knowledge

984

The first contradiction emerged in the co-design team’s initial post-lesson meeting when theyreflected on the implementation of a simulation-based math classroom utilizing PhET.
Mathematics teachers made decisions in the teaching process and activities with theirstereotypical thinking that relied on the content of mathematics textbooks. At this phase,
conflicting perspectives between teaching and research activity systems emerged, indicating
the gap between educational theories and practical knowledge. We provide example 1 toillustrate the elements of the activity systems involved in the contradiction and the process
for dealing with it.
4.1.1 Example 1
In terms of teaching strategies, R1 suggested incorporating embodied cognition theory in thelearning science field into the practical math classroom, and explained how to use, such as“hands-on or body movement”, “changing the size and number of paces to reach a certain
number on the number axis.” One of the mathematics teachers T1 questioned the matchingof embodied cognition theory with the content sequence of the textbook. T2 who taught thiscourse expressed similar arguments:
Physical activity is not the main focus of this lesson, right? In the first class, students
need to know what the number axis is. And then in the second lesson, they should be
able to look at numbers on it and describe where they are. Using body language
representations would work better for these previous two lessons. But now, students
need to compare the magnitude of numbers on the number axis. (T2, I1)
Contradiction: object and tools in research versus tools and rules in teaching
In example 1, it seems that the emerging contradiction is between the object (e.g., using
educational theory to transform the mathematics classroom) and tools (e.g., embodied
cognition theory) of research activity and the tools (e.g., mathematics textbook) and rules(e.g., organize teaching activities according to the sequence of mathematics textbook) ofteacher communities. It appears to be prevalent that such contradiction emerges in theteacher-researcher co-design environment. Teachers often perceive researchers’ solutions
as too theoretical and insufficiently practical for implementation in real classrooms (Shrader
et al., 2001). In contrast, researchers often view teachers’ limited content knowledge as ahindrance to their effective contribution to co-design efforts (Roschelle et al., 2006). This
contradiction reflects the distance between educational theory and pedagogical practice, andthe new theory-oriented pedagogy has impacted teachers' instructional design according toestablished rules and textbook resources.
Boundary crossing remained in the process of identification
Dealing with the contradiction of “theoretical ideas versus practical knowledge” was achallenge for the co-design team. As a broker at the boundary, the researcher expressed theconcept of embodied cognition in a language accessible to teachers and proposed concrete
examples to apply this theoretical concept. Teachers were aware of the different
perspectives of the researcher and the teacher insisted that the "physical activity" did not
apply to teaching the knowledge of comparing the magnitude of numbers on the number
axis. The boundary between the two activity systems explicitly signals an identification
process demarcating the two communities’ goals, tools, and rules.

4.2 Simulation-based versus traditional teaching design
This contradiction occurred during the first and third post-lesson meetings almost throughout
the whole co-design process. The role of educational researchers was critical in formulating
the common co-design goals, as researchers identified educational goals and provided
PhET tools to achieve them based on their numerous similar research experiences.
Teachers have hardly ever experimented with new educational resources in the classroom
before, bringing many obstacles to the introduction of simulation resources. We provide two985examples to illustrate divergent views about the balance between simulation-based teaching
design and traditional teaching design, as well as the discussions for managing them.
4.2.1 Example 2
The goal proposed by researchers was to use simulation technology to drive innovation inclassroom teaching. T2 expressed his confusion about how to achieve it, “We are using
PhET software for the first time and have no clue how to use it to help me teach math.” T2
further pointed out that the software was not effective for the first trial teaching, “Thesoftware was used less in math classes and did not show its benefits.” After the first postlesson meeting, TR1 provided T2 with several excellent exemplary lessons integrating PhET
resources.
Members of the research group, responding to T2’s concerns, clarified that an
inquiry-based teaching approach not only facilitates software functionality but also enhances
student motivation:
The teacher does not need to give a concrete hypothesis and ask the students toverify this hypothesis. Instead, students can assume which is greater between apositive number and a negative number by themselves, and get some findings by
operating on PhET software. (R1, M2)
T2 mentioned in the interview that advice from researchers and other senior teachers
had been instrumental in moving him out of the limitations of traditional instructional design:
After the first trial, several teachers, TR1, and R1 suggested that the students shouldmake full use of the software’s features to explore mathematical rules. I redesigned
the inquiry process of students in more detail. For example, I will demonstrate tostudents how to use this software to explore, and students will learn the teacher's
manipulations to explore the rules of comparing the size of positive and negative
numbers and summarize the method of comparing the size. (T2, I1)
4.2.2 Example 3
Teachers, who had higher original expectations for its functionality, were not satisfied withthe features currently provided by the simulation resource, such as “the position of the input
number cannot be presented on the number axis”. Besides, T2, T3, and T4 expressed their
desire for the PhET software to add the function of comparing the magnitude of negative
fractions:Comparing the size of negative fractions is difficult for students. It would be better if
the fraction could be represented on this software. Then we can use image examples
to help students break through the difficult knowledge point of comparing the size ofnegative fractions. (T2, M3)
TR1 explained that this is due to the existence of diverse teaching orientations andrequirements among different countries, “We started from the Chinese math teaching
methods and textbooks, perhaps this software is mainly for game-based learning in othercountries”. R1 pointed out that “I will write down the problems we have and try to send an
email to the PhET operations team with our suggestions for program improvements.”
In T2’s interview, he reflected on the reciprocal relationship between technology andteaching:
I think that no matter what technology is used, it is ultimately for teaching, and thesetechnologies are designed to help us teach better. The problems we find in theteaching process can sometimes, in turn, drive the development of thesetechnologies. For example, if it can follow the teaching in the improvement, it may
become more and more perfect, making it even more suitable for most school
teachers to use. (T2, I1)

986

Contradiction: object and tools in research versus tools in teaching
Interpreting T2’s position, a contradiction could be recognized which is between the object(e.g., technology integration into classroom practice) and tools (e.g., PhET simulation tool) of
the research activity system and the tools in the teaching activity system (e.g., the content ofmathematics textbooks, traditional teaching approaches). The introduction of advanced
technology undoubtedly presented challenges for current teaching practices, as teachers
were unfamiliar with its implementation and efficacy (Qi et al., 2022). When teachers feel
uncomfortable or unprepared, their ability to instruct with simulations may be diminished(Stern et al., 2008). This contradiction reflects the division between new digital technologies
and traditional instructional design. It is not enough for researchers to simply makeresources available to teachers. What is more important is to teach teachers how to use
these in the classroom to maximize the effect of technology.
Boundary crossing was maintained in perspective making and taking
In dealing with the contradiction, R1 and TR1 act as brokers between research and teachingoffering ideas and explanations for teachers to overcome cognitive limitations. R1 introduced
the student-centered discovery approach, and TR1 gave teachers excellent examples oflessons that integrate PhET resources, including classroom videos and teaching design.
From initial confusion and skepticism about the use of simulation resources in theclassroom, the teacher rethought his teaching practices through an educational researchlens, incorporating a learner-centered inquiry-based teaching approach and adopting criticalperspectives towards technology's role. This indicates boundary crossing in terms ofperspective making and taking. Consistent with existing studies, technology could promoteteachers' pedagogical design capacity and help them design innovative instructional
practices to better support student investigations (Beyer & Davis, 2012; Davis et al., 2016).
Meanwhile, teachers’ beliefs regarding the role of technology in the classroom are pivotal in
shaping their instructional decisions or actions (Bates & Usiskin, 2016; Hermans et al., 2008;
Webb et al., 2015).

4.3 Generative interactions versus structured classroom culture
The contradiction between generative interactions and structured classroom culture
emerged in the subsequent stages of co-design activities (second post-lesson meeting andthird post-lesson meeting). The academic researchers in the team were more focused on
generative interactions between teachers and students from the perspective of learning
analysis. The emerging contradiction here concerns whether generative discourses were too
far from the teaching style and structured classroom culture cultivated over time. The
analysis of example 4 illustrates certain elements between teaching and research activity
systems involved in contradiction and the process of ameliorating it.
4.3.1 Example 4
R2 pointed out that T2 should capture students' generative expressions and guide them
toward deeper elaboration and expression:
The resources generated are abundant, but there is one place T2 needs to grab. A
student answered that the number is infinite, and T2 couldn't just sit her down. Itshould be a little more open in the dialogue between teacher and student, such as afurther question that allows students to elaborate and express their ideas in depth.
(R2, M3)
In contrast, T2 responded to the above-mentioned issues in the interview by arguingthat these generative conversations were not entirely applicable to teaching his class:
If they don’t know students in my class, their suggestions may not be too in line withthe actual situation. I just accept part of it. The students in Ms. Chen's class are very
open and outspoken when answering questions, while the students in my class are987

more timid and less outspoken. This is because my usual training requires students
to strictly answer questions according to my requirements and rarely think outside thebox. (T2, I1)
In the later discussion, T2 reflected that generative interactions have an importantrole in developing students' divergent thinking. He realized that the current didactic classculture should be changed, “We should make students dare to communicate and think moredivergently because students are the center of learning.” Teachers should fully anticipate"how students might react in class".
Contradiction: object and rules in research versus rules in teaching
In terms of the elements of the CHAT triangle, the goals (e.g., adequate student-teacher
generative interaction in the classroom) and rules (e.g., research norms) of the research
activity contradict the rules of the teaching activity (e.g., established classroom rules,
structured culture). As Reiser et al. (2000) noted, teachers and researchers involved in codesign always follow inconsistent workplace norms which bring challenges. Teachers' focus
was on the feasibility of implementation and the matching degree to student characteristics.
However, researchers tended to adopt an analytic perspective to focus on goals first. This
contradiction reflects the uniqueness of cultural norms in teaching and research, as well as
the boundaries between generative interactions and existing classroom cultures.
Boundary crossing was in the process of perspective making and taking
In dealing with this contradiction, R2 seems to play the role of boundary people providing
practical ideas and strategies (e.g., guiding questions) for T2 to change from structured
classroom management forms and teaching paradigms. From the interview data, T2
renewed insight into the importance of generative interactions for developing students'
divergent thinking. Also, he demonstrated his willingness to change his classroom culture to
make it more divergent and open to generative interactions. This displays the boundary
crossing in terms of perspective making and taking.

5. Discussion and conclusion
In our study, contradictions emerged in teaching and research activity systems throughsimulation-based math classroom co-design, and their boundary crossings were explicated.
The object, tools, and rules of the research activity appeared to contradict mainly theelements of tools and rules in the teaching activity. The object of the research activity is totransform the classroom with simulation resources to promote inquiry-based learning andunderstanding of mathematical relationships. The teaching reality and classroom culture
posed the researchers with a dilemma in terms of how to communicate clearly with teachers
about their implementation of the research-oriented teaching strategies and simulation
resources in daily teaching.
Contrary to the results of previous studies, boundary crossing mainly involves
constructing a new system that takes different perspectives into account rather than simply
questioning (Potari et al., 2019). This is partly due to the strong and rich research base of
the researcher group. Teachers trusted the expertise of their co-design colleagues, who
acted as brokers to bring plentiful resources (PhET simulation tool, educational theories,
principals, strategies, and even examples of course design). When teachers understood andformulated a shared meaningful object with the support of researchers, their developing
beliefs about instructional orientation and the value of technology can drive changes inteaching decisions and behaviors that lead to a smooth process of boundary crossing (Webbet al., 2015).
When contradictions emerged, the elements of activity systems (Engeström, 2001)
could be used to understand the sources of these contradictions and their boundaries. Fromthe perspective of boundary crossing, we depict the process of overcoming contradictions in
the interactions between teaching and research activity systems. Incorporating CHAT and988

boundary crossing, our analysis highlights the dynamic process of simulation-basedclassroom co-design by culturally diverse groups, in which contradictions inevitably arise.
However, the dissolution of contradictions contributes to the establishment of more
harmonious collaborative relationships that serve as a driving force for the co-designcommunity. This detailed analysis also depicts how teachers iteratively change theirpedagogical conceptions and actions to learn and develop. Based on these findings, we will
provide further support for teachers to optimize the learning opportunities offered by thecollaborative design and enable them to implement a simulation-based digital mathclassroom independently.

Acknowledgments
The authors wish to acknowledge the following for their contributions to this research:
Zuomin Zhang for help in organizing post-lesson meetings and for participating as amember; Hailong Shen for designing and teaching such simulation-based math course;
Liang Li for his feedback about this project and directions for future research; QianYu Zhang
for help in organizing the content of three meetings.

Funding
This study was funded by the Special Fund for Postgraduates of East China Normal
University to Participate in International Conferences and the sub-project "Empowering
Future Teachers with Artificial Intelligence: How to Reconstruct the Role of Teachers" whichis part of the National Social Science Fund Major Project "Research on Artificial Intelligence
Promoting Future Education Development" (Project No. 19ZDA364) for the year 2019.

References
Akkerman, S. F., & Bakker, A. (2011). Boundary crossing and boundary objects. Review of
Educational Research, 81(2), 132-169.
Bates, M., & Usiskin, Z. (2016). Digital curricula in school mathematics. IAP.
Beyer, C. J., & Davis, E. A. (2012). Learning to critique and adapt science curriculum materials:
Examining the development of preservice elementary teachers' pedagogical content
knowledge. Science Education, 96(1), 130-157.
Biggers, M., Forbes, C. T., & Zangori, L. (2013). Elementary teachers' curriculum design andpedagogical reasoning for supporting students' comparison and evaluation of evidence-based
explanations. The Elementary School Journal, 114(1), 48-72.
Buckley, B. C., Gobert, J. D., Kindfield, A. C., Horwitz, P., Tinker, R. F., Gerlits, B., Wilensky, U.,
Dede, C., & Willett, J. (2004). Model-based teaching and learning with BioLogica™: What dothey learn? How do they learn? How do we know? Journal of Science Education andTechnology, 13, 23-41.
Cai, S., Chiang, F.-K., Sun, Y., Lin, C., & Lee, J. J. (2017). Applications of augmented reality-based
natural interactive learning in magnetic field instruction. Interactive Learning Environments,
25(6), 778-791.
Davis, E. A., Janssen, F. J., & Van Driel, J. H. (2016). Teachers and science curriculum materials:
Where we are and where we need to go. Studies in science education, 52(2), 127-160.
Engeström, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualization.
Journal of education and work, 14(1), 133-156.
Engeström, Y., Engeström, R., & Kärkkäinen, M. (1995). Polycontextuality and boundary crossing inexpert cognition: Learning and problem solving in complex work activities. Learning andinstruction.
Goodyear, V. A., & Casey, A. (2015). Innovation with change: Developing a community of practice tohelp teachers move beyond the ‘honeymoon’of pedagogical renovation. Physical educationand sport pedagogy, 20(2), 186-203.
Hermans, R., Tondeur, J., Van Braak, J., & Valcke, M. (2008). The impact of primary school teachers’
educational beliefs on the classroom use of computers. Computers & Education, 51(4), 14991509.

989

Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020). The potential of digital
tools to enhance mathematics and science learning in secondary schools: A context-specificmeta-analysis. Computers & Education, 153, 103897.
Ko, M.-L. M., Hall, A., & Goldman, S. R. (2022). Making teacher and researcher learning visible:
Collaborative design as a context for professional growth. Cognition and Instruction, 40(1),27-54.
Moeller, K., Fischer, U., Nuerk, H.-C., & Cress, U. (2015). Computers in mathematics education–
Training the mental number line. Computers in Human Behavior, 48, 597-607.
Potari, D., Psycharis, G., Sakonidis, C., & Zachariades, T. (2019). Collaborative design of a reformoriented mathematics curriculum: Contradictions and boundaries across teaching, research,and policy. Educational Studies in Mathematics, 102, 417-434.
Qi, C., Liu, X., Wang, R., Zhang, J., Fu, Y., & Huang, Q. (2022). Contradiction and its solutions in themathematics teacher–researcher partnership: an activity theory perspective. ZDM–
Mathematics Education, 54(3), 639-652.
Reiser, B. J., Spillane, J. P., Steinmuller, F., Sorsa, D., Carney, K., & Kyza, E. (2000). Investigating
the mutual adaptation process in teachers’ design of technology-infused curricula. Fourthinternational conference of the learning sciences,
Remillard, J. T., & Heck, D. J. (2014). Conceptualizing the curriculum enactment process inmathematics education. Zdm, 46, 705-718.
Roschelle, J., Penuel, W., & Shechtman, N. (2006). Co-design of innovations with teachers: Definitionand dynamics.
Ross, J., & Bruce, C. (2012). Quantitative inquiry into collaborative action research: Measuring
teacher benefits. Teacher development, 16(4), 537-561.
Rutten, N., van der Veen, J. T., & van Joolingen, W. R. (2015). Inquiry-based whole-class teachingwith computer simulations in physics. International Journal of Science Education, 37(8), 12251245.
Rutten, N., Van Joolingen, W. R., & Van Der Veen, J. T. (2012). The learning effects of computer
simulations in science education. Computers & Education, 58(1), 136-153.
Shrader, G., Williams, K., Lachance-Whitcomb, J., Finn, L.-E., & Gomez, L. (2001). Participatory
design of science curricula: The case for research for practice. Annual Meeting of theAmerican Educational Research Association, Seattle, WA,
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology,translations' and boundary objects:
Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social
studies of science, 19(3), 387-420.
Stern, L., Barnea, N., & Shauli, S. (2008). The effect of a computerized simulation on middle school
students’ understanding of the kinetic molecular theory. Journal of Science Education andTechnology, 17, 305-315.
Webb, A. W., Bunch, J., & Wallace, M. F. (2015). Agriscience teachers’ implementation of digital
game-based learning in an introductory animal science course. Journal of Science Education
and Technology, 24, 888-897.
Yan, Y., & Yang, L. (2019). Exploring contradictions in an EFL teacher professional learning
community. Journal of Teacher Education, 70(5), 498-511.

990

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

The Impact of Development-questioning
Activities on Students' Pre-writing Ideas
Jia Ling HONG a*, Chang-Yen LIAO b, Tak-Wai Chan a & Jui-Fen Chang a
Graduate Institute of Network Learning Technology, National Central University, TaiwanbDepartment of Hakka Language and Social Sciences, National Central University, Taiwan*110524025@cc.ncu.edu.twaAbstract: Studies suggest that questioning is a potent cognitive tool that can guidestudents toward a more profound engagement with their learning and comprehension,
mainly driven by their innate curiosity. In conventional teaching methods, instructors
often assign writing topics directly, causing a lack of enthusiasm and hindering
students' ability to generate original ideas. This circumstance can gradually breed anaversion to writing. In response to this challenge, our research aims to introduce aquestion-oriented activity that provides students with a participatory platform for idea
generation and organization. By engaging in group collaboration and brainstorming,
questioning elements are presented to stimulate students' curiosity. Participation in thisactivity allows students to compile a broad range of ideas for future writing tasks. The
intended outcome is to aid students in developing coherent ideas, bolster theirmotivation for writing, and refine their skills in questioning.
Keywords: Questioning, Round-robin brainstorming, Pre-writing, Group collaboration

1. Introduction
Hidi's (2006) research emphasizes that curiosity and interest profoundly impact learning
motivation. Clark et al. (2019) similarly discovered that student questioning can positively
stimulate their curiosity. However, in many Taiwanese classrooms, students avoid asking
questions, resulting in an unnerving silence. The reasons behind students' avoidance ofquestioning could range from fear of inadequate knowledge, potential criticism, or simply alack of interest in the topic. As prior research has illustrated, questioning skills are vital tocontinually promote student thinking, facilitate valuable feedback, and enhance topic
comprehension, enabling students to uncover more profound knowledge (Wardani, 2014).
Writing is divided into three stages: pre-writing, writing, and post-writing. The pre-writing phaseholds significant importance for students as it not only prompts them to explore the topic but
also assists them in envisioning how to apply it in their subsequent writing, thus aiding inconstructing their content (Hagtvedt et al., 2019). However, in the current education scenario,
with an emphasis on the outcomes and quality of writing, the importance of the writing process
is often overlooked. Hence, urging students to contemplate the topic before writing caneffectively influence their subsequent writing development (Hashempour, Rostampour &Behjat, 2015), sparking curiosity to pose questions and prompting more profound thought
about the subject matter.
Given the above, this study centers on pre-writing activities and explores the impact ofstudents' questioning abilities and idea generation before writing. Cultivating a questioning
habit can stimulate students' curiosity and enhance their questioning skills, encouraging active
writing participation and gradually fostering students' interest in writing (Hidi & Renninger,2020). With the research question in mind, this study will endeavor to design a Question
Generation & Combination (QGC) activity for implementation before writing, offering students
an environment to gather and organize their ideas. This activity combines students' curiositywith round-robin brainstorming (Etemadzadeh, Seifi & Far, 2013), guiding students to delve

991

into and reflect on the topic by continually sparking ideas based on their teammates' thoughts.
While curiosity diminishes over time, interest can sustain ongoing exploration (Grossnickle,
2016). Thus, facilitating the establishment of sufficient background knowledge through theactivity can smoothen composition in subsequent writing.

2. Literature Review
2.1 Writing Ideas and Challenges
Writing is a form of self-expression, where individuals organize their knowledge to share withreaders and develop skills in collecting information. Previous research has divided the writing
process into three cognitive stages: planning, translating, and revising (Berninger & Swanson,1994). It involves planning and generating ideas, organizing thoughts, and transforming theminto written expression (Baaijen, Galbraith & Glopper, 2014), followed by evaluating andrevising the text. The process of writing instruction typically includes pre-writing, drafting,revising, and sharing (Keen, 2017), with pre-writing being a crucial and often overlooked stage
in the writing process. Engaging in pre-writing activities can assist students in planning ororganizing their ideas, effectively influencing the development of subsequent writing(Hashempour, Rostampour & Behjat, 2015), and fostering motivation for writing (Mogahed,2013).
In previous writing experiences, teachers typically adopted a product-oriented
approach, focusing on students' writing skills and performance. Students' works were
evaluated and commented on based on their content, and students might or might notincorporate the suggestions provided by the teacher. This traditional approach of using
prompt-based essays often presented challenges for students, particularly in terms of knowing
how to begin their writing. This could be due to students feeling unable to generate ideas,
lacking interest in the topic, or not having relevant background experiences associated withthe topic. Consequently, students found it difficult to express their creativity within the given
time constraints. Over time, due to a lack of ideas and limited curiosity in writing, coupled withinsufficient background knowledge, students developed an increasing aversion towards
writing.

2.2 Brainstorming
The biggest challenge at the beginning of writing is the lack of ideas, and brainstorming is aneffective way to generate new ideas (Hashempour, Rostampour & Behjat, 2015). Scholar
Osborn (2012) mentioned that brainstorming enhances creative output, involving four
conditions: no criticism of ideas, building on others' suggestions, embracing any
unconventional ideas, and generating a large quantity of ideas. By creating a stress-freeenvironment through brainstorming, students can improve their writing skills (Hashempour,
Rostampour & Behjat, 2015). Stimulating students' thinking through brainstorming helps them
explore ideas related to writing activities and how they can integrate or generate ideas basedon their prior background knowledge. Prior to conducting brainstorming, students do not needto make any preparations, and they can freely record their ideas during the process. The
purpose is to guide students in utilizing this new thinking approach to generate a multitude ofideas.
In this study, the round-robin brainstorming approach is adopted as one of the methodsof brainstorming. According to scholar Roestiyah (2008), it aims to generate ideas within arelatively short period, accumulate ideas, and engage in discussions without being influenced
by team members, ensuring that each individual has an equal opportunity to share ideas.
Round-robin brainstorming is a method of collaborative idea generation (Mogahed, 2013) that
also enhances students' autonomy and sense of responsibility, thereby increasing theirmotivation (Fawzi & Hussein, 2013).
In the past and up to the present, many students have encountered a lack of inspirationand curiosity towards writing, which limits their imagination. Therefore, this study aims to992

stimulate students' curiosity through multimedia materials and generate motivation byprompting them with interesting questions. The round-robin brainstorming method is thenemployed, giving each student an opportunity to contribute ideas and unleash their creative
thinking. By drawing inspiration from the ideas of their team members, who bring different
perspectives, the students can explore new directions of thought and collect ideas generated
collaboratively. These ideas will serve as the basis for the subsequent writing process in termsof direction and content.

3. Model Process Design
This study focuses on the generation and organization of students' ideas, based on the core
concept of the "Interest Creator Theory" (Chan et al., 2018). A "QGC" (Question Generation& Combination) activity is designed and a user-friendly platform is developed for students touse prior to writing. The study adopts a "design research method" with the aim of designing
the "QGC" model based on theoretical directions and improving students' writing skills. Afterestablishing the research direction and designing the curriculum, the study is implemented ina primary school teaching environment. The research process, including students' writing
progress, is documented and discussed with the research team. After each class, theresearchers and the teaching staff discuss the students' learning outcomes and makeadjustments to the activities based on the analysis and feedback. The investigation spans one
semester and explores topics in social and natural sciences.
The "QGC" (Question Generation & Combination) activity consists of two main
components: idea generation and idea organization. It includes four stages: generating ideas,
categorizing ideas, integrating ideas, and transforming ideas. By employing the principle ofcollecting a large number of ideas through posing questions, the aim is to organize scatteredideas, make them more concrete and explicit, and enhance the understanding of the coreideas. After the idea generation stage, group discussions are conducted to categorize andintegrate ideas, assisting students in organizing their thoughts. Prior to engaging in the QGC
activity, related materials such as multimedia resources can be used. These materials provide
inspiration based on different themes, integrating elements like text and images, andconveying information interactively to facilitate student engagement. Through this approach,students can establish preliminary background knowledge and stimulate their curiosity.
In the first stage, idea generation, it is essential to have a plethora of ideas for thesubsequent process of organizing ideas. The round-robin brainstorming technique isemployed, providing an environment where students take turns writing down their ideas.
Through group collaboration, students engage in associative thinking and generate moreideas. The objective is to allow students to generate additional ideas by leveraging the inputof others. Additionally, the system platform will offer features for preserving and reviewing
ideas, preparing for subsequent collaborative discussions. The following three stages focus
on further organizing the generated ideas, namely categorizing, integrating, and transforming
ideas. These ideas can stem from the knowledge and experiences of group members.
Through group discussions, students organize their perspectives and understandings of the
problem, enabling them to consider the problem from different angles. During thecategorization stage, all ideas within the group are classified and organized through group
discussions. Students can categorize ideas based on their types, sources, and observed
characteristics. The system platform assists students in swiftly organizing ideas by dragging
and dropping ideas into relevant categories, establishing a conceptual framework of thecollected ideas within the group. As students progress through the stages and gain a bettergrasp of the ideas, they reach the integration stage. Here, they can consolidate seemingly
similar ideas, further organizing ideas within each category to reduce redundancy and clutter.
The system platform allows students to view the before-and-after merging of ideas, enablingthem to freely adjust and create more precise ideas. The final stage, transformation, involves
students converting their ideas into open-ended questions. The purpose is to employ
questioning techniques to help students gather more information and continuously stimulate
their curiosity. Through the collaborative process within the group, ideas can be expanded,leading to the collection of a broader range of ideas. The platform provides transformation

993

examples and assists in identifying ideas that can be transformed, fostering imagination andcreativity during the discovery of new ideas, and developing students' cognitive flexibility.
Through this activity, students may begin to conceptualize the structure of their own
writing in their minds and contemplate how to apply the materials to their articles. Despitecollecting the same set of questions within the same group, each student's presentation differs.
As students organize these idea materials into a coherent essay, they actively structure theirthoughts and employ questioning techniques. This encourages students to think more
attentively and further promotes their active engagement in the activity.

4. Preliminary assessment
4.1 Research participants
The research participants of this study were fourth-grade and five-grade students from anexperimental school in Taoyuan, as well as teachers and researchers. The researchers
provide the relevant theoretical basis, and discuss with the teachers the design of the course
theme textbooks, the selection of multimedia materials, and the learning status of the students.
The selected students are expected to use QGC activities to promote the process ofgenerating their ideas, and through data analysis and physical classroom observation, further
adjust the planning of activities suitable for students.

4.2 Data collection and research tools
This study will use quantitative and qualitative analysis, conduct two open questionnaires
before and after the activity, including questioning ability questionnaire and writing ability
questionnaire, for evaluation and analysis. At the same time, the idea database stored by
students in the activity is viewed Source for gathering ideas. In addition, after the event, we
will conduct interviews with the students to gain a deeper understanding.
Questionnaire: A test used to assess the quality of students' ability to ask questions
about the topic before and after the activity. This questionnaire focuses on assessing students'
questioning ideas and the number of ideas, and evaluates the quality indicators of ideas basedon three aspects, Including topical relevance, fluency and extension.
Writing Ability Questionnaire: It is used to assess students' ability to write descriptionsof topic-related pictures within a limited time. This questionnaire will assess the number ofwords written by students and use punctuation as a unit of calculation to measure the numberof ideas in writing.

4.3 Preliminary Results and Discussion
The subject of this study, which carries significant malleability, is the energy component of
natural science. We segregate the class into three groups to conduct Question Generation &Combination (QGC) activities. This study tentatively scrutinizes students' questioning ability
and its bearing on writing through QGC activities. To decode and analyze the ideas
engendered by students, we hark back to the writing creativity indicators and standards
employed in earlier studies and define scoring standards for the questions. These include
theme relevance, fluency, and originality as three guiding indicators to procure the question
quality scores of each group and the entire class for statistical scrutiny, as demonstrated inTable 1.

994

Table! 1. Paired sample T-test results
SD

t

df

ρ

Question _Average Score

0.64

4.33

30

0.002

The total amount of question ideas

3.74

3.79

30

0.004

Write word count

5.21

0.40

30

0.701

The total amount of Write ideas

5.22

1.88

30

0.093

The principal aim of this study is to probe whether the questioning activities will alter
the students' writing and questioning capabilities. By the analytical outcomes of the pre-and
post-activity tests, the quality of questions, and the abundance of ideas, the average word
count and the number of ideas in the post-test outweigh those of the pre-test. The study
identified a significant divergence between the average quality of questions and theabundance of ideas, signifying that students bolstered their questioning skills through the QGC
activity(p<0.05). Even though the influence on writing did not reach statistical significance, theactivity facilitated students in reflecting upon the writing process, on average, as indicated by
the data.
Most students deem writing their poorest skill, and the shortage of ideas is viewed as
one of the vital elements affecting creativity. Furthermore, students are customarily in a stateof passive participation in the classroom. Therefore, stimulating thought and soliciting
questions in writing activities is paramount. Future research will delve more deeply into thevarious stages of the movement to fathom how students amplify and organize their ideas
during the activity and the linkage between idea transformations and writing. The preliminary
research outcomes are elucidated based solely on the results above.

Acknowledgements
This study was funded by the National Science and Technology Council of Taiwan (NSTC112-2423-H-008-003) and by Research Center for Science and Technology for Learning,
National Central University, Taiwan.

References
Baaijen, V. M., Galbraith, D., & de Glopper, K. (2014). Effects of writing beliefs andplanning on writing performance. Learning and Instruction, 33, 81-91.
Berninger, V. W., & Swanson, H. L. (1994). Children’s writing; toward a process theoryof the development of skilled writing. Children’s writing: Toward a process theoryof development of skilled writing, 57-81.
Chan, T. W., Looi, C. K., Chen, W., Wong, L. H., Chang, B., Liao, C. C., ... & Ogata,H. (2018). Interest-driven creator theory: Towards a theory of learning design for
Asia in the twenty-first century. Journal of Computers in Education, 5(4), 435-461.
Clark, S., Harbaugh, A. G., & Seider, S. (2019). Fostering adolescent curiosity througha question brainstorming intervention. Journal of adolescence, 75, 98-112.
Etemadzadeh, A., Seifi, S., & Far, H. R. (2013). The role of questioning technique indeveloping thinking skills: The ongoing effect on writing skill. Procedia-Social and
Behavioral Sciences, 70, 1024-1031.
Fawzi, M., &Hussein, A.A. (2013). Enhancing students’ motivation to write essays
through brainstorming: A comparative study. International Journal of Humanities
and Social Science, 3(9), 191-196.
Grossnickle, E. M. (2016). Disentangling curiosity: Dimensionality, definitions, anddistinctions from interest in educational contexts. Educational Psychology
Review, 28(1), 23-60.
995

Hagtvedt, L. P., Dossinger, K., Harrison, S. H., & Huang, L. (2019). Curiosity made thecat more creative: Specific curiosity as a driver of creativity. Organizational
Behavior and Human Decision Processes, 150, 1-13.
Hashempour, Z., Rostampour, M., & Behjat, F. (2015). The effect of brainstorming asa pre-writing strategy on EFL advanced learners’ writing ability. Journal of Applied
Linguistics and Language Research, 2(1), 86-99.
Hidi, S., & Boscolo, P. (Eds.). (2006). Writing and motivation. Brill.
Hidi, S. E., & Renninger, K. A. (2020). On educating, curiosity, and interest
development. Current Opinion in Behavioral Sciences, 35, 99-103.
Keen, J. (2017). Teaching the writing process. Changing English, 24(4), 372-385.
Mogahed, M. M. (2013). Planning out pre-writing activities. International Journal of
English and Literature, 4(3), 60-68.
Osborn, A. (2012). Applied imagination-principles and procedures of creative writing.
Read Books Ltd.
Wardani, I. (2014). Improving the ability in writing Descriptive Text through GuidedQuestions technique. E-Journal of ELTS (English Language TeachingSociety), 2(1).

996

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Promoting Teachers’ Digital Literacy
Achievement: A Nationwide Survey of
Education Informatization in China
abcd

Ziyan CHEa, Jiumei YANGb , Longkai WUc* & Di WUD
Faculty of Artificial Intelligence in Education, Central China Normal University, China
Abstract: Education informatization is one of the paramount issues in educational
reform. To resonate with Chinese national progress in technology development andnational policy for promoting education reform and modernization, this study ascertains
the current status of education informatization and teachers’ use of technology inteaching and learning in China. Data from a national survey on primary and secondary
schools and teachers about the performance of education informatization was
analyzed. According to the results, teachers’ use of technology in teaching is areinforcement of traditional pedagogy; innovative functions such as the communicationand evaluation of teaching and learning are under-utilized. On the basis of this finding,promoting education informatization needs to promote teachers’ coherent Digital
Literacy (DL) integration competence. Meanwhile, a professional gap in using digital
educational resources was found among teachers. The Ministry of Education, Science& Technology (MEST) and the research team have put sustained efforts to guide thedevelopment of education informatization in China, and the results of this study serve
as evidence for policymakers and principles to promote teacher professional
development in line with the changes of new requirements for teachers’ teaching ability
in this information era.
Keywords: Teacher Professional Development, education informatization

1. Introduction
Education has evolved alongside global informatization. In 2019, drawing on a range ofeducational reforms China has undergone since 1978, the Chinese State Council devised
and instituted China’s Education Modernization 2035 Plan (2035 Plan) and the
Implementation Plan for Accelerating Education Modernization (2018-2022). These two
documents aimed at substantially modernizing China’s education system by 2035. Over the
past three decades, the central government strengthened national network coverage,
connecting all primary and secondary schools by 2020, with half of them having wirelessnetworks (Dong et al., 2021; Zeng, 2022). Recent focus gradually shifted from device
construction to innovating Digital Educational Resources (DERs). To resonate with Chinese
technology development and national policy for promoting education reform andmodernization, the Ministry of Education has carried out training programmes for teachersand principals in Information and Communications Technology (ICT) in the pedagogicalapplication during the past years (Zeng, 2022).
Even though the nationwide infrastructure has been built, education informatization
will be accomplished in actually only when teachers and principals are competent enough tosystematically utilize technologies to leverage career-related activities (Kim & Lee, 2011). In
pursuit of advancing education informatization, a national survey was conducted by MEST in
2020, focusing on primary and secondary schools. This study presents national survey data
concerning teachers’ technology integration in teaching and learning (T&L) and the present
state of education informatization in schools. Additionally, this study aims to identify potential
areas for improvement and strategic focus to promote education informatization in China
based on the current landscape.

997

2. Theoretical BackgroundThe necessity of education informatization emerges as technology develops. The
development trajectory of national policy for education informatization usually follows thesequence from simply using technology and modifying technology to serve the educationneeds to integrating technology for innovative education practices (Zeng, 2022). Education
informatization has had a significant impact on improving classroom teaching. Except for
upgrading infrastructure, teachers’ DL is an important step to leverage education
informatization (Howard & Tondeur, 2023) for maximum potential.
Empirically, DL encompasses competencies like computer literacy, ICT literacy,
information literacy, and media literacy, allowing individuals to access, manage, evaluate,
and create information through digital technologies (Law, Woo, de la Torre, & Wong, 2018).
Prior research emphasized the importance of effective ICT support in universities forpreservice teachers (Wang & Zhao, 2021). Although society and industry introduce newtechnology to schools, its actual integration is usually reported as a reinforcement oftraditional approaches (TAN & PARK, 2016) rather than truly transformative education. Only
when the integration and utilization of technology are led by education itself, the education
informatization is truly meaningful. True education informatization requires teachers'
coherent DL integration competence, fostering productive activities and the creation of
pedagogical knowledge (Howard et al., 2023; Kim et al., 2011).
Past research focused on theoretical aspects of China's educational informatization.
This study presents up-to-date data and analysis on the current state of China's educational
informatization, offering insights to government and higher education decision-makers,
aiding policy formulation. This study also adds to the previous research by exploring themediation and moderation role of teachers’ professional background and DERs habits.
Three research questions (RQs) underpin this research: RQ1: What is the current DL
achievement of teachers? RQ2: What is the current education informatization level ofschools? RQ3: What needs to be focused on to advance the DL achievement of teachers?

3. Study Method
Our study was based on the national survey data conducted in 2020 by Chinese MEST aspart of a territory-wide evaluation of implementing the Education Informatization Promotion
Plan. Considering the large population and enormous education scale in China, for theconvenience of analysis, this study selected data from three representative provinces fromthe eastern (Province A), central (Province B), and western (Province C) regions of China.

3.1 Index Description and Data Analysis
The education informatization index in this study, drawing from prior research (Kim et al.,2011), encompasses three dimensions: accessibility, utilization, and usability. Accessibility
refers to the accessibility level of infrastructure (hardware, software, internet connection,
multi-media classroom, etc.) established for T&L in schools. Utilization is the extent to which
education informatization is employed within schools. Usability is teachers’ overall use oftechnologies, including the use of DERs and software, and the coherent DL integrationcompetence in different T&L sectors, which refers to overall DL achievement in this study.
Diversity of ICT curriculums provided by schools, frequency of the teacher
professional development on DL competency, and years of teaching are adopted to explore
further possible elements that could advance the DL achievement of teachers.
Data analysis utilized SPSS and Microsoft Excel. Descriptive statistics summarized
the questionnaire's primary outcome. Educational background differences were assessedwith the Chi-square test. The moderation and mediation effect models were applied touncover potential moderating and mediating factors affecting teachers' DL achievement.

998

4. Result
4.1 Descriptive Statistics
A sample size of 131347 in-service primary and secondary teachers completed the online
questionnaire, with 103739 (79%) teachers graduating from a teacher education programme.
A total of 13352 schools from 57 cities completed the online questionnaire.
4.1.1 Teachers’ DL Performance (RQ1)

Figure 1. Types of DERs Used in Different T&L Activity
Figure 1. indicates most teachers tend to use traditional DERs like Slides in in-class teachingand preparation stage, and use DERs like assessment tool for assessments. The use of
DERs in teaching research and experiment are rare.

Figure 2. Types of Software Used in Different T&L Activity
Figure 2 illustrates the prevalent usage of multimedia productivity tools, professional
teaching software, and assessment tools among teachers. The adoption of open educational
resources platforms, teaching and management platforms, and online teaching platforms islimited.

Figure 3. DL Achievement in Different T&L Activity
Figure 3. indicates an overall DL achievement of teachers that teachers actively utilize
information technology in T&L activities, including learning analysis, supportive activities,
teaching, teaching optimization, online learning, designing DERs, and teaching sharing
sessions. The utilization of technology in other activities is relatively low.
4.1.2 Schools’ Education Informatization (RQ2)999

Figure 4. Schools’ Utilization Level of Education Informatization
Figure 4. indicates that at the school level, the utilization of education informatization mainly
appears in preparation, teaching, resource sharing, and professional development. Teacherto-parents and teacher-to-student communication through technology are rare. And theutilization of education informatization in learning analysis and teaching evaluation is low.

4.2 Possible Influence Factors on Teachers’ DL Achievement (RQ3)The Chi-square test results (Table 1.) indicated a trend toward significance for the effect ofeducational background on the frequency of using DERs among in-service teachers (p <0.001). Teachers who graduated from the teacher education programme (TEP) had a higher
percentage of using DERs more frequently, while teachers who graduated from the nonteacher education programme (NTEP) tended to use DERs less frequently.
Table 1. Group Differences in the Frequency of Using DERs (***p<0.001)
Background Never Rarel Some Often Alway Total
Chiy
times
s
Square
test
Prepa
TEP
2322 5974
10725 34072 50631 103724 192.738
ration
***
NTEP
556
1298
2535
8511
14708 27068
InTEP
1527 5235
10092 30484 56401 103739 86.735*
class
**
NTEP
345
1213
2451
7788
15811 27608
Experi
TEP
46018 33432 13006 4693
1908
1043
167.808
ment
***
NTEP
11156 9176
3836
1480
578
305
Exami
TEP
60907 27556 7975
2482
981
546
127.191
nation***NTEP15261 7688
2403
797
311
170
Innov
TEP
30821 30043 17691 8829
4934
3249
38.533*
ation
**
NTEP
7822 7874
4872
2386
1359
926
4.2.1 Mediation Effects on Teachers’ DL Achievements
Table 2 shows the mediating effects of DER diversity on the relationship between software
diversity and DL achievements. The direct effects of software diversity on both DER diversity
and DL achievements were strong and significant (Coeff. = .590, p 烋 .001; Coeff. = .396, p
烋 .001). Similarly, the direct effect of DER diversity on DL achievements was also significant(Coeff. = .209, p 烋 .001). The mediation condition (Figure 5.) was met as the indirect effectof software diversity on DL achievements through DER diversity was significant.

Figure 5. Hypothesis Model for Software Used, DERs Used, and DL Achievement
Table 2. Mediation Effect on DL Achievement (via Diversity) (*** p < 0.001)

1000

Diversity _ Software
Diversity _ DERs

Diversity _ DERs
0.590***(235.914)

DL Achievement
0.396***(119.165)
0.209***(57.628)

4.2.2 Moderating Effect on the Diversity of the Used DERs
Table 3 displays the interaction (Figure 6.) between DL achievement and years ofteaching in predicting DER diversity. The interaction effect was significant (p < .05). Simple
slope test results indicated a significant positive relationship between DL achievement andDER diversity when years of teaching were low (CI {-0.0031, -0.0016}, t = -6.1289, p < 0.05).
Years of teaching significantly predicted DER diversity when DL achievement was the same.
Less experienced teachers showed higher diversity in using DERs for teaching.
Furthermore, the above mediating and moderating analyses suggest a reciprocal
relationship between teachers' DL achievement and the diversity of used DERs. It proposesthat higher diversity of used DERs is indicative of enhancing teachers’ DL achievement, andhigher DL achievement leads to increased diversity of used DERs.

Figure 6. Hypothesis Model for DL Achievement, Years of Teaching and DERs Used
Table 3. Moderation Effect of Years of Teachingtp
LLCI
Constant
192.6895
08.9412
DL achievement
99.488900.8259Years of teaching-5.29690-0.0142Int_1-6.12890-0.0031ULCI
9.125
0.8591-0.0065-0.0016

4.2.3 Mediation Effects on Teacher Professional Development
Table 4 shows the mediated effects between schools' utilization level of education
informatization and teacher professional development frequency through accessibility level.
The utilization level had a strong and significant direct effect on accessibility (Coeff. = 4.311,
p 烋 .001) and on teacher professional development frequency (Coeff. = 6.513, p 烋 .001).
Similarly, accessibility level had a strong and significant direct effect on teacher professional
development frequency (Coeff. = 1.269, p 烋 .001). This meets the mediation condition(Figure 7.) where the indirect effect of schools’ utilization level of education informatizationon teacher professional development frequency through accessibility level was significant.

Figure 7. Hypothesis Model for Software Used, DERs Used, and DL Achievement
Table 4. Mediation Effect on DL Achievement (via Diversity) (*** p < 0.001)
Accessibility level
Teacher professional
development
Utilization level4.311***(15.964)
6.513***(3.823)
Accessibility level1.269***(23.728)

1001

5. Discussion
For RQ1, the descriptive statistics reveal that the most frequently used DERs and software
were quite traditional, aligned with previous study (TAN & PARK, 2016). More innovative
resources like a micro lecture and teaching and management platform are under-utilized.
Regarding RQ2, the utilization of education informatization at the school level was
mainly focused on teaching and teacher professional development. The communication and
T&A evaluation functions need to be more utilized.
Regarding RQ3, younger teachers tended to use more DERs, and more diversity of
DERs used led to higher DL achievement in different T&L activities, which aligned with theconclusions revealed in previous studies (Wang & Zhao, 2021). This may be due to younger
teachers perceived the power of technology in their teacher preparation programs, whileolder teachers usually received traditional programs. Also, teachers who graduated from
TEP showed a higher frequency of using DERs. At school levels, with the same utilizationlevel of education informatization, schools with higher accessibility levels tended to provide
more teacher professional development.

6. Conclusion
This paper reports on the current situation of education informatization in China andcontributes to revealing important insights for administering education informatization
promotion. From a theoretical aspect, observations can be made that teacher professional
development is the significant factor that would cause an increase in the use of DERs,
leading to higher DL achievements. From the aspect of ICT policy-making, these findings
may serve as receptive suggestions for promoting teachers’ DL achievement. The
accessibility level of informatization provides chances for teacher professional development,
but at the same time, policymakers need to realize that except for infrastructure building, on
top of recognition for education informatization, resources should also be allocated forteacher capacity building. After all, the return on infrastructure investment depends onteachers who implement the curriculum.
MEST and the research team have put sustained efforts to guide the education
informatization in China and better promote the 2035 Plan. The results of this study serve as
evidence for policymakers to focus and reallocate resources for teacher professional
development to line with the changes of new requirements for teachers’ teaching ability inthis information era.

References
Dong, Y. Q., Bi, J. G., Qian, S. L., Bian, J. S., & Qiao, P. X. (2021). Survey on ICT application
development in basic education and strategic adjustment from an international comparison
perspective [in Chinese]. Open Education Research, 04, 50-58.
Howard, S. K., & Tondeur, J. (2023). Higher education teachers' digital competencies for a blended
future. Educational technology research and development, 71(1), 1-6.
Kim, J., & Lee, W. (2011). An analysis of educational informatization level of students, teachers, andparents: In Korea. Computers & Education, 56(3), 760-768.
Law, N., Woo, D., de la Torre, J., & Wong, K. (2018). A global framework of reference on digital
literacy skills for indicator 4.4. 2.
TAN, M. M., & PARK, J. (2016). Diverse approaches to developing and implementing competencybased ICT training for teachers: A case study.
Wang, Q., & Zhao, G. (2021). ICT selfƮefficacy mediates most effects of university ICT support onpreservice teachers' TPACK: Evidence from three normal universities in China. British Journal of
Educational Technology, 52(6), 2319-2339.
Zeng, W. (2022). An empirical research on China’s policy for ICT integration in Basic Education from1988 to 2021. Educational technology research and development, 70(3), 1059-1082.

1002

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Learning with Conversational AI and
Personas: A Systematic Literature
Review
Antun DROBNJAKa, Ivica BOTICKIa*, Peter SEOWb, Ken KAHNc
Faculty of Electrical Engineering and Computing, University of Zagreb, CroatiabNational Institute of Education, Nanyang Technological University, SingaporecUniversity of Oxford, United Kingdom*ivica.boticki@fer.hraAbstract: This paper describes the results of a systematic review dealing withthe use of personas, avatars, and characters in conjunction with AI-supported
tools such as chatbots or generative AI in education. Although the use generative
AI in education is gaining traction, this study seeks to systematically review the
body of knowledge dealing with personified and conversational approaches to
education with both pre-generative and generative AI. The results of the study
emphasize the importance of the three key elements of such systems: the use of
pedagogical agents, interaction, and personalization. These key elements can berelevant when considering the adoption of the new generation of generative AI in
education. Such systems should scaffold learners’ understanding providing
guidance and support, promote self-directedness and ensure effectiveness inlearning, provide customized learning paths, and promote ethical use.
Keywords: Conversational AI, Generative AI, learning, education, pedagogicalagents, personas

1.

Introduction

This paper deals with the domain of AI use in education to achieve greater interaction
and hence a more immersive educational environment for the students. With the rise of
AI in all facets of society, researchers and practitioners have been keen on using it as atool to supplement pedagogies, which could possibly lead to improved educational
outcomes (Huang et al., 2021). The development of AI has had a profound impact on
the field of pedagogy by providing new opportunities and opening new challenges.
Traditional educational approaches often rely on human teachers delivering instruction,
but AI opens up possibilities for automated teaching and content delivery.
With the appearance of generative AI, the potential of AI in education is evenmore obvious and has gained traction across the globe. In terms of interaction,
generative AI has led to the creation of various interactive tools and systems.
Additionally, large language models (LLMs), such as the GPT-based models, enable
learners to interact with AI systems through natural language, facilitating more nuanced
and contextualized conversations.
With generative AI there is a potential to contribute to the improved interaction in
the educational process potentially leading to educational benefits (Xu et al., 2022), butpedagogical approaches to achieve so are still work in progress. The challenges
become increasingly evident when we delve into the technological peculiarities ofgenerative AI. In this domain, one must closely examine these tools within the context
of the machine learning language models they produce, which inherently operate on aprobabilistic basis (Lim et al., 2023).

1003

The study is a systematic review study conducted according to the PRISMA
methodology yielding a specific set of relevant research papers in the field. The study
results indicate that the most relevant studies using virtual agents, avatars, personas,
chatbots and other AI-empowered tools in education emphasize pedagogical agents,
interaction and personalization as key elements contributing the improved educational
results and processes. These are further discussed in the paper.

2.

Background

Pedagogy for AI in education refers to the principles and practices that should guide thedevelopment and implementation of AI technologies in educational settings. The goal ofpedagogy for AI in education is to ensure that these technologies support effective
teaching and learning practices and align with the educational objectives (Huang et al.,2021). AI technologies have the potential to revolutionize education, providing students
with customized learning experiences and educators with valuable insights into learning
processes and outcomes.
According to Järvelä (2006), personalized learning has the potential to enhance
student interest and motivation, particularly when students are provided with
opportunities to stimulate their creativity and curiosity. Personalized learning can takemany different forms, from adaptive learning systems that adjust the difficulty of content
based on learner performance to project-based learning that allows learners to explore
topics that interest them. For example, according to the findings of Brown et al. (2006),
university students expressed a preference for utilizing a personalized virtual learning
environment (VLE) for exam revision as compared to non-personalized methods.
Ouyang & Jiao (2021) proposed three paradigms - intelligent tutoring systems,
learning analytics, and educational data mining, that can support personalized learning
experiences, improve educational outcomes, and inform decision-making in educationalsettings. These paradigms align with the goals of pedagogy for AI in education, whichprioritize the ethical and responsible use of AI technologies while also promoting
effective teaching and learning practices.
In their paper Van Brummelen et al. (2021) presented four design
recommendations for Human Computer Interaction (HCI), including considering
personification, transparency, playfulness, and utility that could improve design of AIbased systems. When designing AI-based learning systems, it is crucial to consider
how users interact with the system, what information they need to provide, and how thesystem provides feedback and guidance. Xu et al. (2022) proposed scaffolding as asolution to improve feedback and guidance. In their paper Crearie (2013) emphasizes
the need for HCI to be integrated into the development of such systems and notes that
AI-based learning systems have the potential to revolutionize education by providing
personalized learning experiences. Main challenge in designing HCI for AI-basedlearning systems is to balance the need for personalization and standardization.

3.

Methodology

The PRISMA method is a systematic approach to conducting reviews that involves
three main stages: identification, screening, and eligibility assessment. (Liberati et al.,2009).
The search query was designed to cover the three main areas and to explore
the studies at their intersection: personas, characters and avatars; AI-powered tools
including chatbots and generative AI tools; and education (Table 1).
Table 1. Search query
Search query (Web of Science)1004

ALL=((persona or personas* or character or characters* or avatar or avatars*) and( chat-bot or "chat bot" or chatbot or chatgpt or "gpt*" or ai or "open ai") and(education* or school*))
The query was applied (in April 2023) to the Web of Science database yielding
576 results (both journal and conference papers). Papers had to focus on the use of AI,
chatbots and virtual environments in education and had to be oriented towards theinteraction between AI and students. Additionally, the papers should be based on a
conducted study rather than being purely theoretical (Table 2).
Table 2. The first set (A) of the study inclusion and exclusion criteria
Criteria
A1
A2
A3

The paper should focus on the use of AI-empowered chatbots, avatars,
personas and virtual environments in education
The paper should include studying interaction between the tools and students
The paper should be based on a conducted study rather than being purely
theoretical

These initially agreed-upon criteria were applied to 80 papers by examining thepapers’ titles and abstracts by both coders. The degree of mutual agreement wasmeasured in the form of Cohen’s kappa coefficient (Altman, 1990; Landis & Koch,
1977). Coding resulted in excellent agreement with the interrater reliability for the raters
found to be Kappa = 0.91. The selection of the remaining papers was performed by one
coder by applying the approach agreed upon by the two coders.
The application of inclusion and exclusion criteria resulted in a set of 56 studies.
To extract the most relevant and the most rigorous studies, the second set of inclusionand exclusion criteria was applied to further narrow the results (Table 3).
Table 3. The second set (B) of the study inclusion and exclusion criteria
Criteria
B1
B2

Study needs to explore the concept of interactive or personalized learning
Study needs to measure the impact on learning and wellbeing

This finally resulted in 35 studies for which full-text screening was conducted.
The full-text screening was more thorough compared to the title and abstract screening
and especially focused on selecting the studies that used techniques based oninteraction and personalized learning in improving the educational processes. Following
that, 25 papers were excluded, leaving 10 papers in the final list.
The whole PRISMA process is depicted in the diagram in Figure 1.

Figure 1. Prisma diagram flow for the study.

1005

4.

Results

A total of 10 studies were selected for close examination as part of this review. All ofthem focus on education or propose an approach that could be applied in educational
scenarios. They are oriented towards all ages of participants including students andchildren, with sample sizes ranging from 20 to 47. (Table 3).
Table 3. The final set of selected studies (N=10) after the application of the literature
review steps
Author and year

Topic

Terzidou et al., 2016
Pataranutaporn et al., 2021
Van Brummelen et al., 2021
Haller & Rebedea, 2013
Chien & Yao, 2020
Xu et al., 2022
Park et al., 2022
Almahri et al., 2019
Fu et al., 2022

Pedagogical agents
AI generated characters
Conversational agents
Chatbots
Chatbots
Interactive video
Humanoid robot
Persona design
Knowledge-grounded
conversation
Persona design

Almahri et al., 2021

Targeted
age groups
Students
All ages
Students
All ages
Students
ChildrenAll ages
All ages
All ages

User study
participants
41
47
12
20
-

All ages-

Overall, the studies problematize interaction between humans and technology as
a key to students’ better performance and. Xu et al. (2022) suggest that enabling thekind of contingent interaction between child viewers and media characters can bring
additional educational benefits not available through standard video programming.
There is an overarching focus on pedagogical agents which are intelligent
computer programs that can interact with students to provide guidance and feedback.
They are designed to be conversational and engage students in a personalized andinteractive learning experience. Terzidou et al. (2016) problematized the effectivenessof pedagogical agents in supporting student learning in online courses and found thatthe use of pedagogical agents can have a positive impact on students' learning.
Pataranutaporn et al. (2021), Fu et al. (2022), Almahri et al. (2019), Almahri et al.
(2021), Park et al. (2022) and Chien & Yao (2020) found that personalization can
positively affect students' motivation, engagement, and outcomes while similarly Van
Brummelen et al. (2021) found that collaborative virtual environments can enhance
student learning outcomes.
Haller & Rebedea (2013) presented a method for building a conversational agent
with personality and knowledge about historical figures, which can be used in
educational contexts. Similarly, Fu et al. (2022) highlighted importance of agents'
personal memory in conversation that can significantly improve their responses. Xu etal. (2022) proposed a more advanced framework for designing pedagogical agents thatask questions, give feedback, use scaffolding, and can adapt to students' learningneeds in real-time. The study showed that this kind of framework can improve theeffectiveness of pedagogical agents in supporting student learning.

5.

Discussion

By integrating pedagogical approaches into generative AI, educators can leveragethese systems as valuable teaching tools that support and enhance student learning
outcomes. For example, in their study Xu et al. (2022) show that scaffolding, a concept
rooted in pedagogy that refers to providing learners with support and guidance as they1006

progress in their learning journey, can enhance learning experience. In the context ofgenerative AI, the AI models can be designed to scaffold learners' understanding by
providing step-by-step explanations, hints, or examples tailored to their individual needs
and can act as virtual tutors offering personalized guidance and support.
Interaction is a key element in the integration of generative AI into educationalsettings, enabling learners to actively engage with AI systems in meaningful anddynamic ways. One form of interactive AI is chatbots, which have gained popularity in
education. Chatbots can serve as virtual tutors or assistants, providing instantfeedback, answering questions, and guiding learners through various educational tasks.
Another form of interactive AI is humanoid robots, which offer a physical presence andmimic human-like interactions. These robots can engage in conversations, demonstrateconcepts, or facilitate hands-on activities. Language models, such as large language
models (LLMs), are another powerful tool for interactive AI in education. These models
have been trained on vast amounts of data and can process and generate human-like
responses to text inputs. LLMs can engage in natural language conversations withlearners, providing information, explanations, and insights.
Through dialogue and interaction, AI systems can adapt to learners' responses,
tailor content and activities to their specific needs, and offer customized feedback. Theinteractive nature of AI systems fosters active learning, promotes self-directed
exploration, and empowers learners to take an active role in their educational journey.
However, the integration of interactive AI in education should be carefully designed andaccompanied by human guidance to ensure its effectiveness and ethical use.
Personalization is a key objective in education, and generative AI has thepotential to adapt to individual learners, enhancing personalized learning experiences.
By leveraging data and advanced algorithms, generative AI models can analyze student
performance, preferences, and learning patterns. This information can then be used to
tailor the content and delivery of educational materials to meet each student's specificneeds. The configuration and adaptability of generative AI models allow for dynamic
adjustments, providing customized learning paths, adaptive assessments, andindividualized support. This personalized approach empowers students to learn at theirown pace, explore their interests, and maximize their learning potential.

6.

Conclusions

The development of generative AI has brought the potential for significant
advancements in the fields of pedagogy, interaction, and personalization in education.
By integrating pedagogical principles into AI systems, educators can leverage
generative AI as powerful teaching tools, supporting and enhancing student learning
outcomes. The interactive nature of AI systems, such as chatbots, humanoid robots,and LLMs, promotes engagement and facilitates dynamic learning experiences. Thesesystems allow learners to actively interact with educational content and receive
immediate feedback fostering active participation and deeper understanding. The
combination of pedagogy, interaction, and personalization in generative AI holds great
potential to revolutionize education, providing learners with more effective, engaging,and personalized learning experiences. However, it is important to ensure that the
integration of AI in education is accompanied by ethical considerations and humanguidance to maximize its benefits and mitigate any potential challenges. With further
research, development, and collaboration between AI technologies and humaneducators, the future of generative AI in education looks promising, with atransformative impact on teaching and learning.

1007

References
Almahri, F. A. A. J., Bell, D., & Arzoky, M. (2019). Personas design for conversational systems in
education. Informatics, 6(4). https://doi.org/10.3390/informatics6040046
Almahri, F. A. A. J., Bell, D., & Arzoky, M. (2021). Applications of machine learning in education:
Personas design for chatbots. In Machine Learning Approaches for Improvising Modern
Learning Systems. https://doi.org/10.4018/978-1-7998-5009-0.ch004
Altman, D. G. (1990). Practical Statistics for Medical Research. In Practical Statistics for Medical
Research. https://doi.org/10.1201/9780429258589
Brown, E., Brailsford, T., Fisher, T., Moore, A., & Ashman, H. (2006). Reappraising cognitive styles
in adaptive web applications. Proceedings of the 15th International Conference on World
Wide Web. https://doi.org/10.1145/1135777.1135827
Chien, Y. H., & Yao, C. K. (2020). Development of an ai userbot for engineering design education
using an intent and flow combined framework. Applied Sciences (Switzerland), 10(22).
https://doi.org/10.3390/app10227970
Crearie, L. (2013). Human Computer Interaction (HCI) factors in technology enhanced learning.
ICICTE 2013 Proceedings.
Fu, T., Zhao, X., Tao, C., Wen, J. R., & Yan, R. (2022). There Are a Thousand Hamlets in a
Thousand People’s Eyes: Enhancing Knowledge-grounded Dialogue with Personal Memory.
Proceedings of the Annual Meeting of the Association for Computational Linguistics, 1.
https://doi.org/10.18653/v1/2022.acl-long.270
Haller, E., & Rebedea, T. (2013). Designing a chat-bot that simulates an historical figure.
Proceedings - 19th International Conference on Control Systems and Computer Science,
CSCS 2013. https://doi.org/10.1109/CSCS.2013.85
Huang, J., Saleh, S., & Liu, Y. (2021). A review on artificial intelligence in education. Academic
Journal of Interdisciplinary Studies, 10(3). https://doi.org/10.36941/AJIS-2021-0077
Järvelä, S. (2006). Personalised Learning? New Insights into Fostering Learning Capacity.
https://doi.org/10.1787/9789264036604-3-enLandis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical
Data. Biometrics, 33(1). https://doi.org/10.2307/2529310
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M.,
Devereaux, P. J., Kleijnen, J., & Moher, D. (2009). The PRISMA statement for reporting
systematic reviews and meta-analyses of studies that evaluate health care interventions:
Explanation
andelaboration.
In
PLoS
Medicine
(Vol.
6,
Issue
7).
https://doi.org/10.1371/journal.pmed.1000100
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI
and the future of education: Ragnarök or reformation? A paradoxical perspective frommanagement educators. International Journal of Management Education, 21(2).
https://doi.org/10.1016/j.ijme.2023.100790
Ouyang, F., & Jiao, P. (2021). Artificial intelligence in education: The three paradigms. Computers
and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100020
Park, C., Jang, Y., Lee, S., Park, S., & Lim, H. (2022). FREETALKY: Don’t Be Afraid!
Conversations Made Easier by a Humanoid Robot using Persona-based Dialogue. 2022
Language Resources and Evaluation Conference, LREC 2022.
Pataranutaporn, P., Danry, V., Leong, J., Punpongsanon, P., Novy, D., Maes, P., & Sra, M. (2021).
AI-generated characters for supporting personalized learning and well-being. In Nature
Machine Intelligence (Vol. 3, Issue 12). https://doi.org/10.1038/s42256-021-00417-9
Terzidou, T., Tsiatsos, T., Miliou, C., & Sourvinou, A. (2016). Agent Supported Serious Game
Environment.
IEEE
Transactions
on
Learning
Technologies,
9(3).
https://doi.org/10.1109/TLT.2016.2521649
Van Brummelen, J., Tabunshchyk, V., & Heng, T. (2021). Alexa, Can i Program You?": Student
Perceptions of Conversational Artificial Intelligence before and after Programming Alexa.
ProceedingsofInteraction
DesignandChildren,IDC
2021.
https://doi.org/10.1145/3459990.3460730
Xu, Y., Vigil, V., Bustamante, A. S., & Warschauer, M. (2022). “Elinor’s Talking to Me!”:Integrating
Conversational AI into Children’s Narrative Science Programming. Conference on Human
Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3491102.3502050

1008

Shih, JL. et al. (Eds.) (2023). Proceedings of the 31st International Conference on Computers in
Education. Asia-Pacific Society for Computers in Education

Supporting Content Creators in Creating
Accessible Digital Materials in Higher
Education
Eirik HANSSENa & Weiqin CHENa,b*
Oslo Metropolitan University, Norwayb
University of Bergen, Norway*weiche@oslomet.noa

Abstract: The recent growth of the digitalization of education has increased theimportance of digital accessibility. In higher education, faculty members andadministrative staff are the main content creators who are responsible for creating andpublishing digital content for both students and university staff. Literature shows thatthese content creators do not have the necessary knowledge to create accessible
digital content. At the same time, there is an abundance of information available online
about how to create accessible digital content. However, due to the lack of time thecontent creators are often not able to find the necessary information to help them withthe particular accessibility task they have at hand. In this study we have developed an
online guide aiming at assisting content creators to create accessible content wheretarget users have been involved in the iterative design, development and evaluation
process. A summative evaluation was also conducted with a mixed-method approach,combining observation, survey, semi-structured interview, and document inspection.
The results show that the online guide has potential to help faculty members andadministrative staff to improve the accessibility of their digital content.
Keywords: Digital accessibility, content creator, higher education, summativeevaluation

1. Introduction
The recent growth of the digitalization of education has increased the importance of digitalaccessibility (Chen, 2022; Lazar, 2022) which refers to the design and development of digitalcontent, technologies, and environments that can be accessed and used by all people,
regardless of their disabilities or impairments. Many students who have previously
experienced barriers in the digital part of their education risk complete exclusion from theuniversity experience after the education has gone mostly or entirely digital (Lazar, 2022).
Digital technologies and content that students use include websites, e-books,
content/learning management systems for courses, registration mechanisms, video andother multimedia content (Lazar, 2021), as well as learning materials and documents invarious formats such as .docx and PDF, presentations such as PowerPoint files. In higher
education, faculty members and administrative staff are the main content creators who areresponsible for creating and publishing digital content for both students and university staff.
Content creators use many different languages, platforms, devices, operating systems, andcloud-based as well as locally installed software when creating digital content.
Literature has shown that content creators in higher education, such as faculty
members, do not have the necessary knowledge to make their digital content accessible(Sanderson et al., 2022). There is already a wealth of online resources where one can learn
about digital accessibility and how to make content accessible such as the information
provided by the Web Accessibility Initiative (WAI) and many YouTube videos. However, due
to the lack of time, it is often difficult for faculty members and administrative staff to look forthe necessary information that can help them with the particular accessibility task they have1009

at hand. It is, therefore, necessary to provide customized guidance and support to content
creators so that they can gain practical knowledge in the process of making digital materials
accessible.
The goal of this research is to better support content creators in creating accessible
digital materials in the context of higher education. We developed an online guide forassisting content creators and conducted a summative evaluation with the target users tounderstand how the online guide can contribute to improving the accessibility of the content
created by faculty members and administrative staff in higher education.

2. Related Research
Digital technologies have been viewed in scientific literature as resources for achieving
inclusion education, both in virtual and in-class environments (Perera-Rodríguez & Díez,2019). However, studies have shown that digital technologies create new barriers forstudents with disabilities. Inaccessible information and technology have been found to be
one of the main barriers for students with disabilities in higher education (Moriña, 2017). The
inaccessible digital materials, including peer-reviewed journal databases and digital tools
and environments, hinder the full participation of students (Seale et al., 2015). Fichten andcolleagues (2022) argued that future publishers, web designers and educators must be
trained on the fundamentals of accessible documents and web design.
Several studies examining accessibility and inclusion in higher education havehighlighted that faculty members and administrative staff lack the necessary knowledge ofaccessibility (Shinohara et al., 2018). A more recent study (Sanderson et al., 2022)
conducted interviews with 35 faculty members in the faculties of computer science andengineering. The findings from the study showed that most participants lack sufficientunderstanding of digital barriers and practical knowledge on how to make digital learningmaterials and courses accessible. Providing training to faculty members and administrative
staff is considered essential to increase competencies in digital accessibility (Bong & Chen,2021). However, despite of the willingness to learn, faculty and staff were found in lack oftime and resources to gain knowledge about and implement digital accessibility in addition totheir already full workload (Sanderson et al., 2022).

3. The Interactive Guide for Content Creators
The Interactive guide is a website where content creators can look for guidance for creating
accessible digital materials based on the task they are working on.
The first step in the design of the interactive guide is to gather the types of digitalcontent often used in higher education and the accessibility requirements and guidelines.
Digital content comes in many file formats ranging from word processor documents,
spreadsheets, and presentations to web-based content presented in text and structured HTML,with presentation styled in CSS and possibly being made interactive either using native
constructs of the HTML language or by being supplemented with JavaScript. Multimedia suchas bitmap images, Scalable Vector Graphics, audio, and video are presented within different
types of files and web pages. In addition to accessibility guidelines and standards such as the
Web Content Accessibility Guidelines (WCAG), which have clear requirements concerning
digital content, different software, platforms, and operating systems have also developed theirown guidelines for creating accessible content. For example, Microsoft has supporting videos
and manuals on how to make accessible Word documents and PowerPoint presentations fordifferent versions and operating systems. Apple has provided guidelines for accessible
documents and presentations.
The interactive guide was developed through an iterative process. Three iterations ofprototyping and user testing were carried out. During the iterations three participants wereinvolved in the testing process and gave feedback on the design, content, and organizationof the content in the interactive guide (uuguiden.no). Figure 1 shows that users can apply

1010

filters to the guide to get the necessary information related to the task they are working on.
The filters include the types of content and the software they use, as shown in Figure 1.
Another option for users is to browse a table of contents.

Figure 1. The interface where users can apply filters to the guide.

4. Evaluation
4.1 Method
The summative evaluation included two parts which collected both qualitative andquantitative data. The first part was a usability study with observations and interviews. The
participants were asked to test the interactive guide by completing some tasks in creating
accessible digital content or finding information about making a component (e.g. table,
image) accessible. Observation notes were taken during the testing, followed by a semistructured individual interview. The interview questions focused on the participant’s
experience with the interactive guide. The interview was audio recorded and analyzed using
a thematic analysis approach. The second part involved document inspection. Eachparticipant was first asked to submit one of their Word documents or PowerPoint
presentations (source file) and the corresponding PDF file. Then they were asked to makethe documents (both sources and PDF files) accessible using the information provided by
the interactive guide and submit the accessible source files and the accessible PDF files.
The Word and PowerPoint files were analyzed using the Accessibility Checker built into
Microsoft 365 and manual checks. PDF files were analyzed using the Accessibility Checker
in Adobe Acrobat Pro, and the command-line exiftool for checking the meta-data of the PDF
files. At the end of each part of the evaluation, the participants were asked to complete the
System Usability Scale (SUS) questionnaire (Brooke, 1995) which includes ten statements
adapted to the interactive guide. The study was reported to the Data Protection Services in
the National Agency for Shared Services in Education and Research (Sikt) which evaluatedand approved the processing of personal data in this study.

4.2 Results
4.2.1 Participants
The participants were recruited using a convenience sampling method. Emails were sent tocolleagues in the university and in our professional networks. Six participants (P1-P6) took1011

part in the usability testing and eight (P7-P14) took part in the document inspection. All
participants completed the SUS questionnaire. The working areas of the participants areshown in Table 1.
Table 1. The working areas of the participants
Position typeParticipants
Administrative
P1, P2, P4, P5, P12
Teaching and Research
P3, P6, P7, P8, P10, P11, P13, P14
LibraryP9
4.2.2 Document Inspection
Seven participants submitted the before and after versions of Word documents and PDF
files and one participant submitted the before and after version of a PowerPoint
presentation. The accessibility inspection of the submitted Word documents, PowerPoint
presentations and the corresponding PDF file focused on the components including title,
language, heading, text, image, table, link, reading order and slide title for PowerPoint
presentations and language and bookmark in PDF files.
Table 2 presents the results from the document inspection. Because the prototype of
the interactive guide has not yet provided all the guidance concerning the components, inTable 2 we only include those that the guide has covered. The document inspection showedthat participants were able to improve their documents in text alternatives of images, color
contrast, and the correct use of headings. Accessibility issues have also been found in
document titles, language (both in source and PDF files) and PDF bookmarks. Most of theaccessibility issues found with tables were not corrected after the participants used theinteractive guide. One possible explanation could be that some of those issues are notcovered by the built-in accessibility checker in Microsoft 365, which is the case for tables
missing heading rows. The automatic checker did report tables with split or merged cells, but
those errors were not fixed by the participants in this study. It could be that this topic needed
to be addressed better in the interactive guide or that the participants needed more
motivation or time to correct them.
Table 2. Accessibility issues found in document inspection (before/after)
Components
BeforeAfter
Heading
29
28
Text*
136
36
Image*
96
65
Table*
49
46
Link*
1
1
Reading order (PowerPoint)
5
4
Slide title (PowerPoint)11
Note: components with * mean that the interactive guide does not cover all the accessibility
issues, and the number in the table for these components shows only the covered issues.
4.2.3 SUS Score
The lowest SUS score is 42.5 given by P9 and the highest is 82.5 given by P12. The meanscore is 66.1 and standard deviation is 15.2. According to Bangor et al. (2009), if a SUS
mean score is over 60 the system evaluated can be considered “Good”, and only when themean score is over 75 can the system be considered “Excellent”. In our study, the meanscore is 66.1, so it has only reached the “Good” level.

1012

4.2.4 Results from Qualitative Analysis
The qualitative data is collected through observations and semi-structured interviews
associated with user testing and the open-ended questions after the SUS questionnaire.
The participants generally thought the interactive guide was clear and easy to use.
We observed that the participants had different preferences concerning using the table ofcontent or the filters. The table of contents gave an overview of the guides, including theimportant points for users to remember when creating accessible content. Some participantspreferred the filtering function while others felt that they did not need the filter options
because the table of contents is not very long. The participants have also reported that they
have learned new techniques from the online guide, the websites linked from the online
guide and through discussions during the user testing sessions. For example, one
participant was asked to create an accessible table in the user testing. The participant
carried out the task as s/he used to do. After being encouraged to look at the online guideand the linked video from the guide, the participant was able to create an accessible table.
Soon after the testing, the participant needed to prepare a large report with many tables andwas happy to report back that s/he was able to put the new skills to use.
The participants have also provided valuable feedback to improve the online guideconcerning both the usability and the content of the guide. Some suggested more content,
including more details related to image text alternatives, working with tables in an HTMLsource editor in an LMS, and captioning videos. Several participants suggested the guide
should be more practically oriented and use images, short videos or more advanced HTMLcode examples to demonstrate the techniques.
For people who would not have the expertise, […] it's difficult for them to understandwhat it means without pictures or images that should demonstrate how things should be
done. […] if it's a user guide you have pictures to guide the users on how to do things. (P9).
Concerning the organization of the content in the online guide, one of the participantssuggested that the topics should be organized in different ways (e.g. based on software orapplications) so that they satisfy the needs of users with different preferences.

5. Discussion
Studies have shown that students with disabilities face a series of barriers that hinder theirsuccess in higher education and faculty members have been identified as one of the mainbarriers for students with disabilities in higher education (Zhang et al., 2018). Research has
also shown that faculty members lack motivation and relevant knowledge and skills to
respond to the needs of students with disabilities. Staff training and administrative support
have been recommended as measures to address the barriers related to faculty members.
Several studies (Chen et al., 2018; Linder et al., 2015; Sanderson et al., 2022) haveargued that in order to provide training and support to faculty members, higher education
institutions should establish policies and strategies, and allocate resources for
implementation. Some participants in our study have also commented on the responsibilities
of institutions and pointed out that faculty members should not be the only responsible party.
In our study, the participants are motivated to learn and make their digital content
accessible. They are aware that they lack the necessary knowledge and skills andexpressed that they wish to have someone they can ask when facing challenges with theaccessibility of their digital content or get help about what tools are good and how to use
them. They thought that this online guide was timely and important.
The online guide does not cover all possible accessibility issues. The accessibility
improvements the participants made in their files when using the online guide were not very
impressive, but there were some improvements. Some participants had yet to improve on all
the issues the guide covered after using it. This could partly be explained by how motivatedthe participants were and how much time they were willing to spend testing the guide andimproving their files in a busy life. Nevertheless, the data suggest that the online guide has
potential but needs further improvement.

1013

6. Conclusion and Future WorkThe research presented in this paper aims to address the challenges faced by content
creators in higher education in creating accessible digital content by providing an easy-touse online guide. The online guide includes carefully selected content which can help
content creators with the task at hand so that they do not have to search through a large
amount of online information. Despite some usability challenges, the participants were able
to use the guide to learn new techniques and improve the accessibility of their digitalcontents. The results suggest that there is potential in an online guide such as this to help
content creators in creating accessible digital content.

References
Bong, W. K., & Chen, W. (2021). Increasing faculty’s competence in digital accessibility for inclusive
education: a systematic literature review. International Journal of Inclusive Education, 1-17.
https://doi.org/10.1080/13603116.2021.1937344
Brooke, J. (1995). SUS: A quick and dirty usability scale. In W. P. Jordan, B. Thomas, I. L.
McClelland, & B. Weermeester (Eds.), Usability Evaluation In Industry (1st ed.). Taylor & Francis.
Chen, W. (2022). Designing a Professional Development Program on Digital Accessibility andInclusive Education for Faculty Members. In S. Iyer, J.-L. Shih, W. Chen, & M. N. Md Khambari(Eds.), Proceedings of the 30th International Conference on Computers in Education. (pp. 684686). Asia-Pacific Society for Computers in Education.
Chen, W., Sanderson, N. C., & Kessel, S. (2018). Making Learning materials accessible in higher
education—attitudes among technology faculty members. In G. Craddock, C. Doran, L. McNutt,& D. Rice (Eds.), Studies in health technology and informatics (Vol. 256, pp. 87-97). IOS Press.
https://doi.org/https://doi.org/10.3233/978-1-61499-923-2-87
Fichten, C. S., Martiniello, N., Asuncion, N., Coughlan, T., & Havel, A. (2022). Changing times:
Emerging technologies for students with disabilities in higher education. In J. Madaus & L. Lyman(Eds.), Handbook of higher education and disability.
Lazar, J. (2022). Managing digital accessibility at universities during the COVID-19 pandemic.
Universal Access in the Information Society, 21, 749–765.
https://doi.org/https://doi.org/10.1007/s10209-021-00792-5
Linder, K. E., Fontaine-Rainen, D. L., & Behling, K. (2015). Whose job is it? Key challenges and future
directions for online accessibility in US Institutions of Higher Education. Open Learning: The
Journal of Open, Distance and e-Learning, 30(1), 21-34.
https://doi.org/https://doi.org/10.1080/02680513.2015.1007859
Moriña, A. (2017). Inclusive education in higher education: challenges and opportunities. European
Journal of Special Needs Education 32(1), 3-17. https://doi.org/10.1080/08856257.2016.1254964
Perera-Rodríguez, V.-H., & Díez, A. M. (2019). Technological challenges and students with disabilities
in higher education. Exceptionality, 27(1), 65-76.
https://doi.org/10.1080/09362835.2017.1409117
Sanderson, N. C., Kessel, S., & Chen, W. (2022). What do faculty members know about universal
design and digital accessibility? A qualitative study in computer science and engineering
disciplines. Universal Access in the Information Society, 21(2), 351–365.
https://doi.org/https://doi.org/10.1007/s10209-022-00875-x
Seale, J., Georgeson, J., Mamas, C., & Swain, J. (2015). “Not the Right Kind of ‘Digital Capital’? An
Examination of the Complex Relationship between Disabled Students, Their Technologies andHigher Education Institutions. Computers and Education, 82, 118–128.
https://doi.org/10.1016/j.compedu.2014.11.007
Shinohara, K., Kawas, S., Ko, A. J., & Ladner, R. E. (2018). Who Teaches Accessibility? A Survey of
U.S. Computing Faculty. In Proceedings of the 49th ACM Technical Symposium on Computer
Science Education (pp. 197–202). https://doi.org/https://doi.org/10.1145/3159450.3159484
Zhang, Y., Rosen, S., Cheng, L., & Li, J. (2018). Inclusive higher education for students withdisabilities in China: What do the university teachers think? Higher Education Studies, 8(4), 104–
115. https://doi.org/doi:10.5539/hes.v8n4p104

1014

PANEL

1015

Towards a Collaborative Vision for Redesigning
Education for Harmonious and Thriving
Educational Futures in Asia and Beyond:Will Seamless IDC Theory lead us there?
Chee-Kit LOOIa*, Siu-Cheung KONGa, Ronghuai HUANGb Jon MASONc,
Hiroaki OGATAd, Hyo-Jeong SOe, Lung-Hsiang WONGfa

Education University of Hong Kong, Hong Kongb
Beijing Normal UniversitycCharles Darwin University, AustraliadKyoto University, Japane
Ewha Womans University, South Korea
f
Nanyang Technological University, Singapore*cklooi@eduhk.hk

Abstract: A framework of future education has been articulated by Asian researchers in the form of theInterest-Driven Creator (IDC) Theory. Recent collaborative efforts have attempted to broaden theframework towards visions for future education with the IDC theory still as the basis but now embracing
consideration of the ultimate educational goals of education. Such extensions can serve the purpose offurther informing the development of the theory, as well as motivating and driving us towards realizationof educational futures. The advent of the digital world and its fusion with the real world has also
necessitated the bridging of seamless learning and working spaces in the daily lives of the learners. In
this panel, we invite scholars to respond to, critique and build on building such visions of educational
futures and argue for the role of IDC theory and what it can contribute.
Keywords: Interest-Driven Creator Theory, Seamless Learning, Educational Futures

1.

Introduction

In 2018, a group of 23 (mostly) Asian researchers collaborated to develop the Interest-Driven Creator
Theory (IDC Theory), which focuses on learning activity designs that foster interest-building,knowledge creation and forming such practices habitual (Chan et al., 2018; Chan et al., 2019; Chen atel., 2020; Wong, et al., 2020). It was developed to mitigate the drawbacks of examination-driven
education and guide the design of future learning. Concomitantly, the rapidly-accelerating digitaltransformation of the real world which includes education has necessitated the seamless interleaving ofwork and learning across different learning spaces supported by various forms of technologies (Chanet al., 2006): the physical world, the cyber-world, AR and VR-based environments, and digital
environments in which learning companions thrive and support learning. We postulate the fusion of
IDC Theory with seamless learning to form an integrated theory about the continuity of IDC-basedlearning in future seamless learning spaces - called Seamless IDC Theory (SIDC Theory).
There have also been international efforts to define international education goals, such as the reports by
UNESCO. The report on “Reimagining Our Futures Together: a New Social Contract for Education”
identifies key issues that all countries should center their educational systems around, and other efforts
around the world to imagine educational futures (UNESCO, 2019). The recent July 2023 UNESCO
GEM report provides some provocative policy advice such as a worldwide ban on smartphones in theclassroom (UNESCO, 2023). In envisaging harmonious and thriving Educational Futures in Asia andbeyond, we need informed visions, strategies and frameworks for re-imagining education. We postulate1016

that SIDC is one such vision, and it calls for collective efforts by interested researchers to work towards
some consensus of educational goals for global educational futures, and think about the means toachieve these goals. This panel will discuss the role of the SIDC Theory in contributing towards these
global educational futures.

2.

Abstracts of Individual Panelists’ Presentations

There are questions that need to be answered in order to fully realize the potential of the SIDC theory.
For example, how can we ensure that the theory is adaptable and flexible enough to account for the
rapidly changing landscape of education and technology? How can we ensure that SIDC-based learningis accessible to all learners, regardless of their background or circumstances? Additionally, how can wemeasure the effectiveness of SIDC-based learning and ensure that it is aligned with global educational
goals and futures? Another important question to consider is how SIDC Theory will impact daily lifein the digital future. With the shift towards SIDC-based work, family life, and entertainment, or will
daily life itself become SIDC-based, what challenges and opportunities will arise as a result of this shift,and how can we ensure that we are prepared to meet them? How can we ensure that SIDC-based learningand daily life are balanced and integrated in a way that supports learners' overall well-being andsuccess? These are just a few of the questions that will need to be explored and answered in this panel.

1.1
Revisiting the value of IDC Theory (Ronghuai Huang)
The IDC Theory, comprising three phases (Imitating, Combining, and Staging), serves as a
powerful framework for designing effective learning activities. Its aim is to actively engage
students, foster curiosity, and provide continuous support throughout their learning journey.
Sustaining engagement over time is essential for empowering students to overcome challenges
and develop lifelong learning habits. Authentic learning is a main focal point of the IDC Theory.
It involves real-life experiences where students create tangible products to share with theircommunities. Cultivating and sustaining students' interest and curiosity is crucial to enhancethe effectiveness of authentic learning. Learning engagement is another crucial aspect
addressed by the theory, encompassing the time, energy, positive emotions, and learning
strategies that students invest in educational activities. Considering this issue is important toensure effective learning for all students. Active learning is the third core issue addressed by
the IDC Theory. It actively engages students in meaningful activities that encourage criticalthinking. Self-regulated learning, including self-planning, self-monitoring, and self-assessment,is recognized as a prominent feature of active learning. The IDC Theory provides acomprehensive framework for creating effective learning activities by actively engaging
students, fostering curiosity, and providing continuous support. It addresses the key issues ofauthentic learning, learning engagement, and active learning. Implementing the recommended
strategies will contribute to enhancing future learning experiences.
1.2
We need more evidence! (Hiroaki Ogata)
People are all different and may prefer different ways of learning. IDC theory is just one of
methods, and we need more evidence for IDC theory, for example, for whom, for what kind of
domains, when and where, IDC theory will succeed or not. Although evidence is not always
correct, we as a researcher need to do more research and find empirical evidence. Also we needto establish a social information infrastructure (platform) to extract and share evidence fromdaily learning log data.
1.3
Fusing Seamless Learning and IDC Practice: Does it Foster Lifelong Learning and
Enhance Students' Well-Being? (Lung-Hsiang Wong)
At the nexus of educational innovation, the fusion of seamless learning and Interest-Driven
Creator (IDC) Practice emerges as a compelling vision to reshape the landscape of learning.
"Seamless IDC is envisioned to bring the IDC learning process (interest, creation, habit) into a1017

seamless learning experience where the practice shall be ongoing and across time and spaces."
This encapsulates the cornerstone of our exploration – a seamless confluence of IDC principles
and the fluidity of seamless learning. Empowered by this fusion, students embark on an
engaging odyssey, where curiosity sparks creation, creation fuels habit, and habit nurtures
sustained exploration. Furthermore, the fusion propels students toward a balanced well-being,enabling them to navigate challenges with perseverance and engage with the world
purposefully. I envision that this approach will lay the foundation for a life where learning
becomes a joyful and intrinsic part of existence. Lifelong learners, equipped with thedisposition and skills honed through seamless IDC, approach each day with a desire forintellectual growth. They are more resilient in the face of challenges, more adaptable amidchanges, and more capable of contributing meaningfully to the ever-evolving global
community. As the boundaries between formal education and real-world application blur, thefusion of seamless learning and IDC practice becomes a beacon guiding individuals towards alife enriched by perpetual learning and personal fulfilment.
1.4
Engaging with and within the digital environment (Jon MASON)
As educators, we must engage in theorising our practice. This scholarship of teaching andlearning helps bring into alignment our vision with the realities of our changing world. This isdemanding for several reasons: the evolving digital environment is both an enabler anddisruptor; the field of educational technology is profoundly multidisciplinary, requiring us tolook beyond educational theory. Moreover, demands for global citizenship and intercultural
understanding are increasingly adopted within school curricula. Enriching IDC Theory with'seamless' perspectives in relation to transitioning between the virtual, artificial, and real worlds
provides a timely example of the theorising that is necessary. At the same time, however, I
have concerns whether such theorising is sufficient. While the Global Harwell vision iscompelling, the question 'will Seamless IDC Theory lead us' there is one I'm not so sure about.
It may contribute, but I sense we will need to navigate a lot more theorising as the impact ofGenerative AI and the Web 3.0 become clearer. My main argument is that (digital) technology
is no longer 'just a tool'; it is a complex adaptive environment of systems that we are often
immersed within. In my experience, both seamlessness and boundaries support learning.
Conundrums and wicked problems demand we pause. Solving them is often not a seamlessexperience. Learning to navigate the 21st century is not necessarily going to be an easy ride.
As ‘smart’ systems and frameworks for learning develop in response to advancements intechnology we also need to pay attention to how we might develop our human ‘smarts’ (Mason,
2023).
1.5
Revisiting the Promises of Seamless Learning (Hyo-Jeong SO)
The SIDC theory envisions continuous interest-driven learning in an increasingly connected
and intelligent world. By revisiting the original concept of seamless learning, we can gain aclearer understanding of why and how this idea was introduced. The original notion of seamlesslearning, put forth in 2006, underscores the importance of students being able to "learn
whenever they are curious," particularly emphasizing the value of "out-of-school learning
driven by the personal interests of students." This notion inherently encompasses two key
elements: learner curiosity and personal interests. Within the Asian research community, it isessential for us to reflect on whether we have deep understandings of how to support learnercuriosity and interests, both with and without the use of technological tools. Chan's recent
perspective on "well-seamlessness" and "ill-seamlessness" (2022) sheds light on potential
negative impacts that overly seamless experiences might have on our lives. Much of the public
perception surrounding seamlessness also points to instances of "ill-seamlessness," including
media/device addiction, mental distractions, diminished human interaction, excessive1018

dependence on technology, and more. During the panel, I will revisit some of the promises
underlying seamless learning and discuss what are the fundamental barriers and challenges that
require our attention as to reach the aspiration of harmony and well-being as the educationalgoal.
3. Discussion and Conclusion
We articulate the Seamless IDC Theory (SIDC Theory) as a promising direction for envisioning
educational futures. This panel is the start of many discussions that seek to achieve consensus on the
educational goals of such futures, and to operationalize how theories like SIDC can lead to a thriving
sustainable eco-system to help achieve these educational goals. Panelists from different countries andregions provide diverse perspectives that help a collaborative vision of educational futures in the digitalage.

References
Chan, T.-W., Looi, C. K., Chen, W., Wong, L.-H., Chang, B., Liao, C. C.-Y., … Ogata, H. (2018).
Interest-driven creator theory: Towards a theory of learning design for Asia in the twenty-first century.
Journal of Computers in Education, 5(4), 435-461.
Chan, T.-W., Looi, C. K., Chang, B., Chen, W., Wong, L.-H., Wong, S. L., … Chou, C.-Y. (2019). IDC
theory: creation and the creation loop. Research and Practice in Technology Enhanced Learning, 14,
26 (2019).
Chan, T.-W., Roschelle, J., Hsi, S., Kinshuk., Sharples, M., Brown, T., … Cherniavsky, J. (2006). Oneto-one technology-enhanced learning: An opportunity for global research collaboration. Research andPractice in Technology Enhanced Learning, 1(1), 3-29.
Chen, W., Chan, T.-W., Wong, L.-H., Looi, C. K., Liao, C. C.,-Cheng, H. H.-N., … Pi, Z. (2020). IDC
theory: habit and the habit loop. Research and Practice in Technology Enhanced Learning, 15, 10.
Fawns, T. (2022). An entangled pedagogy: Looking beyond the pedagogy—technology dichotomy.
Postdigital Science and Education, 4(3), 711-728.
Mason, J. (2023). The search continues: Content used to be king, but context is now everything – and‘smart’ can morph into ‘stupid’ [Online]. The Sociological Review Magazine.
https://doi.org/10.51428/tsr.ngtq5087
UNESCO (2023). Global education monitoring report, 2023: Technology in education: A tool on whoseterms? - UNESCO Digital Library. https://unesdoc.unesco.org/ark:/48223/pf0000385723
UNESCO (2019). "Futures of Education: learning to become" available at:https://unesdoc.unesco.org/ark:/48223/pf0000370801?posInSet=444&queryId=812a8f42-ff53-4977ab6a-cd13641a499e.pdf
Wong, L.-H., Chan, T.-W., Chen, W., Looi, C.-K., Chen, Z.-H., Liao, C. C.-Y., King, R. B., & Wong,
S. L. (2020). IDC theory: Interest and the interest loop. Research and Practice in Technology Enhanced
Learning, 15, 3 (2020).

1019

Mobile Learning: Reflections on the Past
and Visions for the Future
Lung-Hsiang WONGa*, Daner SUNb, Hiroaki OGATAc, Hyo-Jeong SOd, Xiaoqing GUe, &Ting-Chia HSUfa
Nanyang Technological University, Singaporeb
The Education University of Hong Kong, Hong Kongc
Kyoto University, Japanc
Ehwa Womans University, South Koreac
East China Normal University, ChinacNational Taiwan Normal University, Taiwan* lhwong.acad@gmail.com
Abstract: The research field of mobile learning emerged approximately two decades
ago and experienced a period of robust growth, peaking around a decade ago.
However, in recent years, the research community's attention on mobile learning has
shown signs of waning. This shift in focus could be attributed to the maturation andwidespread adoption of mobile learning intervention methods in educational practices,
rendering the field less novel and demanding. Additionally, the emergence of newtechnologies has prompted researchers to explore alternative avenues of inquiry, albeit
often incorporating mobile learning elements into their evolving techno-pedagogicalmodels. This panel discussion endeavors to convene early mobile learning researchers
for introspection, retrospection, and foresight. Through the lens of their individual
research trajectories, the panelists will critically reflect upon the past achievements,
challenges, and contributions of mobile learning. Furthermore, the discussion willencompass an exploration of potential future directions and the symbiotic integrationof mobile learning with novel technological advancements. The aim is to elucidate theevolving role of mobile learning in the realm of educational technology and its
continuing relevance within an ever-changing landscape.
Keywords: Mobile learning; Learning log; Seamless learning; Learner data-driven
paradigm; Artificial intelligence in education

1. Introduction
Mobile learning, as a specialized domain within educational technology, came into existence
approximately two decades ago, accompanied by a surge of enthusiasm and pioneeringresearch efforts (Goksu, 2021; Kukulska-Hulme & Traxler, 2005). Around a decade ago, it
attained its pinnacle, marked by a proliferation of studies, innovations, and implementation
projects across diverse educational contexts (Sun et al., 2020; Zubanova et al., 2021).
However, in recent years, a perceptible shift in the research community's focus has become
apparent, with a gradual decline in attention towards mobile learning as a discrete area of
inquiry.
This shift in attention may be attributed to several interwoven factors. Firstly, the once
burgeoning landscape of mobile learning intervention methods has undergone significant
maturation, with many strategies becoming widely accepted and integrated into educational
practices (Pishtari et al., 2020; Suárez et al., 2018). Consequently, the research emphasis on
developing and validating intervention approaches has somewhat diminished, resulting in areframing of research priorities.
Secondly, the advent of novel and alluring technologies, such as augmented reality,
deep learning or generative artificial intelligence, immersive virtual environments, and learning
analytics, has captivated researchers' interest, enticing them to venture into exploring the1020

potential of these emergent technologies in education (Goff et al., 2018; Hwang & Chen,
2023).
Amidst this evolving academic milieu, researchers who had previously been dedicatedto mobile learning investigations have redirected their research agendas, while stillrecognizing the value of mobile learning concepts and methodologies. Many researchers havebegun incorporating mobile learning elements into their novel techno-pedagogical models,
recognizing the importance of leveraging mobile devices' ubiquity and contextual capabilities.
This panel discussion aspires to congregate former chairs of APSCE Classroom,
Ubiquitous and Mobile Technology-Enhanced Learning (CUMTEL) Special Interest Group
who have played pivotal roles in building the mobile learning community within the ICCE
conference series. The intention is to engender a collective introspection, retrospection, andforesight, with the individual research trajectories of the panelists serving as case studies.
Through this reflective endeavor, we seek to elucidate the transformative journey of mobile
learning in the realm of educational technology and envision its potential future trajectories
within the broader context of emerging educational technologies. By contemplating the pastand envisioning the future, we aim to contextualize the relevance and enduring impact ofmobile learning in contemporary educational research and practice.

2. Abstracts of Individual Panelists’ Presentation
2.1 Mobile Learning + Learning log data = AI support + Evidence (Hiroaki OGATA)
Learning happens at anytime and anyplace. Mobile computing technologies allow us to recordand analyze all the learning log data throughout mobile learning support tools (Ogata et al,2011). The data then provides AI support for learning as well as evidence that shows
scientifically significant proofs of the methodology of teaching and learning. Currently, we havebeen developing LEAF (Learning and Evidence Analytics Framework) system that consists of
e-Book reader (BookRoll) and a learning analytics dashboard (LogPallet) (Ogata, et al, 2023).
LEAF is used in K12 schools and universities in Japan as well as other countries (Ogata, etal, 2022).
Our study has the potential to revolutionize the mobile learning field by merging AIdriven data analysis with educational tools. LEAF not only provides personalized learning
support but also establishes a data-centric approach to education, enabling the identificationof effective teaching methods through learner data. This shift towards evidence-based mobile
learning research is instrumental in harnessing technology for educational advancement on a
global scale.

2.2 Seamless Learning: From a sub-field of mobile learning to a learning notion at its
own right (Lung-Hsiang WONG)
Seamless learning stands out as an advanced pedagogical approach capable of catering to
the evolving needs of 21st-century learners (Sharples et al., 2012). ith its remarkable ability to
seamlessly traverse diverse learning landscapes spanning locations, social contexts, andtechnological environments, its underlying aim is to instill in students a habitual inclination tocontinually embark on the trajectory of learning-application-reflection throughrecontextualizing previously assimilated knowledge (Wong et al., 2015).
Initially, the concept of seamless learning was introduced within the realm of a
distinctive form of mobile learning. Chan et al. (2006) championed the use of mobile
technology in a 1:1, 24x7 environment to facilitate individual students' ongoing, crosscontextual seamless learning. This seminal work effectively inaugurated the trajectory of
research and application in mobile-assisted seamless learning, a trajectory that has now
extended its reach to more than 40 countries over the past two decades.

1021

Over time, researchers' perspectives on seamless learning have shifted progressively.
Initially viewed primarily from a technology-enabled (particularly 1:1-enabled) perspective(e.g., Hwang et al., 2008; Ng & Nicholas, 2007), the focus evolved to encompass curriculum
design considerations (e.g., Looi & Wong, 2013; Obisat & Hattab, 2009), and eventually, thecultivation of a conducive learning culture (e.g., Milrad et al., 2013; Wong et al., 2017). Thistransformation indeed reflects the evolving priorities of the educational technologist
community engaged with seamless learning — a shift from the creation of innovative seamlesslearning technologies to an exploration of the fundamental nature of seamless learning and its
tangible impacts in educational institutions and among adult learners. The notion that a 1:1,
24x7 setting is an absolute prerequisite for seamless learning has been reevaluated. Recent
scholarly discourse contends that seamless learning is an independent modern pedagogicalconcept — an aspiration (Sharples et al., 2012), a habit-of-mind (Wong & Looi, 2011) or evena set of metacognitive abilities (Sha, 2015). Consequently, alternative technological models
have been introduced, such as the "division of labor" model (involving the use of distinct
devices, computer setups, or even non-digital tools across different locations) (Wong, 2012;
Wong & Looi, 2022) and the incorporation of (mobile) social media (Charitonos et al., 2012;
Laru & Järvelä, 2015).
Looking ahead, the trajectory of mobile-assisted seamless learning offers promisingavenues for exploration. Integration of immersive technologies like AR/VR and metaverse,
personalized learning strategies, collaborative learning across settings, and ethicalconsiderations are key directions that hold potential to further enhance the effectiveness andreach of seamless learning experiences.

2.3 Promises and Challenges of Seamless Learning in Learner-Data Driven
Approaches (Hyo-Jeong SO)
The field of mobile learning has undergone a substantial expansion attributable to thewidespread integration of mobile devices and seamless network connectivity. Almost two
decades ago, a global network of mobile learning researchers collectively proposed thenotion of seamless learning, denoting a continuity of learning experiences when learners are
curious, across a diverse array of scenarios, facilitated through mobile devices as a main
mediator (Chan et al., 2006). Subsequently, a substantial volume of research has
investigated the theoretical and design underpinnings inherent to seamless learning. The
key contribution of seamless learning is that the notion helped researchers and practitioners
to understand (missed) learning opportunities in various informal learning contexts,
supporting experiential and contextualized learning experiences.
Nonetheless, it is unclear whether the promises of seamless learning have beenrealized and whether mobile learning has transformed the culture of teaching and learning
beyond easy access to mobile devices. In many educational settings in Asia, the utilizationof mobile devices is still limited within classrooms, coupled with educators’ and parents’concerns about the excessive usage of mobile devices. Selwyn and Aagaard (2021) assertthat within the field of educational technology, there has been a notable lag in addressing
concerns and issues linked to the utilization of mobile phones in classrooms. These
concerns encompass critical matters such as cyberbullying, phone addiction, and digital
distractions. Further, given the prevalence of learner data-driven approaches in mobile
learning, certain underlying assumptions and challenges of seamless learning necessitate
thorough reconsideration. For instance, the collection of learner data spanning various
timeframes and locations is often promoted as essential for constructing seamless learningmodels or applications. Nevertheless, it is important for researchers and developers to
uphold learners’ agency, ensuring that individuals retain the right to opt out of the tracking
and recording of their learning data at any point during their learning trajectory (Ocheja et al.,2019).
With this backdrop, the talk will center on the intrinsic tensions and challenges likely
to confront mobile learning researchers as they adopt learner data-driven paradigms within1022

their design, analytical procedures, and implementation strategies. These challenges
encompass the promotion of learners’ curiosity and a habit of mind, a critical examination ofcontinuity in learning experiences, the need for socio-emotional support, and the ethical useof learner-generated data in mobile learning.

2.4 Mobile Learning in the Age of Artificial Intelligence Education: A Personal
Journey and Future Directions (Xiaoqing GU)
In the formative stages of our research, the spotlight was on developing resources optimized
for fragmented, bite-sized learning experiences. Prior to the widespread acceptance offlipped classrooms, our investigations led to micro-learning design innovations, establishingearly precedents in mobile learning (Gu et al., 2011). With the advent of blended learning,
our research emphasis pivoted towards exploring the potential of blended mobile learning tofacilitate student progress in these increasingly integrated learning environments (Du & Gu,2019).
As the educational landscape transitioned into the era of artificial intelligence (AI), therole of mobile learning underwent a significant transformation in my team’s researchparadigm. No longer an isolated field, mobile learning became an instrumental medium toactualize AI-driven education. We believe that the essence of AI in education hinges on
capturing the continuum of the learning process. By persistently tracking learners, learning
trajectory across varied temporal and spatial dimensions, we can constructcomprehensive learner digital profiles, ensuring individualized instruction and support (Dai &Gu, 2023).
In this schema, mobile learning stands out as more than just a mere modality—it
represents flexibility, autonomy, and adaptability. As learners navigate through AIpowered educational landscapes, mobile platforms can seamlessly integrate adaptive
content, empowering students to curate their learning pathways. Mobile learning canharness the power of AI to support individual learning needs and preferences, creating a
dynamic, responsive, and personalized learning experience. Looking ahead, we see afuture characterized by a mutually beneficial symbiosis, where mobile learning, infusedwith AI capabilities, will play a significant role in the evolution of education.

2.5 AI and Mobile Learning (Ting-Chia HSU)
Informed by recent research and forward-looking perspectives, the convergence of AI andmobile learning has ignited significant scholarly interest, poised to reshape the educationallandscape. Pioneering investigations, such as those by Gumbheer et al. (2022), underscore
AI's role in tailoring mobile learning experiences, dynamically aligning content with individual
learning requirements and pace. Yazici et al. (2023) echo this sentiment, emphasizing AI,
particularly machine learning algorithms, as a means to decipher student behaviors, predict
performance, and propose interventions. Matzavela and Alepis (2021) delve into AI's
contribution to crafting adaptive dynamic tests, juxtaposing individual student profiles withpredictive models based on decision tree algorithms within mobile learning contexts.
Generative AI, typified by the ChatGPT API, enriches mobile learning through interactive,
personalized educational encounters. Its real-time detailed explanations and responses
facilitate on-the-fly learning tailored to individual needs, buoyed by a continuous learningmodel that curates content via user interactions, nurturing engaging learning environments.
Additionally, AI's image recognition prowess bolsters mobile learning, exemplified in Hsu etal.'s (2023) AI-aided image recognition and self-regulated learning study, which exhibitedenhanced vocabulary acquisition, self-regulation, and diminished learning anxiety among
third-grade students. The symbiotic fusion of AI and mobile learning heralds an era oftransformative educational processes, offering exciting prospects for future exploration.

1023

3. Discussion and Conclusion
The journey of mobile learning, spanning more than two decades, has transformed from anenthusiastic niche to a driving force in educational technology. Initially marked by pioneeringresearch, it reached its zenith about a decade ago with a surge in studies and innovations
across diverse contexts. However, the research landscape has since shifted, influenced byfactors like the maturation of mobile-assisted intervention methods and the allure of
emerging technologies such as deep learning and learning analytics.
The panelists' presentations underscore this evolution. Hiroaki Ogata's work
integrates AI with educational tools, creating evidence-based mobile learning. Lung-Hsiang
Wong's insights into seamless learning show its shift from technology-centric to a broader
pedagogical concept, with immersive technologies and personalized learning strategies
enhancing its potential. Hyo-Jeong So emphasizes the challenges and promises ofseamless learning, especially in data-driven paradigms.
Xiaoqing Gu's perspective demonstrates mobile learning's transformation into a
facilitator of AI-driven education. Ting-Chia Hsu's insights highlight AI's role in tailoring
learning experiences. In conclusion, mobile learning's journey reflects its adaptability withinemerging educational paradigms. The discussion offers valuable introspection and future
insights, showcasing mobile learning's enduring impact on education.

References
Chan, T.-W., Roschelle, J., Hsi, S., Kinshuk, Sharples, M., Brown, T., Patton, C., Cherniavsky, J.,
Pea, R., Norris, C., Soloway, E., Balacheff, N., Scardamelia, M., Dillenbourg, P., Looi, C.-K.,
Milrad, M., & Hoppe, U. (2006). One-to-one technology-enhanced learning: An opportunity for
global research collaboration. Research and Practice in Technology-Enhanced Learning,
1(1), 3-29.
Charitonos, K., Blake, C., Scanlon, E., & Jones, A. (2012). Museum learning via social and mobile
technologies: (How) can online interactions enhance the visitor experience? British Journal of
Educational Technology, 43(5), 802-819.
Dai, J., & Gu, X. Z., J. (2023). Personalized recommendation in the adaptive learning system: The
role of adaptive testing technology. Journal of Educational Computing Research, 61(3), 523545.
Du, H., & Gu, X. (2019). Exploring a blended mobile learning environment to develop students'
scientific inquiry skills in science museums. International Journal of Smart Technology andLearning, 1(4), 310-322.
Goff, E. E., Mulvey, K. L., Irvin, M. J., & Hartstone-Rose, A. (2018). Applications of augmented reality
in informal science learning sites: A review. Journal of Science Education and Technology,
27, 433-447.
Goksu, I. (2021). Bibliometric mapping of mobile learning. Telematics and Informatics, 56, 101491.
Gu, X., Gu, F., & Laffey, J. M. (2011). Designing a mobile system for lifelong learning on the move.
Journal of Computer Assisted Learning, 27(3), 204-215.
Gumbheer, C. P., Khedo, K. K., & Bungaleea, A. (2022). Personalized and adaptive context-aware
mobile learning: review, challenges and future directions. Education and Information
Technologies, 27(6), 7491-7517.
Hsu, T.-C., Chang, C., & Jen, T.-H. (2023). Artificial Intelligence image recognition using selfregulation learning strategies: effects on vocabulary acquisition, learning anxiety, and learning
behaviours of English language learners. Interactive Learning Environments.
https://doi.org/https://doi.org/10.1080/10494820.2023.2165508
Hwang, G.-J., & Chen, N.-S. (2023). Exploring the potential of generative artificial intelligence ineducation: Applications, challenges, and future research directions. Educational Technology& Society, 26(2), I-XVIII.
Hwang, G.-J., Tsai, C.-C., & Yang, S. J. H. (2008). Criteria, strategies and research issues of contextaware ubiquitous learning. Educational Technology & Society, 11(2), 81-91.

1024

Kukulska-Hulme, A., & Traxler, J. (2005). Mobile learning. Routledge.
Laru, J., & Järvelä, S. (2015). Integrated use of multiple social software tool and face-to-face activities
to support self-regulated learning: A case study in a higher education context. In L.-H. Wong,
M. Milrad, & M. Specht (Eds.), Seamless Learning in the Age of Mobile Connectivity (pp. 471484). Springer.
Looi, C.-K., & Wong, L.-H. (2013). Designing for seamless learning. In R. Luckin, P. Goodyear, B.
Grabowski, & N. Winters (Eds.), Handbook of Design in Educational Technology (pp. 146157). Routledge.
Matzavela, V., & Alepis, E. (2021). Decision tree learning through a predictive model for student
academic performance in intelligent m-learning environments. Computers and Education:
Artificial Intelligence, 2, 100035.
Milrad, M., Wong, L.-H., Sharples, M., Hwang, G.-J., Looi, C.-K., & Ogata, H. (2013). Seamlesslearning: An international perspective on next generation technology enhanced learning. In Z.
L. Berge & L. Y. Muilenburg (Eds.), The Handbook of Mobile Learning (pp. 95-108).
Routledge.
Ng, W., & Nicholas, H. (2007). Ubiquitous learning with handhelds in schools. International
Conference on Mobile Learning 2007, Melbourne, Australia.
Obisat, F., & Hattab, E. (2009). A proposed model for individualized learning through mobile
technologies. Computers, 1(3), 125-132.
Ocheja, P., Flanagan, B., Ueda, H., & Ogata, H. (2019). Managing lifelong learning records throughblockchain. Research and Practice in Technology Enhanced Learning, 14(1), 1-19.
Ogata, H., Li, M., Hou, B., Uosaki, N., El-Bishouty, M.M., Yano, Y., (2011). SCROLL: Supporting toShare and Reuse Ubiquitous Learning Log in the Context of Language Learning, International
Journal of Research and Practice on Technology Enhanced Learning (RPTEL), Vol.6, No.2,
pp.69-82.
Ogata, H., Majumdar, R., Yang, S.J. H. and Warriem, J.M., (2022). LEAF (Learning & Evidence
Analytics Framework): Research and Practice in International Collaboration, Information andTechnology in Education and Learning, 2(1).
Ogata, H., Majumdar, R., Flanagan, B., and Kuromiya, H., (2023). Learning Analytics and Evidencebased K12 Education in Japan:Usage of Data-driven Services for Mobile Learning AcrossTwo Years., International Journal of Mobile Learning and Organisation, 1(1):1.
Pishtari, G., Rodríguez‐Triana, M. J., Sarmiento‐Márquez, E. M., Pérez‐Sanagustín, M., Ruiz‐Calleja,
A., Santos, P., Prieto, L., Serrano-Iglesias, S., & Väljataga, T. (2020). Learning design andlearning analytics in mobile and ubiquitous learning: A systematic review. British Journal of
Educational Technology, 51, 1078–1100.
Selwyn, N., & Aagaard, J. (2021). Banning mobile phones from classrooms—An opportunity toadvance understandings of technology addiction, distraction and cyberbullying. British Journalof Educational Technology, 52(1), 8-19.
Sha, L. (2015). Self-regulation: A critical learner characteristic for seamless learning as habit of mind.
In L.-H. Wong, M. Milrad, & M. Specht (Eds.), Seamless Learning in the Age of Mobile
Connectivity (pp. 91-108). Springer.
Sharples, M., McAndrew, P., Weller, M., Ferguson, R., FitzGerald, E., Hirst, T., Mor, Y., Gaved, M., &Whitelock, D. (2012). Innovating Pedagogy 2012.
Suárez, Á., Specht, M., Prinsen, F. R., Kalz, M., & Ternier, S. (2018). A review of the types of mobile
activities in mobile inquiry-based learning. Computers & Education, 118, 38-55.
Sun, D., Looi, C.-K., Yang, Y., & Sun, J. (2020). Design and implement boundary activity basedlearning (BABL) principle in science inquiry: An exploratory study. Educational Technology &Society, 23(4), 147-162.
Wong, L.-H. (2012). A learner-centric view of mobile seamless learning. British Journal of Educational
Technology, 43(1), E19-E23.
Wong, L.-H., Chai, C. S., & Aw, G. P. (2017). Seamless Language Learning: Second Language
Learning with Social Media. Comunicar, 25(50), 9-21.
Wong, L.-H., & Looi, C.-K. (2011). What seams do we remove in mobile assisted seamless learning?
A critical review of the literature. Computers & Education, 57(4), 2364-2381.
Wong, L.-H., & Looi, C.-K. (2022). Seamless hybrid science learning: Streamlining the technopedagogical designs for wider diffusion. In E. Gil, Y. Mor, Y. Dimitriadis, & C. Köppe (Eds.),
Hybrid Learning Spaces. Understanding Teaching-Learning Practice. Springer.
Wong, L.-H., Milrad, M., & Specht, M. (Eds.). (2015). Seamless Learning in the Age of Mobile
Connectivity. Springer.

1025

Yazici, İ., Shayea, I., & Din, J. (2023). A survey of applications of artificial intelligence and machinelearning in future mobile networks-enabled systems. Engineering Science and Technology,
an International Journal, 44, 101455.
Zubanova, S., Didenko, E., & Karabulatova, I. (2021). Location-based mobile learning systemfacilitating English learning. Interactive Learning Environments, Published online: 29 Sep
2021.

1026