9

ISSN 2188-8612

JACET International Convention
Selected Papers
Volume 9

The JACET 61st International Convention
Redesigning of Language and Culture Education in the Global Process of Digital Transformation

August 24 – 26, 2022
OnlineJACET International Convention Selected Papers, Volume 9
Published by the Japan Association of College English Teachers (JACET)JACET Selected Papers Committee
Senior Director-in-Charge
KAWANO, Madoka (Meiji University)

Director-in-Charge
LEIS, Adrian (Miyagi University of Education)

Editorial Board
Chair
ITO, Kenichi (University of Kitakyushu)

Deputy ChairHAENOUCHI, Hiroko (Nihon University)

FURUMURA, Yumiko (Nagoya University of Foreign
Studies)

University)
MUSTY, Nicholas (Kobe Gakuin University)

HAYASHI, Hideo (Kumamoto Gakuen University)

NALL, Matthew (Miyagi University)

HIRAMOTO, Satoshi (Yasuda Women’s University)

OHSAKI, Satsuki (Soka University)

KANAMARU, Toshiyuki (Kyoto University)

OKADO, Hiroko (Meijo University)

MATSUMOTO, Hiroyuki (Hokkai-Gakuen

YONEOKA, Judy (Kumamoto Gakuen University)

Advisory Board
Deborah Healey (University of Oregon, Emerita)
Wei Keong Too (University of Nottingham Malaysia)
Hiroaki Ogata (Kyoto University)

JACET Prize & Academic Publication Selection Committee
Senior Director-in-Charge, HeadODA, Masaki (Tamagawa University)

Director-in-Charge, Vice-Head
SATO, Takehiro (Nagoya University of Foreign Studies)ii

Reviewers
AKIYAMA, Tomoyasu (Bunkyo University)

OTOSHI, Junko (Okayama University)

ALLEN-TAMAI, Mitsue (Aoyama Gakuin University)

OZEKI, Shuji (Nagoya University)

BABA, Chiaki (Teikyo University of Science)

SAKAI, Hideki (Shinshu University)

DALSKY, David (Kyoto University)

SAKATA, Naoki (Kumamoto Gakuen University)

HIJIKATA, Yuko (University of Tsukuba)

SASAJIMA, Shigeru

JOEL C, Meniado (Regional Language Centre (RELC),SASAO, Yosuke (Kyoto University)

Singapore)

SHIMIZU, Haruka (Tohoku Gakuin University)

KAWANARI, Mika (Meikai University)

YANAGI, Yoshikazu (Nagoya Gakuin University)

KOJIMA, Masumi (Gifu City Women’s College)
OI, Kyoko (Chiba University (Professor Emerita))

YOKOKAWA, Hirokazu (Kobe University)

Cover Design
KASUNO, Shin-ichi
Editorial Office
The JACET Office
Address: 55 Yokotera-machi, Shinjuku-ku, Tokyo 162-0831 JAPAN
Phone: +81-3-3268-9686/ Fax: +81-3-3268-9695

E-mail: jacet@zb3.so-net.ne.jp

First published in March 2023
Copyright © 2023 by JACET
All rights reserved. No part of this publication may be reproduced or republished in any form without
permission in writing from JACET.
The articles published herein do not reflect the opinions of JACET.

iii

Contents

Invited Papers*
Designing Culturally Responsive Teaching With Technology···········································Deborah Healey & Associates

3

Towards Data and Evidence-driven Education in the Context of Language Teachingand Learning···················································· Hiroaki Ogata
Huiyong Li
Rwitajit Majumdar
Yuko Toyokawa
Kensuke Takii
Changhao Liang 15

Selected Papers
Practitioner Report

The Horizon Report: What can we Learn About Educational Technology Trends?
···························································· Neil Cowie
Keiko Sakui 49
The Impact of a Curriculum-based Formative Assessment on EGAP···························································· Ryan Spring
Shizuka Sakurai 65

Submission Guidelines ················································································· 86

*Plenary Lectures
Wei Keong Too. Reorganising Priorities for TESOL Teacher Education in the Post-COVID Era – Goals,
Environment and Technology(August 24)

Deborah Healey. Designing culturally-responsive teaching with technology(August 25)
Hiroaki Ogata. Towards data and evidence-informed teaching and learning in the context oflanguage learning(August 26)

Invited Papers

JACET Selected Papers Vol. 9 (2023) 3-14

Designing Culturally Responsive Teaching With Technology

Deborah Healey & Associates
University of Oregon, Emerita

Abstract
An extensive body of research has indicated that learning environments and materials need totake into account learner backgrounds and differences for best results. One helpful approach
to incorporating learner backgrounds into instruction is “culturally responsive teaching.”
Culturally responsive teaching makes learner needs and identity central to classroom material
and activities. Universal Design for Learning (UDL) (CAST, 2018; Burgstahler, 2021) is
another useful element. Designing with learners’ cultural, physical, and intellectual
differences in mind before the class starts, not just adapting while teaching, is quite effective.
Technology fits into both culturally-responsive teaching and UDL. Appropriate use oftechnology enables differentiated learning in a variety of ways. Technology tools, chosen
thoughtfully, can also help teachers design inclusive, culturally responsive teaching withtechnology and better, more meaningful learning for students.
Keywords: culturally-responsive teaching, universal design for learning, differentiation,
curriculum, technology

Teachers want the best for their learners. An extensive body of research has indicatedthat learning environments and materials need to take into account learner backgrounds anddifferences for best results. Thoughtful and appropriate use of technology also enables
differentiated learning in a variety of ways and leads to good outcomes. One helpful approach
to incorporating learner backgrounds into instruction, discussed here, is “culturally responsive

3Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

teaching.” Culturally responsive teaching makes learner needs and identity central toclassroom material and activities. According to Gay (2002),
Culturally responsive teaching is defined as using the cultural characteristics,
experiences, and perspectives of ethnically diverse students as conduits forteaching them more effectively. It is based on the assumption that when
academic knowledge and skills are situated within the lived experiences andframes of reference of students, they are more personally meaningful, havehigher interest appeal, and are learned more easily and thoroughly (Gay, 2002,
p. 106).
What this means is that learning is more memorable when students feel people who look
like them, with their backgrounds and life experiences, are included at least some of the timein what learners are doing in the classroom.
Technology use, the other theme of this article, encompasses a great deal in teaching.
Word-processors, interactive websites, social media, learning management systems, virtualand augmented reality, artificial intelligence, and more fall into this category. Teachers arepressed to know their learners, design effective environments for learning, provide formativeand summative assessment, encourage learners to create digitally, and of course meet learning
objectives in their curriculum. With Universal Design for Learning (CAST, 2018;
Burgstahler, 2021), teachers, curriculum designers, and technology specialists can beproactive in designing learning environments to meet the needs of all learners. StapletonCorcoran (2022, What section) provides a definition:
Universal Design for Learning (UDL) is an educational framework forcreating learning environments that address the diverse needs of learners. At
its core, UDL provides students flexibility in the ways they access and engage
with course materials and demonstrate mastery of learning objectives.

4JACET Selected Papers Vol. 9 (2023) 3-14

Designing with learners’ cultural, physical, and intellectual differences in mind before
the class starts, not just adapting while teaching, is the most effective approach. Even thoughteachers are often not included in curriculum decisions, teachers can examine the curriculumthat they are given for options to be more inclusive and find ways to plan ahead. Technology
tools, chosen thoughtfully, can also help add culturally responsive universal design elements.
Technology can add the flexibility and choice called for in UDL in a variety of ways.
(The reference and resource list at the end provides some specific examples.) Multiple modesof learning can come through various media. Learners can be writing, recording on a phone,listening, and more. Including audio with text and text with graphics and audio/video help avariety of learners, including our multilingual language learners (MLLs).1
Different learners are ready at different times, so offering multiple places to start andmultiple modes helps everyone. Teachers found during lockdown that having recorded
lectures allowed all learners to go back and review what they didn’t understand. With closed
captioning, video became reading and listening, offering multiple modes and channels forlearning. Different learners prefer different media, and teachers can let them build from theirstrengths whenever possible. Pace is a similar consideration. Some learners can go faster, andteachers should encourage them by giving them more to do. Others need more scaffolding to
get to the same place. Well-designed learning management systems help teachers track wherelearners are, as well as offering learners access to additional resources for enrichment andsupport.
Alternative assessment lets teachers have a better idea of what learners can actually do.
It can and should be included systematically as part of design. A wide variety of alternative
assessments are available; the choice needs to fit the learning objectives and then allow
learners to reflect their own contexts. Here are a few examples. A portfolio lets learners
highlight what they do well and see progress over time. All ages and levels can use portfolios,5Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

and they can be digital or physical. Digital portfolios are more flexible in many ways, as longas there is adequate technical support. Self record-keeping and reflection (which can be partof portfolios or separate from them) work well for tweens, teens, and adults. They can think
about their own work and be responsible for recording it.
Peer assessment can be done by all ages and levels, but it will look quite different,
depending on the context. Young learners can talk about whether they liked someone else’s
work. Older learners can use a rubric to give a more detailed assessment. The whole class isinvolved, as in Figure 1, with learners looking and talking. Peer assessment goes both ways;
teachers assess the peer giving the assessment as well as the student that the peer is assessing.
Peer interview is a way for the teacher to observe two (or three) learners and how theyinteract. Having a rubric in hand in paper or digital form lets teachers track and assess easily.
Figure 1
Students Doing Peer Assessment

Many alternative assessments are performance-based. For teens and adults, performancecan include concept mapping, writing (summary, report, blog, social media, comics),
presentations, video-making, role-plays, and project-based assignments such as WebQuests.
With young learners, performance is more likely to include identifying, drawing or painting,6

JACET Selected Papers Vol. 9 (2023) 3-14

movement such as dancing and singing, role-playing, creative writing (stories and comics),
and short reports. The objects that learners produce can be added to their portfolios as well.
Culturally-responsive teaching starts with whether the assessments fit the learning
objectives and reflect learners’ contexts. Teachers should be considering how to allow
learners to bring their own interests and contexts into what they are asked to do. The projects
and presentations can showcase learners’ homes, families, cultural background, and the like.
When young learners are identifying, drawing, doing role plays, etc., they should be working
with images, songs, and more that reflect them.
To offer a few specific tech-focused examples, one type of performance-based
assessment is a concept map. This is a sample template from GitMind, which offers a numberof concept mapping templates (Figure 2).
Figure 2
Concept map from GitMind: Essay Writing Process template

Learners can use paper templates or just draw their own. Digital tools can be more helpful
with intermediate and advanced learners or those who like digital tools. GitMind, Miro, andBubbl.us are just a few of the many digital concept mapping tools that can let learners

7Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

demonstrate their vocabulary, content knowledge, and grammar. With any of these, group
work can be used to demonstrate collaboration.
An interesting story creator geared toward young learners is WriteReader. This is a very
simple, easy-to-use way to create a book with photos and recorded audio. It includes both freeand paid versions. Figure 3 shows several templates: All about me, Animal, My feelings,
Fairy tales, My favorite game, Learning journal, A lucky day, School year memories, My
sport, Wild animals, Alphabet book, and My learning.
Figure 3
WriteReader Templates

The class could do much the same thing with Word or Google Docs, but this makes it
look more like a book when the learner is finished, and learners can share with classmates andfamily easily.
Young learners, teens, and adults can all enjoy creating comics. Again, this can be doneby hand, but young learners and those with less artistic ability may benefit from good-looking
ready-made graphics. Learners supply the language in the form of text in bubbles or as

8

JACET Selected Papers Vol. 9 (2023) 3-14

captions. Working with a partner also encourages language production as learners negotiate
decision-making. One example of a comic strip creator is Make Beliefs Comix (see Figure 4).
Learners choose from the images in the app for the background, characters, and different
props, but they can arrange the characters and props in different ways, and then add their own
speech bubbles to create the story. Learners can’t create their own graphics, which limits how
representative this can be, but there are quite a few choices.
Figure 4Some Elements in Make Beliefs Comix

In addition to people with a variety of ethnicities and costumes, learners can choose
animal or creature characters (Figure 5).
Figure 5
Sample Characters From Make Beliefs Comix

Producing comics is a very fun approach to writing, and it allows learners to express
themselves creatively. Tasks can be chosen to encourage learners to write about themselves
and their own backgrounds.
WebQuests are a way of organizing project-based learning. A WebQuest doesn’t need
to be online; this can be an organizing scheme, with the elements printed out. In the Figure 6

9

Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

template, the elements are home, introduction, task, process, resources, evaluation,
conclusion, credits, and sitemap. Each element has its own page:

The Task page includes a description by the teacher of what learners are expected to
do and the project they are working on. The end result should be something tangible.

On the Process page, the teacher describes how the work is to be done, such as the roleof each team member and the process, step by step. More linguistically advanced orolder learners can help set roles themselves for their team.



The Resources page generally includes at least some resources/links provided by theteacher. Lower-proficiency learners should find almost all that they need here rather
than trying to do a web search on their own.

The Evaluation should be clearly explained from the beginning. Project-based learningoften has one or more rubrics to use in evaluating different elements of the projects.
Teamwork may be one assessment; the final project(s) would have other rubrics.

Figure 6
WebQuest Elements

Project-based learning appeals in particular to students who want to do something
useful and make a difference. Learners collaborate to produce a tangible product that goesoutside the classroom, often in multimedia form. Using project-based learning takes someplanning and allows teachers to consider learner strengths and where they need more support.

10

JACET Selected Papers Vol. 9 (2023) 3-14

Learners can do a project relevant to them and their community. This is wonderful for both
universal design for learning and culturally responsive teaching.
This article has been a brief overview of the ways that culturally responsive teaching
and universal design for learning help teachers create inclusive, vibrant classrooms wherelearners can see themselves in the material and activities. With universal design for learning,teachers can plan ahead and use technology in the most appropriate way. The result is better,more meaningful learning for students.
For more information, explore CAST’s UDL Guidelines (2018). Each element is
described in detail: how to provide multiple means of engagement to produce learners whoare purposeful and motivated; how to provide multiple means of representation, to produce
learners who are resourceful and knowledgeable; and how to provide multiple means ofaction and expression, to produce learners who are strategic and goal-directed. Additional
helpful ideas and resources are on the CAST website, which is in the reference and resources
list below.
Footnotes
1. I use multilingual language learners or MLLs in this paper to be as inclusive as possible.
MLLs include those learning English as an additional language inside or outside an
English-speaking environment. They are learning English for their own varied purposeswith the goal of adding to their linguistic repertoire.
References
Burgstahler, S. (2021). Universal design: Process, principles, and applications.
https://www.washington.edu/doit/universal-design-process-principles-and-applications
CAST (2018). Universal design for learning guidelines version 2.2.
https://udlguidelines.cast.org

11

Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

Center for Universal Design in Education, University of Washington. (2022). Introduction to
universal design in education. https://www.washington.edu/doit/programs/centeruniversal-design-education/introduction-universal-design-education
Gay, G. (2002). Preparing for culturally responsive teaching. Journal of Teacher Education,
53(2), 106–116.
Stapleton-Corcoran, E. (2022). Universal design for learning (UDL).
https://teaching.uic.edu/resources/teaching-guides/inclusive-equity-minded-teachingpractices/universal-design-for-learning-udl/
West Virginia Department of Education. (n.d.). 7. UDL guidelines checklist.
https://wvde.state.wv.us/osp/UDL/7.%20UDL%20Guidelines%20Checklist.pdf
Technology tools
General Google. (n.d.). Accessibility: Products and features.
https://www.google.com/accessibility/products-features/ - extensive list of accessibility
features within Google products.
 Microsoft.com. (n.d.) Accessibility products. https://www.microsoft.com/enus/accessibility/ - accessibility features within Microsoft products.
 Road to Grammar (n.d.). Text analyzer. http://www.roadtogrammar.com/textanalysis/ free tool that determines the CEFR level of text.
 WebAIM. (n.d.). WAVE web accessibility evaluation tool. https://wave.webaim.org/ checks web pages for accessibility.
E-portfolios
 WordPress – https://www.wpexplorer.com/online-portfolio-wordpress/ and video athttps://youtu.be/ne5Obuwg7oM for help

12

JACET Selected Papers Vol. 9 (2023) 3-14

 Google Sites with Google Drive –how-to explanation at https://www.uwb.edu/it/trainingand-support/teaching-learning/eportfolios/new-google-sites; helpful video athttps://www.youtube.com/watch?v=EX7EfkcXyAQ Moodle or Blackboard component – ask your administrator
Concept maps (mind mapping, constellations) GitMind - https://gitmind.com/ - free registration for access to downloaded, mobile, andonline app; more education examples and templates than many others listed here Canva – https://www.canva.com – many templates, including concept maps; powerful andcomplex app; free registration
 Bubbl.us – https://bubbl.us – three concept maps for free. Relatively easy to use.
 Templates from Template Hub - https://www.templatehub.org/concept-map-templates6870.html - variety of concept map templates free to download
Blogs
 Blogger – https://www.blogger.com – easy to use, free blog site; part of Google Edublogs - https://edublogs.org/ - free; designed for educational use. No ads.
 WordPress – https://www.wordpress.com - website creator includes blogs; free basic site
Story/book and Comic Creation for Learners
 MakeBeliefsComix – https://makebeliefscomix.com – free comic strip creator (up to 18
cells); register to save. Free to print.
 Storyjumper – https://www.storyjumper.com – free to create an online book; pay to turn it
into PDF or print it.
 WriteReader – https://app.writereader.com – free basic app; register to use
Video Tools
 Flip (formerly Flipgrid) – https://flip.com - free, widely used for remote learning; easy tocreate a classroom video sharing site (Microsoft product)

13

Deborah Healey & Associates
Designing Culturally Responsive Teaching With Technology

 iMovie – https://www.apple.com/imovie/ - available for Apple’s iOS; works well on iPad
 Microsoft Photos – part of Windows 10 and later; creates movies. Windows 8 and earlier
can use MovieMaker; MovieMaker does not work on later versions.
 Screencast-o-matic – https://screencast-o-matic.com – screen recorder; 15 minutes for freeor get a paid plan for longer recordings.
WebQuest Tools
 Google Doc Webquest Template:
https://docs.google.com/document/d/1tIVITnkuSfl0JYzj9SZXkMM3Rk5Jj108JMiHC5eFzgw- helpful, step-by-step approach QuestGarden: http://questgarden.com/ - search for examples (free search); subscriptionbased to create webquests on their site
 WebQuest.org: http://webquest.org/ - the original site
 Zunal.com: https://zunal.com - you can make a few webquests there for free
Rubrics
 Rubistar: http://rubistar.4teachers.org/ - free rubric templates with suggested descriptors

14

JACET Selected Papers Vol.9 (2023) 15-46

Towards Data and Evidence-driven Education in the Context ofLanguage Teaching and Learning

Hiroaki Ogata
Huiyong Li
Rwitajit Majumdar
Yuko Toyokawa
Kensuke Takii
Changhao Liang
Kyoto University

Abstract
Language teaching has a rich research history. A research discipline of Computer-Assisted
Language Learning (CALL) has focused on technology integration in that practice. However,integrating the learning logs to create a data-driven workflow for the teachers and students isstill limited. We design a technology framework called LEAF (Learning and Evidence
Analytics Framework) to integrate daily evidence-based educational practices. The datadriven learning tools integrated into LEAF were implemented in actual live classrooms acrossmultiple universities and schools within Japan. In this paper, we discussed the teaching andlearning practices with the LEAF system in a language learning context and its impact. We
highlight how to use LEAF for active reading, recommendation-based vocabulary learning,self-directed language learning approaches, and group work for language learning andteaching, specifically in English classes. Moving ahead, we aim to have an evidence-driven
approach where the technology can continuously support and update best practices byanalyzing the log data gathered continuously from the real-world educational setting.
Keywords: LEAF, data-driven service, evidence-driven education, language learning
15

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

16

Nowadays, there is a rapid adoption of educational technology to enhance teaching andlearning in regular educational settings and emergency remote learning due to the COVID-19
pandemic. Many countries and institutions already had resourced infrastructures to speed up
the transformation from the conventional educational approach to technology-enhanced
educational innovation. For instance, in Japan, the Ministry of Education, Culture, Sports,
Science and Technology (MEXT) introduced the Global and Innovation Gateway for All(GIGA) School Program to transform learning using an ICT environment and one device per
student (MEXT, 2020). The program rolled out nearly 461 billion Japanese Yen over the
fiscal year of 2020 and 2021 to build the capacity. Across the globe, many countries havetaken the e-learning infrastructure to the next level to integrate learning analytics to informand support their stakeholder’s decision-making. Some studies report those implementationsin the educational context of the United States (e.g., Krumm et al., 2018), Spain (e.g.,
Martínez Monés et al., 2020), Bulgaria (e.g., Gaftandzhieva et al., 2021), and Uruguay
recently (Queiroga et al., 2022). Globally it is seen that e-learning infrastructure is necessary,
however, not a sufficient condition for effective teaching-learning at an institution level andnationwide level.
Generally, e-learning infrastructure across various institutions often includes a Learning
Management System (LMS), various ubiquitous classroom learning tools, Learning Record
Stores (LRS), and Learning Analytics Dashboards (LAD). Such an infrastructure can apply
Learning Analytics (LA) methods to process log data and support various stakeholders.
Teachers can refine their instructional practices, learners can enhance learning experiences
and researchers can study the dynamics of the teaching-learning process with it. While LA
platforms gather and analyze the data, there is a lack of a specific framework to capture thetechnology-enhanced teaching-learning practices and promote evidence-driven education
across different institutions and countries.

16

JACET Selected Papers Vol.9 (2023) 15-46

This paper focuses the research agenda on evidence in a data-driven educational
scenario. We propose the Learning Evidence Analytics Framework (LEAF) and present the
research challenges. Four data-driven services with LEAF are introduced for language
teaching and learning. Moving ahead, we aim to have an evidence-driven approach where thetechnology can continuously support and update best practices by analyzing the log data innear real-time.
Learning and Evidence Analytics Framework (LEAF):
An Infrastructure for Data-driven Education
LEAF is the overall technical framework that seamlessly integrates research andproduction systems to enable educational data science research as well as AI-driven services
for the users (Ogata et al., 2018, 2022, in press). The stakeholders include the institutes'
students, teachers, researchers, and course coordinators. This paper presents the componentsof LEAF and illustrates some of the teaching-learning practices that the framework supports
toward data and evidence-driven language education. Figure 1 shows the various componentsof the framework.
A Learning Management System (LMS) is an online system that provides anenvironment to structure courses and maintains user roles such as teachers, students, andcourse administrators to grant access to different course materials and activities shared in
LMS. Moodle or Sakai are examples of such LMS that are already common at various
educational institutes in Japan and abroad. LEAF integrates various e-learning tools withLMS using standardized Learning Tools Interoperability (LTI) protocol for a seamlessauthentication transition from an existing LMS in an institution using an anonymized unique
ID. For instance, BookRoll (Ogata et al., 2015) is an E-book reader that records users reading
and annotation behaviors. The interactions are logged using Experience API (xAPI), which isan open-source statement API for outputting anonymized event logs to a centralized17

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

18

independent Learning Record Store (LRS). The data-driven services are built based on theselearning logs.

Figure 1
Learning and Evidence Analytics Framework (LEAF) and Its Components

BookRoll serves as a learning material distribution platform and captures the readinginteraction logs used for students’ LA in content and behavior. Teachers can upload learningmaterials to BookRoll in PDF format, while students can access them from a wide range of
devices through a standard web browser. Figure 2 presents the user interface of the BookRoll
E-book reader. BookRoll tracks reading interactions such as navigating the reading content(going to the next or previous page, or jumping to different pages) and annotating parts of thelearning materials that are hard to understand or important with memo comments, bookmarks,
and markers. Other features of BookRoll include a smart dictionary that allows readers to lookup the meaning of any highlighted word and can be used for vocabulary learning activities. A
hand-written memo function allows users to make strokes using a digital pen or a touchenabled mobile device directly. A recommender and quiz panel allows the recommendation

18

JACET Selected Papers Vol.9 (2023) 15-46

engine to share the link to external resources and lets teachers create a reflective quiz forformative evaluation.

Figure 2
User Interface of BookRoll E-Book Reader

Log Palette is a learning analytics dashboard in LEAF, as shown in Figure 3 (Ogata etal., 2022). Log Palette analyzes the trace log in BookRoll via LRS and visualizes the results inthe dashboard in near real-time. As with BookRoll, Log Pallet can be added as an external
tool to an LMS by LTI protocol. LTI protocol automatically handles the user roles anddisplays different panels of graphs with customizable views for teachers and students. The
analysis results about learning material, individual engagement, and group engagement are
displayed in three graphical panels in the dashboard. Firstly, a Page panel provides
summarized and detailed information regarding learners’ frequency of learning trace and a listof learning content from memos or maker annotations by pages as shown in Figure 3.
Secondly, a Student panel gives individual statistics regarding learning engagement in the19Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning20

selected learning material, such as achievement rates of reading, reading time, interaction
frequency, markers and memos count. Thirdly, an Activity panel provides aggregated statistics
about group engagement in the selected learning material. In this panel, both teachers andstudents can see the average statistics of learning engagement in the selected material andlearning performance in the quiz and compare them with the selected student.

Figure 3
User Interface of Log Palette Learning Analytics Dashboard

Advanced data-driven tools and micro-services are linked in the Log Palette dashboard
in Figure 4. The tools and micro-services enable teachers and students to integrate other data
beyond BookRoll reading logs, such as uploading offline test scores, their perception survey
responses, and so forth. The tools and micro-services can utilize reading and other data tocreate an open learner model in LEAF. They can broadly be grouped into three categories:
20

JACET Selected Papers Vol.9 (2023) 15-46

tools to conduct classroom activities using trace data easily, tools for efficient learning andreflecting on learning, tools for data exchange and utilization monitoring. For instance, agroup formation tool linked to LEAF uses the learner model to form heterogenous or
homogenous groups for collaborative work (Liang, Majumdar, & Ogata, 2021). Also, the data
about content interaction and performance on a specific topic can be used to create knowledge
maps of a particular topic of a cohort of learners (Flanagan et al., 2021).
Thus, BookRoll and Log Palette prepare the foundation for collecting, analyzing andsharing learning logs to support teachers and students in their daily teaching and learning
context.

Figure 4
Advanced Data-driven Tools Linked in Log Palette

Learning Analytics Cycle With LEAF for Language Teaching and Learning
The LEAF infrastructure is designed and developed to enrich teaching-learninginteractions with various data-driven tools and services. It can be utilized by teachers from21

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

22

different subject domains to promote the engagement between teachers, students and learning
contents and improve teaching and students’ learning in an offline, online, or flipped mode. A
typical learning analytics cycle is shown for language teaching and learning using the LEAF
system (see Figure 5). Importantly, such a LA cycle can be adopted flexibly by teachers toorchestrate classes in a face-to-face mode (Chen et al., 2022), online mode (Yang et al.,2022), flipped mode (Toyokawa et al., 2021a) or during remote teaching due to the COVID19 pandemic (Majumdar, Flanagan, & Ogata, 2021).

Figure 5
Advanced Data-driven Tools Linked in Log Palette

Facilitating Language Teaching and Learning Using Data-driven Tools in LEAF
Given the various data-driven tools in the LEAF infrastructure, we will demonstrate thecapabilities of those LEAF services to facilitate teaching-learning activities, especially in thecontext of language teaching and learning in Japan. Four data-driven tools and their empiricalstudies in English teaching and learning class will be introduced: digitally enhanced active
22

JACET Selected Papers Vol.9 (2023) 15-46

reading, online self-directed extensive reading, AI recommendation for language learning, andreading-based online group work.
Digitally Enhanced Active Reading With LEAF
Introducing appropriate reading strategies would assist students in acquiring reading
skills, improve reading comprehension, and eventually lead to their motivation and autonomy
toward reading. Active Reading (AR) is a strategy that can foster and develop learners'
reading performance and skills. It activates the reader's prior knowledge and experience
gained in the past and relates them to new information to facilitate understanding of thecontent of the written text (Ogle, 1986; Spivery, 1987). One well-known strategy is SQ4R
(Survey, Question, Read, Record, Recite, and Review) (Khusniyah & Lustyantie, 2017). We
could digitally enhance SQ4R using the affordance of the LEAF system, enabling students toperform AR individually and cooperatively during group work. It is orchestrated as follows:
predicting the content by skimming and scanning through such as the title and pictures in the
text (Survey), asking questions to self about the content (Question), highlighting unknown
words and main ideas, leaving annotations and questions in a memo (Question and Record),
confirming their understanding with discussing with peers or in a group (Recite), and writing
a summary of the story and working on extra review activities (Review). Further, in order tofurther strengthen reading skills, comprehension, and learners' independence, it is necessary togive learners an opportunity to reflect on their learning and lead to the next step. The Active
Reading Dashboard (AR-D) in Log Palette presents a visualization of the reading behaviors oflearners for them and the facilitator to reflect on. Figure 6 shows an overview of the digitallyenhanced AR workflow using the LEAF system components. Teachers can decide on
individual or group AR sessions in academic reading classes as ones often conducted in thehigher education level. With LEAF, such sessions can be more self-directed by the students,
and teachers can evaluate effectiveness using the reading behavior data that is logged.

23

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

24

Figure 6
The Flow of Digitally-enhanced Active Reading With LEAF

Since 2020 we have conducted several AR experiments using LEAF to investigate theeffects of improving learners' reading comprehension and scaffolding teachers' teachingpractice. These AR strategies were implemented across learning contexts of individual andgroup levels, face-to-face and online and in self-directed learning sessions. In this chapter, weshall introduce two studies: Jigsaw AR (Toyokawa et al., 2021b), a collaborative learning
study conducted in a public high school during their English class, and Flipped online AR(Toyokawa et al., 2021a), an individual activity conducted as part of a university reading
course.
Jigsaw AR: In Class Cooperative AR Activity With LEAFThe effects of AR with LEAF in English classes have mainly been measured with the
integration of AR class with collaborative group learning. An example is an integrated AR
with Jigsaw group learning. Through jigsaw activities, students work on their part of a
subject, cooperatively assemble the parts to fill in missing information, and complete the tasks
24

JACET Selected Papers Vol.9 (2023) 15-46

together (Aronson, 1978). The existing jigsaw methods neither explored much technology
affordances to conduct the activity nor its effectiveness has not been verified by using log data
while using the technology platform. Most of the preparation, activities, tests, and evaluations
manually invest teachers' time and effort. Therefore, we proposed JigsawAR as a new versionof jigsaw learning which incorporates AR strategies and is modified to the current form toorchestrate the class activities using LEAF. Using the logged data further provides
opportunities to utilize it for teaching activities and evaluate effectiveness.
Participants of the Jigsaw AR were high school students who belonged to the Integrated
English Competency (IEC) classes in Japan. There are two levels for the classes: standard andadvanced, whose classes were divided based on their previous academic records. Jigsaw AR
activities were conducted over two days. The students were divided into groups on the firstday and conducted their individual reading portion in the pre-jigsaw phase. The next day they
conducted the rest of the Jigsaw phases. Each class was 50 minutes long. In total, 62 students
were registered in the course (advanced = 36 and standard = 26). Figure 7 shows the Jigsaw
AR procedure as an example used for individual and group reading activities using BookRoll.

Figure 7
Jigsaw Active Reading With LEAF

25

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

26

Facilitating Reading Skills and Learning Engagement Through Jigsaw AR
As for Jigsaw AR using LEAF, first, Jigsaw groups were formed using a groupformation system based on the logs of students’ pre-quiz scores, and vocabulary quizzes weremade based on the logs from marker use and DicoDico dictionary use in BooRoll.
Through Jigsaw AR, students could work individually and with their peers in a group to
understand the contents of the reading and assemble the flow of the story together. As a result,the vocabulary quiz scores improved significantly from pre to post-quiz (p < .001).
Differences in annotation behavior among students at different levels were also observed inlogs. Based on the findings of the present study, it was concluded that data-driven Jigsaw AR
design played a positive role in enhancing English skills in terms of vocabulary and reading
comprehension and facilitating students’ engagement in English classes.
Flipped Online AR With LEAF
AR using LEAF not only can be applied to individual and group activities in class but
also be used for self-regulated reading activities outside of class and online classes. For
investigating AR learning outside of class, we conducted AR experiments in flipped learning.
University freshmen (n = 16) in a basic AR class learned through flipped online AR over asemester, and student learning engagement and the relationship between their grades and theirlearning attempt were examined. Students were asked to conduct a reading task as flipped
learning using BookRoll. The online-class activities were basically based on what was left in
the logs from students’ flipped assignment attempts. Students used the dashboards to checktheir own and their classmates' logs during and outside of class for reflection on learning, andteachers made decisions about the next action based on the logs. The teacher’s role for theflipped assignments was to upload quizzes and learning materials on BookRoll, provided
feedback to students, and assembled online-class activities based on their assignment

26

JACET Selected Papers Vol.9 (2023) 15-46

outcomes. For the online classes, the teacher’s role was to assist students as a facilitator:
encouraging students to participate in activities, facilitating and monitoring breakout-room
activities by checking students’ engagement with the analysis tool available in near real-time,and dealing with questions from students and visiting each room if needed.
Improving Reading Participation and Comprehension by Incorporating Logs Into AR
Activities
From the implementation of online flipped AR with LEAF, it was found that there wererelations between freshman English learners’ learning achievement and their flipped AR
performance. For example, students who got lower scores on pre-quizzes and spent less timeon their homework got lower scores on the midterm test. Logs from BookRoll provided
students’ reading interactions and engagements during the flipped assignments (for example,
the maximum length of time spent on BookRoll was 75 minutes per day and the average was
20.56 minutes), and gave the teacher opportunities to decide, conduct and monitor theactivities during the synchronous online phase. In digitally enhanced AR research using
LEAF, it was found that learning logs can be used to reflect on AR learning, and byincorporating logs into AR activities, learners' reading comprehension and participation levels
are improved.
Online Self-Directed Extensive Reading With LEAF
Self-directed learning (SDL) is the basis of lifelong learning and it has been identified
as an increasingly important skill in the 21st century (Brockett & Hiemstra, 2018). SDL canbe defined as a process in which learners take their own responsibility to direct one’s learning
to meet personal learning goals (Boyer et al., 2014). SDL requires learners to be active andpurposefully harness a number of skills to maximize their learning. There is a greater need forstudents to develop SDL ability following the shift from teacher-centered traditionalclassrooms to learner-centered approaches with advanced technologies (Toh & Kirschner,27

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

282020). Extensive reading (ER) is a way of language learning through substantial amounts ofreading, chosen and mastered by the learners themselves (Day & Bamford, 1998). ER has
been considered in the areas of EFL (English as a Foreign Language) and ESL (English as a
Second Language) studies as an effective way to promote learners’ language development.
ER has positive effects on general foreign language development, including vocabularyacquisition, reading fluency, reading strategies, and writing performance. Despite thepotential positive effects of ER, not all learners benefit from its support due to individual
differences in SDL ability. SDL ability may influence individual preferences and further
affect academic performance. Students highly require Self-Directed Extensive Reading(SDER) skills to persist in reading effectively and efficiently in the online learning
environment. Considering that students have significant differences in ER autonomy and SDL
ability, it’s essential to support students with personalized technology scaffolding to help
them engage in SDER.
Goal Oriented Active Learning (GOAL) is a data-driven tool in LEAF to support
students’ self-directed learning (SDL) (Li, et al., 2021a; Majumdar et al., 2018; Yang et al.,2022). GOAL automatically synchronizes learners reading activity logs that are captured in
BookRoll e-book reader and stored in the LRS. Combining the theory of SDL and selfregulated learning, GOAL implements the technology-enhanced SDL cycle from a datainformed perspective into five phases. The initial phase of data collection gives learners theinitiative in their learning contexts, followed by the other four phases: data analysis, planning,
execution monitoring and reflection. The GOAL tool can not only provide critical indicators
in a learning activity such as extensive reading for engaging learners in SDL but also provide
computer-based scaffolds that aim to improve learners’ SDL ability in five phases.
Figure 8 shows the online SDER support environment with LEAF. It contains four
components: a learning management system (LMS) such as Moodle, the BookRoll E-book

28

JACET Selected Papers Vol.9 (2023) 15-46

reader for ER, the GOAL tool for SDL, and a learning record store (LRS). The LMS provides
the entrance of all learning resources and a discussion forum for sharing the learning
experience with classmates. Both BookRoll and GOAL can be launched through the LTI
protocol from the LMS. More than 500 E-books are provided for students to execute ER in
BookRoll. The E-books are from level pre-A1 for beginners to level B2 for upperintermediates of the Common European Framework of Reference for Languages. LRS stores
BookRoll learning logs, such as the number of pages they read and the time spent on theactivity. GOAL synchronizes BookRoll learning logs from the LRS automatically. The
learning logs are aggregated into various learning indicators with hourly, daily, and weekly
scales, such as daily reading time spent in ER. Then SDL scaffoldings are implemented andprovided to students in five phases (Data collection - Analysis - Planning - Execution
monitoring - Reflection).

Figure 8
Online Self-Directed Extensive Reading Support Environment With LEAF

29

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

30

Figure 9 shows GOAL's user interfaces for planning, monitoring, and reflection. The“Planning” panel assists students in creating, editing and deleting their reading plans with aplan template. The plan template includes the activity type, plan name, start date, end date,frequency, value for each day, and notes. The “Monitoring” panel contains a visual graph ofreading activity and a self-evaluation form. Learners can track their detailed self-progress andcompare them with the average value of the group, recommended value, or target value. They
can further evaluate their progress status through self-rating and unstructured note-taking. The“Reflection” panel contains the plan details, achievements, and self-reported reflection
journals related to a specific plan. All five panels provide feedback based on the learner’s
SDL skills, measured by the interactions in each panel.

Figure 9User Interface of GOAL for (a) Planning, (b) Monitoring and (c) Reflection

30

JACET Selected Papers Vol.9 (2023) 15-46

Online SDER Support in K-12 Schools in JapanTo evaluate the effects of online SDER support on students’ learning, we carried out
multiple experiments in a combined public junior high and high school in Japan.

Figure 10
Workflow of Self-Directed Extensive Reading in K-12 Settings
Regularly

Self-Directed Extensive
Reading Orientation

Extensive Reading
In BookRoll

Self-Directed Learningin GOAL
GOAL

supER powER

Data
Analysisread more, read fast &understand better with
Self- Directed
Extensive Reading!

Planning

Monitoring

自主多読
実践しやすい + モニタリングしやすい= スキル習得しやすい!

Reflection

The workflow of SDER in K-12 settings is shown in Figure 10. Firstly, students wereintroduced to four key principles in the SDER orientation: read easy materials without
dictionaries, self-select what they want to read, read as much as possible, and takeresponsibility to set their own goals. They were also given instructions on how to selectappropriate e-books, read e-books online, and manage their reading plans at their own pace in
the SDER orientation. Then, students were encouraged to read picture E-books extensively in
BookRoll and manage SDL in five phases in GOAL. For instance, students could select a
fiction e-book to read or create a personal reading plan with a daily target value in reading
pages. They were also suggested to execute the SDER cycle regularly to form good reading
habits. More than 1,000 students have used the online SDER support from 2020 to 2022.

31

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

32

Importance of SDL Ability and Online SDER for Language Learning
Regarding the effects of SDL ability, we found that high SDL ability students had
significantly more reading engagement, SDL behaviors, motivation and autonomy forextensive reading than those with low SDL ability (Li, et al., 2021a). Further, by applying
cluster analysis and transition analysis methods it was possible to differentiate groups oflearners with different planning behaviors during a short-term (1-week reading plan) andlonger-term (1-month reading plan) (Li, et al., 2021b).
Regarding the effects of online SDER support, we found that the students with SDER
support spent more time reading and read more picture books than those without SDER
support (Yang et al., 2022). The group with SDER support read an average of 169 minutes
and 6953 words, while the group without SDER support read an average of 19 minutes and1794 words in a 3-week experiment. Furthermore, we also found that all students can improvetheir reading speed with online SDER support; however, the benefits varied by theengagement of online SDER. All students increased their reading speed by 33 words per
minute (WPM) from 99 to 132 WPM in a 1-year experiment. Among all students, the highand low engagement groups in online SDER learning increased their reading speed by 40
WPM (100 to 140 WPM) and by 27 WPM (97 to 124 WPM) respectively.
The results emphasized both SDL ability and the use of online SDER support with
LEAF played critical roles in ER in English learning. The online SDER support with LEAF
could foster students’ learning achievements and their development of SDL ability in thecontext of language teaching and learning.
AI Recommendation for Language Learning With LEAF
This section describes an application of recommendation module in LEAF for language
learning, which aims to support students’ self-directed extensive reading. ER activities may
have some issues in the difficulty of book selection and low-quality learning paths if they are32

JACET Selected Papers Vol.9 (2023) 15-46

conducted without any recommendation. To solve them, we can use an explainable AI book
recommendation (Takii et al., 2021), which is integrated into the LEAF system. Since therecommendation is based on a wordlist and the difficulty of vocabulary, this recommendation
can be used not only for books recommendation but also for various materials.
Problem and Our Proposal of AI Recommendation in Language Learning
We raise a book recommendation for ER as one of the applications of LEAF tolanguage learning. ER is one of the methods of learning foreign languages, in which learners
read as many books as possible without considering words and phrases they do not know.
However, there are some issues on introducing ER to educational practices. First, students
may not be able to choose books suitable for their English levels. While students are given a
lot of choices of books for the ER, they may have difficulty in choosing appropriate books
due to the information overload. Since books that are too easy or difficult for students
apparently have a negative influence on their learning, and would not delight the students, abook recommendation system is of great importance for the students. Second, even if therecommendation is introduced, the recommendation may not be persuasive enough forstudents to accept. Therefore, the book recommendation should provide them with not only
recommendations but also explanations why these books are recommended to them.
We developed an explainable English material recommendation for ER programs tohandle these issues. In our AI recommendation system, students can receive recommendations
on e-books whose difficulties fit their English skills. Besides, they are provided withsentences that explain why these recommendations are made for them.
Recommendation Flow and Mechanism
First, teachers prepare in advance a wordlist and books for ER in the data storage. This
wordlist has information on vocabulary and its difficulty, which is used to estimate thedifficulties of books. Next, learners read books with an e-book reader system (BookRoll in

33

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning34

our case). This system can send learners’ activity logs to LRS, a general repository forlearning/education records. The data in LRS are sent to the book recommender system, andusing them and book data, the system generates personalized book recommendations for eachlearner. Generated recommendations are displayed on a recommendation UI and provided forthe learners. Figure 11 shows the simple flow of the usage of the recommender system.

Figure 11
Workflow of Utilizing the Recommendation Module in LEAF
View

Recommendation
UI

Algorithm for
Recommendation
Generation

LearnerRead

E-bookReaderReading Learning Record
Logs

Store (LRS)

Wordlist

Vocabulary
Difficulty

Materials

Figure 12 shows a recommendation panel in the learner dashboard. This panel provides
learners with recommended book titles, recommendation weight (a value that shows how
highly the book is recommended), and explanatory sentences that explain why this book is
recommended for the learner. This book recommender system recommends books whose
difficulties are as close to learners’ English skill levels as possible. Since the explanatory
sentences describe the learners’ English proficiency levels and the difficulty of therecommended book, the learners can understand why these recommended books areappropriate for them.

34

JACET Selected Papers Vol.9 (2023) 15-46

Figure 12
Recommendation Panel in the Learner Dashboard

Application of the Recommendation in K-12 Schools
The AI recommendation has been introduced to a junior high school in Japan from2022, and more than 100 students have used it until now. Students can read more than 500 ebooks in BookRoll through the AI recommendation and they can use the recommendation
anytime at their own pace. We have conducted a case study to compare the difficulty of therecommended e-books with the student’s level of English skills in self-directed extensive
reading (Takii et al., 2022). The comparison shows that the average difficulty of therecommended books was very close to the student’s English skill level. We are investigating
how effective the recommendation was on the students’ learning behaviors, achievements,and skills in the ER experiments in K-12 settings.
Although we described the application of the recommendation to ER learning, thisrecommendation can be used in more general contexts. This is because the recommendation
algorithm is based on a wordlist and the difficulty of vocabulary, not on the content of the ebooks. Thus, the course administrator can apply this recommender system not only to therecommendation of graded reader books, but also to that of various materials; for example,
35

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

36

English textbooks, reference books, and even English vocabulary or grammar quizzes. This
recommendation can be used to simply have the learners enjoy reading and to have them learn
English effectively and efficiently. By using the recommendation to choose appropriatematerials, the learners will learn English with less stress and improve their skills effectively.
Reading-based Online Group Work with LEAF
Many researchers have conducted collaborative learning in English reading scenarios.
Some inspected the language learning process with the scaffold of group work designs (Mouri& Ogata, 2015), and others investigated the impact of collaboration on reading achievement
as an outcome (Arisman & Haryanti, 2019). These studies demonstrated several possibilities
for group work in flipped language learning classrooms (Acarol, 2019). However, fewerstudies focused on group formation strategies using learner attributes before group work.
Therefore, LEAF elicited opportunities to task from a data-driven perspective.

Figure 13
General Workflow of Reading-based Group Work with LEAF

Figure 13 shows example procedures for group work implementation using online
reading logs with LEAF, and Figure 14 shows group work modules in LEAF. In a general
workflow of reading-based group work, first, learners do some reading tasks and create somereading data in BookRoll. Before the group work, teachers can use data recorded bybehavioral sensors in LEAF as evidence to create groups using the group formation system(Liang, Majumdar, & Ogata, 2021). During the group work, the teacher and students cancheck the performance in the forum discussion in GWpulse, a system that can display the36

JACET Selected Papers Vol.9 (2023) 15-46

basic statistics of the students’ forum activities such as time interval and the number of posts,
as well as “Assistance Needed Level” for teachers based on sentiment analysis (Nakamizo etal., 2022). In the evaluation phase, it is possible to conduct peer evaluation among students in
the Peer evaluation module for rating and commenting on the peers’ group work by students(Liang, Toyokawa, et al., 2021). The results of these evaluations are fed back to each student
and stored in the LEAF data repository, as evidence for subsequent group formation and otherlearning analytics tools.

Figure 14
Group Work Modules in LEAF

37

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

38

English Reading Course in Kyoto University
At the university level, academic reading courses are universal in Japan, where students
read literature in foreign languages (Majumdar, Bakilapadavu, et al., 2021). In this work, weshall share findings of applying LEAF for group formation from the university context. The
case showed a university academic reading course in Japan in 2022. Thirty-six students
registered for the course at the beginning, and 25 finished the whole course and got a final
course grade. During this course, group work was implemented weekly from week 3 to week
11 across the 15-week semester. Students were grouped five times by the group formation
system (Liang, Majumdar, & Ogata, 2021) across the course using different group formation
inputs for assorted academic reading topics.
The group work design followed the general workflow presented in Figure 15. For each
week, students read several articles in English on BookRoll. Then, they should share anddiscuss their reading progress with their group members in the Moodle forum and prepare abrief presentation of their reading outcome as a group for the next class. During the course,
each group made presentations, which were evaluated by the audience (both the instructor andstudents) using the evaluation system of LEAF. In the meantime, students made peer ratingsof their group mates in the peer evaluation system.

Figure 15
Workflow of the Reading-based Online Group Work in University Course

38

JACET Selected Papers Vol.9 (2023) 15-46

In this study, we considered individual-level and group-level attributes to inspectpredictive input for algorithmic group formation to assist teachers with optimal group creationin the online reading course. Data collected across the eight activities from 36 students wereanalyzed using correlation.
Relation Between Input Attributes and Group Work Performance
For individual attributes, we found that reading engagement in BookRoll and peer
ratings from group members can reveal learning achievement of the course indicated by theforum participation and final course grade. The finding can guide group formation settingsand intervention suggestions in similar language learning contexts with academic reading. For
group-level attributes, a homogeneous grouping strategy based on reading annotations andreceived previous group work ratings can predict desirable performance for a group, which
can elicit guidance for subsequent group formation (see Figure 16). Meanwhile,
heterogeneous grouping based on the difficulty markers can facilitate more detailed forum
discussions with more characters in forum posts. The results can assist teachers withappropriate group formation strategies in the data-driven environment of LEAF.

Figure 16
Suggested Grouping based on Correlations between Group-level Attributes
Antecedents (individual attributes of group members)# of difficulty
markers

Average Forum
posts

Previous peer
ratings for individual

Previous peer
ratings for groups

Average Forumcharacters

Peer ratings ofgroup presentation

Processes

Consequences(group attributes)(group attributes)

Better for heterogeneoous groupings
Better for homogeneous groupings

39

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

40

Discussion and Conclusion
We provided an overview of the various teaching and learning activities that LEAF
platform can support by using the e-book reader, BookRoll and by utilizing the reading
behavior logs for various data-driven services in LEAF. For teachers functions such as
automatically recommending books based on the reading patterns of students can help reduce
their workload to choose the appropriate level of books for each student. Even for learners
picking recommended books which match their level can potentially make choosing aparticular book and reading it less stressful. As the recommendation is based on a knowledge
map it additionally aims to maximize their vocabulary learning by providing books that have
vocabularies in their zone of proximal development. Learners can plan, monitor and continue
their daily reading habits by reflecting on the visualized data in the learning analytics
dashboard and using tools such as GOAL. We found it improves motivation to continue
reading and also improves students' reading speed in words per minute (WPM) over a longer
run. The GOAL tool can also help teachers to monitor the progress of group or individual
students through a dashboard. We recommend using BookRoll to share daily reading basedlearning activities and GOAL tool as support to complete them in a self-directed manner.
Using data in the context of language learning further helps to support the
metacognitive aspects of learning. Specifically for reading, visualizing the process indicatorssuch as content navigation patterns and artefacts of annotations that the students create can beused by the teachers as well as students in the LEAF platform. Implementing active reading(AR) strategies in BookRoll helped to improve learners’ reading performance (vocabulary andreading comprehension) of new text. While learners used the AR dashboard, they could
compare their reading behaviors with their peers as well as from their previous learning
episodes. Teachers can use the dashboard to decide subsequent actions and also use it as part40

JACET Selected Papers Vol.9 (2023) 15-46

of pedagogical design. For instance, they can use the visualization of the highlighted yellow
markers to discuss any part of the text that needs further clarification in class. The groupformation module can help the teacher easily form groups for cooperative learning activities.
It can optimize group members automatically based on accumulated learning evidence fromtheir previous online learning context utilizing the collected learning logs.
While the functions and tools in the LEAF platform are meant to be simple, yet often
we have seen teachers needs some time to integrate meaningful learning activity that can becomplemented by the power of learning analytics. Various data-driven services, such as therecommendation function, can build more robust models with the increase in data logged
within that system in a specific context. Hence in a longer run it becomes easier to conduct
evidence-based practices with the student learning logs. Members of the research teamprepared feature manuals and scenario videos for teacher training for easier onboarding.
However in the university learning context, students who didn’t log in the LEAF platform
also missed any of the data-driven services. We recommend starting with simpler tasks andimmediate feedback to students using the dashboard in Log Palette as a good starting point.
The possibilities of data-driven services are immense by utilizing digital learning tools.
LEAF platform is one of them to realize the potential and hence is under active development.
The algorithms and features are periodically updated based on the results and feedback fromthe implementation studies. Our efforts focus on extracting good practices that emerge fromthe real-world data gathered as learning interaction logs and enabling evidence-based
practices based on that.
References
Acarol, K. (2019). A study on the effectiveness of flipped learning model. Kara Harp Okulu
Bilim Dergisi, 29(2), 267-295.

41

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

42

Arisman, R., & Haryanti, I. S. (2019). Using small group discussion to improve student’s
reading achievement on narrative text. English Community Journal, 3(1), 325-334.
https://doi.org/10.32502/ecj.v3i1.1698
Aronson, E. (1978). The jigsaw classroom. Sage.
Boyer, S. L., Edmondson, D. R., Artis, A. B., & Fleming, D. (2014). Self-directed learning: A
tool for lifelong learning. Journal of Marketing Education, 36(1), 20-32.
https://doi.org/10.1177/0273475313494010
Brockett, R. G., & Hiemstra, R. (2018). Self-direction in adult learning: Perspectives ontheory, research, and practice. Routledge.
Chen, M. R. A., Hwang, G. J., Lin, Y. H., Abou-Khalil, V., Li, H., & Ogata, H. (2022). A
reading engagement-promoting strategy to facilitate EFL students' mobile learningachievement, behaviour and engagement. International Journal of Mobile Learning andOrganisation, 16(4), 489-506. https://doi.org/10.1504/IJMLO.2022.125968
Day, R. R., & Bamford, J. (1998). Extensive reading in the second language classroom.
Cambridge University Press. https://doi.org/10.1177/00336882980290021
Flanagan, B., Liang, C., Majumdar, R., & Ogata, H. (2021). Towards explainable groupformation by knowledge map based genetic algorithm. International Conference on
Advanced Learning Technologies (ICALT) 2021, 370-372.
https://doi.org/10.1109/ICALT52272.2021.00119
Gaftandzhieva, S., Docheva, M., & Doneva, R. (2021). A comprehensive approach to learning
analytics in Bulgarian school education. Education and Information Technologies, 26,
145-163. https://doi.org/10.1007/s10639-020-10261-w
Khusniyah, N. L., & Lustyantie, N. (2017). Improving English reading comprehension ability
through Survey, Questions, Read, Record, Recite, Review Strategy (SQ4R). English
Language Teaching, 10(12), 202-211.

42

JACET Selected Papers Vol.9 (2023) 15-46

Krumm, A., Means, B., & Bienkowski, M. (2018). Learning analytics goes to school: A
collaborative approach to improving education. Routledge.
Li, H., Majumdar, R., Chen, M.R.A., & Ogata, H. (2021a). Goal-oriented active learning(GOAL) system to promote reading engagement, self-directed learning behavior, andmotivation in extensive reading. Computers & Education, 171, 104239.
https://doi.org/10.1016/j.compedu.2021.104239
Li, H., Majumdar, R., Chen, M. R. A., Yang, Y., & Ogata, H. (2021b). Analysis of selfdirected learning ability, reading outcomes, and personalized planning behavior for
self-directed extensive reading. Interactive Learning Environments, 1-20.
https://doi.org/10.1080/10494820.2021.1937660
Liang, C., Majumdar, R., & Ogata, H. (2021). Learning log-based automatic group formation:
System design and classroom implementation study. Research and Practice inTechnology Enhanced Learning, 16(1), 14. https://doi.org/10.1186/s41039-021-00156w
Liang, C., Toyokawa, Y., Nakanishi, T., Majumdar, R., & Ogata, H. (2021). Supporting peer
evaluation in a data-driven group learning environment. 27th International Conference
on Collaboration Technologies and Social Computing: (CollabTech 2021), 93-100.
https://doi.org/10.1007/978-3-030-85071-5_7
Majumdar, R., Bakilapadavu, G., Majumder, R., Chen, M. R. A., Flanagan, B., & Ogata, H.
(2021). Learning analytics of humanities course: Reader profiles in critical reading
activity. Research and Practice in Technology Enhanced Learning, 16, 25.
https://doi.org/10.1186/s41039-021-00164-w
Majumdar, R., Flanagan, B., & Ogata, H. (2021). eBook technology facilitating university
education during COVID-19: Japanese experience. Canadian Journal of Learning andTechnology, 47(4). https://doi.org/10.21432/cjlt28038

43

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

44

Majumdar, R., Yang, Y. Y., Huiyong, L., Akçapınar, G., Flanagan, B., & Ogata, H. (2018).
GOAL: Supporting learner’s development of self-direction skills using health andlearning data. The 26th International Conference on Computers in Education(ICCE2018), 406-415.
Martínez Monés, A., Dimitriadis Damoulis, I., Acquila Natale, E., Álvarez, A., Caeiro
Rodríguez, M., Cobos Pérez, R., ... & Sancho Vinuesa, T. (2020). Achievements andchallenges in learning analytics in Spain: The view of SNOLA. RIED. Revista
Iberoamericana de Educación a Distancia, 23(2), 187.
https://doi.org/10.5944/ried.23.2.26541
MEXT. (2020). Aiming to build Japanese-style school education in the Reiwa era. Retrieved
January 30, 2023, from https://www.mext.go.jp/kaigisiryo/content/20200928mxt_syoto02-000009891_4.pdf
Mouri, K., & Ogata, H. (2015). Ubiquitous learning analytics in the real-world language
learning. Smart Learning Environments, 2(15), 1-18. https://doi.org/10.1186/s40561015-0023-x
Nakamizo, Y., Majumdar, R., Horikoshi, I., Liang, C., Flanagan, B., & Ogata, H. (2022).
GWpulse: Supporting learner modeling and group awareness in online forum withsentiment analysis. The 30th International Conference on Computers in Education(ICCE2022), 230-232.
Ogata, H., Majumdar, R., Akçapınar, G., Hasnine, M. N., & Flanagan, B. (2018). Beyond
learning analytics: Framework for technology-enhanced evidence-based education andlearning. The 26th International Conference on Computers in Education (ICCE2018),
493-496.
Ogata, H., Majumdar, R., Flanagan, B., & Kuromiya, H. (in press). Learning analytics andevidence-based K12 education in Japan: Usage of data-driven services and mobile

44

JACET Selected Papers Vol.9 (2023) 15-46

learning across two years. International Journal of Mobile Learning and Organisation.
https://doi.org/10.1504/IJMLO.2023.10048714
Ogata, H., Majumdar, R., Yang, S. J., & Warriem, J. M. (2022). Learning and Evidence
Analytics Framework (LEAF): Research and practice in international collaboration.
Information and Technology in Education and Learning, 2(1), Inv-p001.
https://doi.org/10.12937/itel.2.1.Inv.p001
Ogata, H., Yin, C., Oi, M., Okubo, F., Shimada, A., Kojima, K., & Yamada, M. (2015). Ebook-based learning analytics in university education. The 23rd International
Conference on Computers in Education (ICCE 2015), 401-406.
Ogle, D. M. (1986). KWL: A teaching model that develops active reading of expository text.
The Reading Teacher, 39(6), 564-570.
Queiroga, E. M., Batista Machado, M. F., Paragarino, V. R., Primo, T. T., & Cechinel, C.
(2022). Early prediction of at-risk students in secondary education: A countrywide K12 learning analytics initiative in Uruguay. Information, 13(9), 401.
https://doi.org/10.3390/info13090401
Spivey, N.N. (1987). Construing constructivism: Reading research in the United States.
Poetics, 16(2), 169-192. https://doi.org/10.1016/0304-422X(87)90024-6
Takii, K., Flanagan, B., & Ogata, H. (2021). An English picture-book recommender systemfor extensive reading using vocabulary knowledge map. Companion Proceedings of the
11th International Conference on Learning Analytics and Knowledge, 40-42.
Takii, K., Flanagan, B., Li, H., Yang, Y., & Ogata, H. (2022). Explainable English Material
Recommendation Using an Information Retrieval Technique for EFL Learning. The
30th International Conference on Computers in Education (ICCE2022), 561-570.

45

Hiroaki Ogata et al.
Towards Data and Evidence-driven Education in the Context of Language Teaching and Learning

46

Toh, W., & Kirschner, D. (2020). Self-directed learning in video games, affordances andpedagogical implications for teaching and learning. Computers & Education, 154,
103912. https://doi.org/10.1016/j.compedu.2020.103912
Toyokawa, Y., Majumdar, R., Lecailliez, L., & Ogata, H. (2021a). A flipped model of active
reading using learning analytics-enhanced e-book platform. The 29th International
Conference on Computers in Education (ICCE2021), 510-515.
Toyokawa, Y., Majumdar, R., Lecailliez, L., Liang, C., & Ogata, H. (2021b). Technology
enhanced Jigsaw activity design for active reading in English. 2021 International
Conference on Advanced Learning Technologies (ICALT2021), 367-369.
Yang, Y., Majumdar, R., Li, H., Flanagan, B., & Ogata, H. (2022). Design of a learning
dashboard to enhance reading outcomes and self-directed learning behaviors in out-ofclass extensive reading. Interactive Learning Environments, 1-18.
https://doi.org/10.1080/10494820.2022.2101126

46

Selected Papers
Practitioner Report

JACET Selected Papers Vol.9 (2023) 49-63

The Horizon Report: What can we Learn About Educational Technology
Trends?
Neil Cowie
Okayama University
Keiko Sakui
Kobe Shoin Women’s University

Abstract
The Horizon Report (Pelletier et al., 2022), published by Educause, is an annual survey ofglobal educational technology trends and forecasts for the future. This proceedings paper
summarizes the main points of the 2022 version which are applicable to JACET members.
These include: 1) social and technological trends such as an expansion of hybrid and online
learning, and an increased use of data analytics; and, 2) the influence of three technologybased practices: artificial intelligence (AI) for learning; hybrid learning spaces on university
campuses; and the development of micro-credentials. Language teachers working at Japanese
universities were asked whether the results of the Horizon Report were resonant with theirexperience. The participants thought that the results were relevant to Japan, especiallyconcerning online teaching and the development of micro-credentials. In addition, thechallenge of working with machine translation tools was highlighted as being particularly
important for language teachers. The survey results suggest that the Horizon Report is a
useful guide to help plan for the use of educational technology in higher education and that it
can be used effectively to identify specific training needs for teachers in such institutions.
Keywords: Horizon Report, technology trends, AI, hybrid, micro-credentials

49

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

Educause is a nonprofit membership organization based in the US. It was established in1998 with a mission “to advance higher education through the use of information
technology,” and has over 100,000 members worldwide, including 242 institutional members
outside of the US (Educause, 2022). Educause annually publishes The Horizon Report(Pelletier et al., 2022) which summates and predicts technological trends worldwide that
could influence the future of higher education. This is particularly important as the pandemicof the last three years has digitally transformed education across the globe (Turnbull et al.,2021). The purpose of this paper is to inform JACET members about the contents of thecurrent Horizon Report and to examine whether such a systematic summary of technologicaldevelopments can be applied to Japan. The paper is divided into two sections: 1) a summaryof the report; and, 2) a discussion of the results of a survey on how the report is relevant tohigher education in Japan.
Summary of the 2022 Horizon Report
The Horizon Report is compiled each year by a number of expert panels. These experts
meet independently and by using the Delphi method of forecasting (Page et al., 2015) identify
what they think are the most important trends in educational technology that are currently
affecting and will affect higher education. The Delphi method relies on experts who give
opinions which are revised and refined through a number of rounds of summaries andquestions. The 2022 version lists 63 experts, including one representative from Japan, that
contributed to the report. These experts in turn recruit a number of further participants who,
through various brainstorming and evidence gathering procedures, try to identify trends intheir region. The authors of this paper took part in the Australasian version of this process inNovember 2021 and again in February 2022 (ASCILITE, 2022). This was a voluntary process
in which members of the Australasian Society for Computers in Learning in Tertiary
Education (ASCILITE) were invited to join an online discussion about the Horizon Report. It

50

JACET Selected Papers Vol.9 (2023) 49-63

is one example of a regional group contributing to the global survey. The main trends that thissub group reported are listed in the Appendix.
Once all the different opinions from the various expert panels are compiled andanalysed, the results of the global survey are written up into the final report. This is divided
into four main areas:
1) Social, Technological, Economic, Environmental and Political (STEEP) trends
2) Key technologies and practices3) A forecast of possible impacts
4) Regional case studies/essays on the implications of the technology trends
In the next section, the first two of these four areas will be briefly described with afocus on the main ideas that are most relevant to JACET members.
STEEP Trends
Firstly, at the “big picture” level, the Horizon Report frames change in higher education
within five, often overlapping, categories: Social, Technological, Economic, Environmental
and Political (STEEP). STEEP is a commonly used framework for analyzing strategic factorsthat may affect future design (uts.design, 2022). Table 1 summarizes the main trends thatwere identified in the 2022 report.

51

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

Table 1STEEP Categories and Main Trends in 2022
STEEP Category

Main Trends

1. Social

Hybrid/Online; Skills-based Learning; Remote Work

2. Technological

Different Modes of Teaching; Learning Analytics; Big Data;
Cybersecurity

3. Economic

Cost/Value of Degrees; Digital Economy; Financial Deficits

4. Environmental

Campus Change; Increase in SDGs; Planetary Health

5. Political

Uncertainty in Higher education; Ideology and Pedagogy;
Decreases in Funding

Adapted from The Horizon Report (Pelletier et al., 2022).

As the first two categories, social and technological, are directly concerned withteaching and learning they are the ones that are thought to be most pertinent to JACET
members. Detailed descriptions of these trends are given below focusing on ones with themost direct pedagogical impact (highlighted in italics). The other three categories, economic,
environmental and political, are all extremely important but as they are not so directly
concerned with pedagogy are not discussed here.
SocialThe first category of trend is social; that is teaching and learning takes place within theinteractions that teachers and learners have in social contexts. The COVID-19 pandemic has
had an enormous impact on this social context from early 2020 onwards, especially in how
lessons have been completely reorganized to minimize face to face contact on university
campuses. As a result, “hybrid” and “online” are terms that most educators in Japan are now
52

JACET Selected Papers Vol.9 (2023) 49-63

very familiar with as they have had had direct experience of these modes of teaching. The
Horizon Report predicts that hybrid and online will continue to have a major effect (Pelletier
et al., 2022, p. 7).
The Horizon Report defines hybrid as being a combination of some students attending alesson face to face and others participating remotely; that is a mix of face to face and online at
the same time. This particular teaching approach is the most challenging for an institution andteacher to organize well as it involves considerable investment in equipment and a high levelof pedagogical skill (Pelletier et al., 2022, p. 23). Wholly online lessons can be divided into
synchronous or asynchronous. The former will probably involve video conferencing using
applications such as Zoom or Microsoft Teams so that the teacher can communicate directlywith students and vice versa. For asynchronous, or “on-demand,” lessons teachers are
expected to create materials that can be used autonomously by students and provide timely
feedback on the tasks that students are required to do.
This predicted trend towards a continuation of hybrid and online learning has many
implications that institutions need to plan for, such as adapting classrooms, being flexible
about schedules, and accepting of remote working. One pedagogical issue is that of how tobest maintain levels of student engagement with fewer face to face interactions. Educatorsneed training in online teaching and materials development in order to create an experience
which does not just substitute for the classroom but actually takes advantage of the online
experience to fully engage students. This can take many forms including increased flexibility
with tasks and assignments, greater use of multimedia, and personalized feedback.
Technological
The second STEEP trend examines in more detail the technological innovations that are
impacting teaching and learning in higher education. After a period of “emergency remoteteaching” (Bozkurt et al., 2020) that began in 2020, many higher education institutions may
be looking to go back to their previous, more familiar mode of working on campus. However,53

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

the Horizon Report (Pelletier et al., 2022, p. 26) predicts that different modes of teaching andlearning will become further embedded in higher education, as with hybrid and online
mentioned above. Institutions will need to define what these different modes of teaching are
and then plan for and invest in them: hybrid, hyflex, and blended classrooms are just some ofthe online teaching approaches that need different equipment, applications and training forstaff that differentiates them from traditional face to face lessons.
A connected technological trend is the use of learning analytics, which is “themeasurement, collection, analysis and reporting of data about learners in their context, forpurposes of understanding and optimizing learning and the environment in which it occurs”(Siemens & Long, 2011, p. 33). Most institutions now have Learning Management Systems(LMSs) in place such as Moodle, Manaba, and Blackboard. These systems incorporate
analytics that could be used to support student learning; for example, by increasing thepersonalization of the curriculum or by identifying students at risk of falling behind.
However, the potential to use this information is not, at the moment, always fully realized as
many staff do not have the expertise to use the systems well. This is more fully explored in
the next section.
Key Technologies and Practices
Building upon the broad STEEP trends, the Horizon Report details six key technologies
and practices that it forecasts will be most important in higher education in the near future(Table 2). These in turn can be divided into three areas: artificial intelligence (AI); hybrid;and, micro-credentials.

54

JACET Selected Papers Vol.9 (2023) 49-63

Table 2
Key Technologies and Practices in 2022
Key Technology or Practice

Example Uses

1. Artificial Intelligence (AI)AI for Learning Analytics
AI for Learning Tools

2. HybridHybrid Learning Spaces
Mainstreaming of Hybrid and Remote Learning
Professional Development for Hybrid and Remote
Learning

3. Micro-credentials

Development of Micro-credential Courses

Adapted from The Horizon Report (Pelletier et al., 2022).

AI for Learning Analytics and Learning Tools
AI is now ubiquitous all around us, ranging from automated email reminders to voicerecognition devices and chatbots (Shadiev & Yang, 2020). The Horizon Report (Pelletier etal., 2022, p. 20) identifies two areas in which AI is having a particular impact on higher
education: for learning analytics and learning tools.
As discussed above, there are learning analytic systems built into LMSs which can
track student work in many ways: time spent on tasks, completion dates, satisfaction rates andso on. Such data can potentially be used to monitor student progress, identify students who
are at risk of falling behind, assess the merits of materials and tasks, and improve course
design. The implications of this trend in AI are that institutions and staff need to be aware of
the potential uses of data analytics and trained in how to use it. Furthermore, institutions needto also be aware of the potential negative aspects of learning analytics such as privacy issues

55

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

(Roberts-Mahoney et al., 2016), and a reductionist view of students as statistics rather thanbeing seen as individual people (Reinders, 2018).
The second use of AI is for creating digital tools which students can use to support theirlearning. Two examples of such tools that the Horizon Report (Pelletier et al., 2022, p. 22)
describes that are of particular relevance to JACET members both support academic writing.
One is Aca Writer (https://acawriter.uts.edu.au) which has been developed by the Universityof Technology Sydney, and the other is Automated Feedback (https://feedbackfruits.com)
from the company Feedback Fruits in partnership with Erasmus University Rotterdam, andRotterdam University of Applied Sciences. Both tools can be used by students to identify
problems and errors in their writing and also to receive comments and suggestions to improvetheir work. Writing teachers typically spend much time giving feedback on basic grammar
errors or organizational weaknesses. It is claimed, therefore, that these AI tools can free up
teacher time to focus on higher order issues connected to writing such as the ideas and thelogic of a writer’s argument (Godwin-Jones, 2022).
Students can access such AI-based tools themselves or they can be bundled with an
institution’s online student support system. The choices that teachers face are to know whatkinds of tools their students are using and how to use them in the most effective way. As withlearning analytics, teachers need information and training on how best to take advantage ofsuch technology. Perhaps the most obvious AI tools that language teachers and learners areaware of are those of machine translation such as Google Translate(https://translate.google.com) or DeepL (https://www.deepl.com/translator). There is muchdebate at the moment as to how best to use these tools that can save so much time and effort
in language learning (Zhang & Torres-Hostench, 2022). At the same time, however, they can
prevent students engaging fully with a foreign language if the tools are only used as a crutch
without understanding how to use them in a way that is beneficial for language learning.
Hybrid Spaces, Learning and Professional Development
56

JACET Selected Papers Vol.9 (2023) 49-63

The second major technological practice is that of hybridity. The Horizon Reporthighlights how hybrid learning spaces are now being developed in many institutions in orderto respond to the needs of remote students and those on campus (Pelletier et al., 2022, p. 23).
This form of learning has been present for many years (Porter & Graham, 2016) but isbecoming mainstreamed due to the demands of the pandemic. Institutions face the dilemma ofwhether to go back to their traditional face to face context or include online methods in someway. As a result, those institutions that do decide to do both need to find effective ways totransform some of their learning spaces into hybrid ones. This will involve considerable
investment and know-how in making classrooms suitable both for on-campus and remotestudents. It will typically involve creating conditions with cameras, monitors and speakers inwhich both sets of students can easily communicate with each other and where a teacher cangive equal attention to both sides. It will, in short, require “new ways of thinking aboutteaching spaces” (Bryant, 2022). In addition, institutions need to provide training andprofessional development for staff in how to use those teaching spaces in the most effective
pedagogical ways.
Micro-credentials
The third important technological practice is the steady development of microcredentials. These are short, usually online, courses that students can take in their own timeand at their own pace rather than registering for a longer and more rigid macro-credential
such as a degree (Oliver, 2019). They are particularly useful for workers who wish to upskill
or reskill in a rapidly changing employment market. The most common form of microcredential is that of Massive Open Online Courses (MOOCs) which are increasing rapidly(Shah, 2020). Recently many MOOC providers have teamed up with higher education
institutions to provide micro-credential courses. One example is Future Learn in the UK
which has a large number of university partners worldwide providing a huge variety ofcourses and materials, including language teaching. In order to create these courses,
57

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

universities need personnel who are skilled in online learning design and have multimedia
experience. With many regions and individual countries investing heavily in the developmentof micro-credential systems (Cowie & Sakui, 2022) it seems that micro-credentials will soon
offer a practical alternative to degree courses. Higher education institutions can see this as athreat or an opportunity in how they might react to this development. As a threat, microcredentials can divert students away from universities; but as an opportunity, universities canoffer a wider range of courses and diversify their student base.
How is the Horizon Report of Relevance to Japan?
The Horizon Report provides a framework to help higher education institutions plan for
the future in terms of investment in educational technology tools and practices. Although theexpert panels that create the report are based in many countries, there is relatively little input
from Japan-based scholars and the report’s findings do not mention any Japanese examples atall. So, how relevant is it to the Japanese context? In order to try and answer this question theauthors carried out a small-scale survey of university teachers in Japan to gauge their reaction
to the Horizon Report.
At an academic conference in Japan in June 2022 (Cowie, 2022), the contents of the
Horizon Report were explained to 20 university language teachers. These teachers discussed
this information and wrote their responses to it on post-it notes which were displayed so allparticipants could read them. In a mini version of the Delphi technique the teachers then votedon the responses that they most agreed with. In this way reactions to the report could becollected and analysed. Below are some of the post-it note quotations that received the mostvotes and which reflect the ideas that those teachers based in Japan think are important. These
include general trends concerning online teaching at university; the potential of microcredentials; and the influence of machine translation.
Firstly, the Horizon Report points to the trend for higher education institutions to try tolearn from the experience of rapidly going online during the global pandemic (Pelletier et al.,58

JACET Selected Papers Vol.9 (2023) 49-63

2022, p. 32). Japan is at risk of losing the experience and diverse opportunities that thisunexpected development provided. The survey participants emphasized the need to continue
with online teaching despite the current pressures to return to pre-pandemic methods: “TheJapanese government should take into account the knowledge teachers and students havegained technologically to allow certain courses to remain online. This is tied to all 5 points in
STEEP.” However, there are concerns that Japanese universities are not training teachers
well enough to allow this to happen and that continuing faculty development is vital:“Teachers need new skills [as] they are no longer knowledge providers because of AI.”
Secondly, there was an awareness that as micro-credentials are becoming more
important teachers need to adapt in order to be part of that change: “More students will
choose to learn online outside university. Less teachers will be needed.” “We need to provide
micro-credentials for students but do we have teachers qualified to do this?”
Finally, one specific trend of significance to language learning is the increasing
influence of machine translation: “How we learn foreign languages need to be shifted. We
need to incorporate and embrace AI based translation systems into our classroom.”
So, what are the implications of these reflections on educational technology trends inJapan? The participants concluded that the Horizon Report is an excellent summary ofimportant influences and developments in the field of educational technology. As such it is anextremely useful resource to aid higher education institutions to plan for the future. However,it needs to be adapted to regional contexts wherever appropriate. In Japan, it would seem one
major priority is to support university teachers with training and information about the mostimportant trends that will affect them in the near future. As part of this kind of initiative there
needs to be a clear debate on what kind of skills and knowledge institutions would like toprovide their students with. Unfortunately, both institutions and individual teachers arelimited in what they can do if current calls for “back to normal” (Garrett et al., 2022) create
uncertainty as to what direction digital education can go.
59

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

Conclusion
Based on the 2022 Horizon Report, this short paper has described the most recent
educational technology trends and practices that are predicted to influence higher education in
the near future. These include social and technological trends such as hybrid and online
teaching, the increasing use of learning analytics as well as AI, hybrid learning spaces andmicro-credentials. A sample of academics based in Japanese universities confirmed that such
trends are relevant to Japan and that university teachers need to adapt to a rapidly changing
digital landscape. Three aspects of particular focus are how to continue to implement future
online teaching, how to provide micro-credentials, and how to face the challenge of using
machine translation to support student learning. It is hoped that this information is useful to
JACET members and can aid them in planning for technological developments in the near
future.
Acknowledgments
Many thanks to the anonymous reviewers for their kind suggestions and comments.
References
ASCILITE (2020). ASCILITE’s Contextualising Horizon Initiative 2022. February 15th,
2022. https://ascilite.org/events/contextualising-horizon-workshop.
Bozkurt, A., Jung, I., Xiao, J., Vladimirschi, V., Schuwer, R., Egorov, G., … Paskevicius, M.
(2020). A global outlook to the interruption of education due to the COVID-19
pandemic: Navigating in a time of uncertainty and crisis. Asian Journal of Distance
Education, 15(1), 1–126. https://doi.org/10.5281/zenodo.3878572
Bryant, P. (2022). Making space for connected learning: An ecosystem approach to designingteaching spaces in higher education part 1. https://cdrg.blog/2022/08/04/
Cowie, N. (2022). Technology trends in language education in Japan. Global Negotiation
60

JACET Selected Papers Vol.9 (2023) 49-63

Symposium. June 25th, 2022. Notre Dame Seishin University, Okayama, Japan.
Cowie, N., & Sakui, K. (2022). Micro-credentials: Surveying the landscape. In T. Cooper & J.
York (Eds.). Remote Teaching and Beyond 2021. Proceedings of the JALTCALL 2021
Conference (pp. 15-27). https://doi.org/10.37546/JALTSIG.CALL.PCP2021-02
Educause (2022). EDUCAUSE helps you elevate the impact of IT.
https://www.educause.edu/about.
Garrett, R., Simunich, B., Legon, R., & Fredericksen, E. E. (2022). CHLOE 7: Tracking
online learning from mainstream acceptance to universal adoption: The changing
landscape of online education, 2022. Retrieved fromhttps://www.qualitymatters.org/qa-resources/resourcecenter/articles-resources/CHLOEproject
Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructedlanguage learning. Language Learning & Technology, 26(2), 5–24.
http://doi.org/10125/73474
Oliver, B. (2019). Making micro-credentials work for learners, employers and providers.
Deakin University. http://dteach.deakin.edu.au/wp-content/uploads/sites/103/2019/08/
Making-micro-credentials-work-Oliver-Deakin-2019-fullreport.pdf
Page, A., Potter, K., Clifford, R., McLachlan, A., & Etherton-Beer, C. (2015). Prescribing forAustralians living with dementia: Study protocol using the Delphi technique. BMJ
Open, 5:e008048. doi:10.1136/bmjopen-2015-008048
Pelletier, K., McCormack, M., Reeves, J., Robert, J., Arbino, N., Dickson-Deane, C., C.,
Guevara, C., Koster, L., Sanchez-Mendiola, M., Bessette, L.S., & Stine, J. (2022). 2022
EDUCAUSE Horizon Report Teaching and Learning Edition (pp. 1–58). EDUC22.
Porter, Wendy W, & Graham, Charles R. (2016). Institutional drivers and barriers to faculty
adoption of blended learning in higher education. British Journal of Educational
Technology, 47(4), 748-762. https://doi.org/10.1111/bjet.12269
61

Neil Cowie, Keiko Sakui
The Horizon Report: What can we Learn About Educational Technology Trends?

Reinders, H. (2018). Learning analytics for language learning and teaching. JALT CALL
Journal, 14(1), 77–86. http://dx.doi.org/10.29140/jaltcall.v14n1.225
Roberts-Mahoney, H., Means, A. J., & Garrison, M. J. (2016). Netflixing human capitaldevelopment: Personalized learning technology and the corporatization of K-12
education. Journal of Education Policy, 31(4), 405–420.
http://dx.doi.org/10.1080/02680939.2015.1132774
Shadiev, R., & Yang, M. (2020). Review of studies on technology-enhanced language
learning and teaching. Sustainability, 12(2), 524. https://doi.org/10.3390/su12020524
Shah, D. (2020). The second year of the MOOC: A review of MOOC stats and trends in2020. https://www.classcentral.com/report/the-second-year-of-the-mooc/
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education.
EDUCAUSE review, 46(5), 30–40.
Turnbull, D., Chugh, R. & Luck, J. (2021). Transitioning to e-learning during the COVID19 pandemic: How have higher education institutions responded to the challenge?
Education and Information Technologies 26, 6401–6419.
https://doi.org/10.1007/s10639-021-10633-w
uts.design (2022). What is a STEEP analysis?
https://www.utsdesignindex.com/researchmethod/steep-analysis/
Zhang, H., & Torres-Hostench, O. (2022). Training in machine translation post-editing forforeign language students. Language Learning & Technology, 26(1), 1–17.
http://hdl.handle.net/10125/73466

62

JACET Selected Papers Vol.9 (2023) 49-63

Appendix: ASCILITE’s Contextualising Horizon Initiative 2022 (February 15th, 2022)
The following were the top seven educational technology trends in the Australasianregion in 2022 (ASCILITE, 2022):
1. Redefinition and interrogation of longstanding pedagogical practices (e.g., lectures andexams)
2. Self-care and well-being for staff and students
3. Blended models of learning
4. Ed tech infrastructure to enable learning
5. Accessible content and digital equity
6. Co-design for higher education
7. Micro-credentials

63

JACET Selected Papers Vol.9 (2023) 65-83

The Impact of a Curriculum-based Formative Assessment on EGAP

Ryan Spring
Shizuka Sakurai
Tohoku University

Abstract
Formative assessments can lead to increases in learning outcomes by helping students torecognize their shortcomings in order to overcome them. Though formative assessments are
generally handled within individual classes, this study reports on a formative assessment thatwas created for the skills mandated in a university-wide curriculum. Students took theassessment on a voluntary basis after their summer vacation and then were provided withfeedback about how well they performed on each of the skills they were meant to havemastered in their first semester. Based on their scores, individualized advice was also given toeach student regarding how to improve at the skills they did not score well on. We used preand posttest TOEFL ITP® scores to determine how much improvement students at theuniversity exhibited, and then compared the delta scores of those that took one, two, or noformative assessments and gathered student opinions via survey to observe how much impact
the formative assessment tests had on general English improvement. We found that theformative assessment tests had a discernable, albeit weak, positive influence on improvement,
and concluded that we need to provide even more feedback in the future.
Keywords: Formative assessment, curriculum, impact, learning outcomes

Formative assessment (FA) is a somewhat loosely defined term that constitutes anumber of different methods used to evaluate students’ learning, needs, or progress during
their learning process. The largest distinction between formative and other types of65

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

assessment is that the goal of FA is not to rate learners for the purposes of ranking or grading,but rather to help teachers and students identify the problems that learners are having so thatthey may focus on these areas and receive assistance as necessary (e.g., Black & Wiliam,
2009; Gardner, 2012). Though FA is generally thought to be a part of a shorter-term learning
period, i.e., a lesson, class, or course, it can theoretically also be applied to longer-term
learning periods if the goals are clear to the students.
As of April 1, 2020, Tohoku University revised its general education English
curriculum, and mandated specific EGAP (English for General Academic Purposes) skills thatstudents are expected to learn in four classes taught over two semesters in their first year, as
outlined in their designated textbook, Pathways to Academic English, 3rd Edition (Spring,2021). However, in transitioning from the first semester to the second, teachers do not always
know how well students acquired the skills outlined in the first semester. Furthermore,
students are not necessarily aware of their skill level. Finally, it is also conceivable thatstudents might lose some of their skill over the summer break between first and second
semester, as has been reported in other studies (e.g., Bardovi-Harlig & Stringer, 2010).
Therefore, we devised a formative assessment that students could use at the beginning of theirsecond semester courses which would: (1) alert students to their level of mastery of each of
the skills from their first semester, (2) provide students with constructive feedback based on
their scores, and (3) alert teachers to the first-semester skills acquisition levels of theirstudents. While such an endeavor could theoretically be helpful to students and teachers, theimpact on the learning outcomes of the general student body is unknown, so this study seeks
to evaluate how meaningful the curriculum-based formative assessment given in 2021 at
Tohoku University was.
Literature Review

66

JACET Selected Papers Vol.9 (2023) 65-83

Although a number of different treatments could technically be considered FA, we
adopted the following general guiding principles for the treatment: it should be (1) done in the
middle of learning (i.e., not at the beginning or end), (2) used for the purpose of facilitating
learning rather than simply evaluating the final end point, and (3) used by teachers andlearners to revise learning effort (Black & Wiliam, 2009; Dick & Carey, 1996; Gardner, 2012;
Mahshanian et al., 2019). Thus, most reports of FAs focus on in-class activities that help
students to notice their weaknesses and revise their learning strategies accordingly (e.g.,
Dalby & Swan, 2018; Kluger & De Nisi, 1996; etc.). However, taking a broader definition ofFA, following works such as Harris and McCann (1994) and Ketabi and Ketabi (2014), a
treatment mid-semester, between classes, as proposed by this study, could also be considered
formative as long as its purposes are to facilitate student learning and revise student or teacher
efforts. Based on this premise, we created our FA to be a no-stakes test that students could
voluntarily take to help them realize how well they acquired the skills of their first semester
courses before the start of their second semester courses. Furthermore, we endeavored tocreate feedback that would encourage students to review the areas that they had difficulty
with, and informed participating teachers of their students’ strengths and weaknesses.
Since FAs are, by definition, designed to influence learning, feedback is a criticalcomponent. Specifically, any feedback provided by a FA should be aimed not only at helping
students realize their current level of acquisition, but also at promoting planning and action to
improve in the future (Black & Wiliam, 2009). Furthermore, works such as Kluger and De
Nisi (1996) suggest that any feedback from a FA should be of low-threat to students’ selfesteem, as harming their self-image may demotivate them, which would not result in thedesired effect of adjusting learning strategies or redoubling effort. Therefore, it follows thatany such advice should likely be as low-risk as possible, and should be focused on positive
action, such as suggestions of what to do going forward. Finally, Gardner (2012) says that67

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

feedback should ensure accountability for progress, and thus should likely be continual. The
feedback created for the FAs put forth by this study could be used to influence teacher andstudent activity and made without posing unreasonable threat to self-esteem. However, as
they are designed to be given once, it is difficult to know how much they could really ensureaccountability for progress.
Another issue to consider when providing feedback for FA is the use of technology toprovide feedback. Though technology can provide feedback for several aspects in EFL suchas pronunciation (e.g., Guskaroska, 2020; Spring & Tabuchi, 2022) and writing (e.g., Gao &Ma, 2020; Shang, 2022), some studies suggest that automated feedback alone is not as
effective as feedback from peers and instructors (e.g., Link et al., 2014; Wu et al., 2022).
However, there is less data available regarding automated feedback specifically within therealm of FA. For example, Dalby and Swan (2018) integrated technology into their FA
feedback, but this was implemented in a mathematics course, rather than EFL, and reliedheavily on teacher input, so can hardly be considered automated. Sheard and Chambers(2014) used multiple-choice style questions to provide FA for EFL students’ writing, but the
only feedback this provided was whether or not answers were correct. More recently, Alharbi
et al. (2021) examined the effects of FA-based feedback given via Google Forms on EFL
learners, finding that they provided a positive impact on both teachers and students, but theirstudy focused on in-class assessments rather than curriculum-wide efforts. Therefore, it is notknown how impactful the automated feedback provided by a one-time curriculum-wide FA,such as the one proposed by this study will actually be.
Finally, there are a few studies that suggest that curriculum-wide FAs can have somepositive impact on language learners. For example, Mahshanian et al. (2019) suggest that a
FA focused specifically on learner strategies increased students’ summative EFL abilities
later in a curriculum-wide effort. Similarly, Kato (2009, 2020) found that a program that68

JACET Selected Papers Vol.9 (2023) 65-83

helped students to set goals and self-assess enabled L2 Japanese learners to attain higher
proficiency over the course of their university studies. However, other studies have found that
FA practices do not always impact student learning as intended or designed (e.g., Widiastuti
et al., 2020). Finally, Sakurai et al. (2021) examined the use of a curriculum-wide FA on L1
Japanese EFL learners’ final learning outcomes reporting that the FA seemed effective, but
only to a very minor degree. However, they were unable to take pre-treatment data, whichleaves the impact of their FA somewhat unclear.
Research Questions
Based on the suggestions of the studies mentioned above, it is possible that a one-time
FA given between first and second semester classes in a one-year EFL curriculum might havea positive impact on learners. However, it is unclear how strong the impact will be, and if
completely automated feedback will actually have any influence on learners. Therefore, thisstudy seeks to fill in the gaps of previous research by answering the following questions:
1. Does taking a formative assessment of first semester skills at the beginning of a second
semester course have a discernable impact on learning outcomes in an EGAP-based EFL
curriculum, i.e., positive change in TOEFL ITP® scores?
2. How well did the feedback work to encourage learners to revise their learning strategies,
i.e., review skills that they had not acquired well?
MethodCreating Formative Assessments
Two FAs were created for use at the beginning of the second semester at Tohoku
University in 2021, one for reading and writing courses, and one for speaking and listening
courses. According to the curriculum designated by Tohoku University, which is outlined in
Pathways to Academic English (students’ textbook; Spring, 2021), in the first semester
reading and vocabulary class, called “English A1,” students should have learned four skills:

69

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

word parts, synonym vocabulary, skimming and scanning, and paraphrasing andsummarizing. In the first semester speaking and listening class, called “English B1,” students
should have learned: note-taking while listening, orally summarizing from notes,
interrogatives and giving opinions, and phrasal verbs and idiomatic expressions. Each FA
consisted of 20 questions, ten of which were knowledge-based (i.e., based on specific phrases
or words that students were asked to learn), and ten were skills-based (i.e., reading or
listening questions that students should be able to solve if they had mastered theaforementioned skills). In an attempt to better observe students’ skimming and scanningability, students were also asked to report how long it took them to complete the skills portionof the A1 FA.
Automated feedback was given to each student using a modified Google Spreadsheet
and the grading and grade-sharing add-on Flubaroo (https://www.flubaroo.com/). A score foreach individual skill in the FAs was calculated, and one of three pieces of advice was given
for each skill based on the score. Students who scored over 80% for the questions on aparticular skill were told that they had mastered that skill to a satisfactory amount, students
who scored between 50% and 80% were told that they had some mastery, but that they should
work to improve the skill by using the materials and resources from Tohoku University andthose suggested by the Educational Testing Service (2019). These materials include a numberof reading, listening, ASR-based speaking, and online-writing activities that are provided toteachers and students though a designated website that accompanies the Pathways to
Academic English textbook, providing practice on the specific skills outlined within it(www.pathwaystoacademicenglish.com). Students who achieved less than 50% on a given
skill were told that they should work to improve the skill by reviewing their textbook andusing specific Tohoku University materials. The same guideline was followed for assessing
the skimming and scanning skill, but the cut-off points were set at a reported time of having

70

JACET Selected Papers Vol.9 (2023) 65-83

finished the reading section within 10 and 13 minutes, respectively, instead of a percentage ofcorrect answers.
In order to ensure that the feedback would be as low-threat and low-risk as possible, thefeedback was given with positive wording and focused on providing advice and guidance
rather than being critical of students (i.e., Kluger & De Nisi, 1996). For example, rather thantelling students that they did poorly at a particular skill, the feedback for poor performancewas “You have some ability to summarize and paraphrase but can still improve. However,
you did better at summarizing than paraphrasing. Consider rereading Chapter 4 and try
practice paraphrasing more by using the online paraphrasing tool found on the Pathways to
Academic English website (<link>).” General percentage scores for each skill were also given
to the students, but information regarding which questions they missed and the correct
answers was not given, as the purpose of the FAs were to alert teachers and students toindividual skill levels and encourage students to improve in areas that they were lacking in(e.g., Black & Wiliam, 2009). Teachers could also request their students’ FA scores and they
were distributed to several teachers. Since the FAs were designed to be taken on a volunteer
basis, we were largely unable to hold students accountable to future action (i.e., Gardner,2012), but instructors could adjust their courses based on FA scores as necessary.
Implementation and ParticipantsStudents studied in their A1 and B1 classes in the first semester (from the middle ofApril to the beginning of August) and then had a summer break from early August to the endof September. The FAs were created in Google Forms and students were given a link to them.
Teachers told students that they could do the assessments outside of class time and that theywould be given feedback based on their scores. Second semester teachers of A2 and B2classes, respectively, were asked to give the links to their students, but some refused (see theDiscussion section for further details). Furthermore, some teachers gave the students the link71

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

and told them participation was on a voluntary basis, while others told their students thatparticipation was required for their classes. Accordingly, participation varied greatly,
resulting in the following participation rate: 232 students took only the A1 FA, 196 took the
B1 FA, 217 took both, and 1774 took neither.
Assessing Meaningfulness
We monitored the impact of the FAs mostly through two means: (a) change in TOEFL
ITP® scores, and (b) student surveys (see Appendix A). Students at Tohoku University took
the TOEFL ITP ® in the middle of May, and again at the beginning of December. Thoughboth tests were conducted part-way through the first and second semesters, respectively, weused the change in these scores (i.e., the delta scores: posttest score – pretest score) as anindicator of learning outcomes. We used the change in TOEFL ITP® test scores because thetest is designed to evaluate EGAP, which the curriculum is based on, and the skills identified
for inclusion in the Tohoku University curriculum generally correlate well to higher TOEFL
ITP® scores (Sakurai et al., 2021). We also conducted student surveys in an attempt to: (a)verify that the feedback encouraged some students to review their A1 and B1 skills, and (b) toglean information that could be useful in subsequent implementations of the FAs at Tohoku
University.
In order to ensure the validity of the tests, Pearson’s correlation analyses wereconducted on the tests and both pre- and post-test TOEFL ITP® scores, repeated both for theoverall scores as well as for the scores of the specific section with which the skills were
designed to impact (i.e., the correlation between A1 tests and reading section scores, and thecorrelation between B1 tests and listening sections scores were also checked). Furthermore,
we used a repeated MANOVA test of students’ TOEFL ITP® test scores using their scores on
the FAs to check whether or not proper skill acquirement in the first semester was correlatedwith greater improvement. Finally, we used an ANOVA test to check for differences in the72

JACET Selected Papers Vol.9 (2023) 65-83

delta scores of students who took one, both, and neither of the FAs, with Tukey’s HD for
post-hoc analysis.
Survey responses were checked for signs that some students followed the suggestionsin the feedback and worked to improve. Furthermore, we simply observed students’ opinions
on the test in general to gather suggestions for improvement.
Results
Scores Analysis
The correlation between the FAs and the TOEFL ITP® test scores are shown in Table
1. In general, it seems that the tests had moderate correlation to the TOEFL ITP® tests,
meaning that the tests were likely at least somewhat representative of EGAP ability.
Furthermore, the repeated MANOVA showed that higher scores on both the A1 FA and theB1 FA were significantly correlated with more improvement on the TOEFL ITP® test
sections they were designed to aid (p = .002, r =.04 for the former, p < .001, r = .28 for thelatter).

Table 1
Correlations (r) Between FAs and TOEFL ITP® Scores
Formative
Assessment
A1 FA
B1 FA

Pre-test
Overall Score Section Score.391**.294**.575**.518**

Post-test
Overall Score Section Score.391**.358**.586**.544**

Note. The “section score” means the reading section for the A1 test and the listening section
for the B1 test. *p < .05, **p < .001

The average amount of improvement for students who did and did not take the FAs isdisplayed in Figures 1 and 2 below. Our results showed significant differences between thegroups; F [3, 2318] = 2.64, p = .05. Post hoc comparison showed that the only difference was73

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

between students who took both formative assessments and those that took neither; p < .001.
Furthermore, students who took the B1 FA showed more improvement on the listening
section than those who did not (p = .04), and students who took the A1 FA improved on thereading section more than those who did not (p = .05).

Figure 1
Average Overall TOEFL ITP® Delta Scores per Group
18
15.3

16
14

13.7

13.6

A1 Onlyn = 232

B1 Onlyn = 196

11.1

12
10
8
6
4
2
0

None Taken
n = 1774

Bothn = 217

Figure 2
Average TOEFL ITP® Section Delta Scores per Group
3.5
3

2.9

2.7

2.9

2.8

2.5
2
1.5

0
-0.5

1

0.8

1
0.5

1.3

1.2

-0.11
None Taken

0.43

0.18

0.01
A1 Only

B1 Only

ListeningSW

74

Reading

BothJACET Selected Papers Vol.9 (2023) 65-83

Student Surveys
When students were provided feedback, they were also sent a link to an online surveythat asked them to evaluate the difficulty of the FAs, the usefulness of the FA feedback, andtheir willingness to improve the skills that they were told needed improvement (according to
the FA feedback). The majority of the questions were 5-point Likert scale questions(Questions 2, 3, 4, 6, 7, and 8), but two open-ended questions were also used to ask about thereason for participation and for general comments (Questions 1 and 5). A list of the exact
questions can be found in Appendix A.
Participation in the survey was voluntary and only seventeen students answered.
According to the answers to Question 1, most students took the FAs because they wereassigned (75%) and some students (25%) took them voluntarily to review or check theknowledge and skills they learned in the previous semester. Regarding the difficulty of theFAs (Question 2), about half of the students (52.9%) chose a neutral answer and 35.3% of thestudents rated them as “difficult.” Combined with the fact that the average percentage ofcorrect answers on the FAs were 66.28% for the A1 FA and 59.01% for the B1 FA, and thatthe scores correlated to TOEFL ITP® scores in general, it seems that the level of the FAs was
mostly appropriate, as they were neither overly difficult nor easy. One participant commented
that it would be more helpful if students could see the FA questions and answers for review.
Table 2 summarizes the results for the Likert scale questions (Questions 3, 4, 5, 7, and 8). The
means in Questions 3 (3.65) and Question 4 (4.35) show that the students self-reported asvery keen on improving their shortcomings indicated by the FA feedback. Additionally, theytend to think that taking the TOEFL ITP® test is effective in attaining EGAP skills.

Table 2
Survey Results
75

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

Q3: Thefeedback was
useful.
M
SD

3.65
0.97

Q4: I want toimprove basedon thefeedback.
4.35
0.59

Q6: Pathways
skills arehelpful for
EGAP.
3.59
0.84

Q7: Pathways
skills arehelpful for
TOEFL.
3.35
0.90

Q8: TOEFL iseffective for
EGAP.
4.06
0.87

Discussion
Considering the results holistically, it seems that the FAs reported on in this study had adiscernable impact on learning outcomes, i.e., TOEFL ITP® score improvement, albeit a low
level of influence. While the tests and skills within them did seem to align with EGAP ability
and mastering the skills does seem to relate to higher TOEFL ITP® scores in general, it
seems that only a small amount of this is due to the FAs, specifically. While this is somewhat
expected because the FAs are one small piece of a much larger curriculum, part of the reason
the impact level is low could be partially due to the fact that some students may not have beenlearning ideally, i.e., properly acquiring skills during their first semester and practicing overthe summer vacation so as not to lose them. However, another reason that the FAs had low
levels of influence is also partially due to the feedback and lack of follow up, which should be
addressed in the future.
The survey results, in combination with the statistical analysis, suggest that the FAs did
encourage some students to adjust their learning strategies, but could stand to be improved ina number of ways, especially in terms of feedback. First, many students did indicate that they
intended to work on the particular skills that the FA suggested they needed to review. In thissense, the feedback was effective in proving positive and encouraging comments that might
have initiated self-reflective student learning. However, survey participants did indicate thatthey wanted the FA feedback to include not only their scores, but also the questions andanswers so that the students can review them. We did not include this information to decrease
the potential of threatening students’ self-esteem, following Kluger and De Nisi (1996), but it

76

JACET Selected Papers Vol.9 (2023) 65-83

seems that students would have found this information an important and relevant way toreview and learn from their mistakes. Furthermore, providing information regarding whichquestions they missed could be impactful in that it can help students to figure out their exact
shortcomings, which would hopefully promote student learning. Therefore, we intend toinclude this information in addition to individualized feedback in future iterations of our FAs.
Furthermore, though students indicated that they wanted to review based on the information
they received from the FAs, based on the survey results, we should include monitored practiceactivities so that we can ensure continual progress, as suggested by Gardner (2012).
With regards to how well the FAs influenced teaching strategies, there is some evidence
that the FAs contributed a small amount to the curriculum. Specifically, though the FAs were
given on a voluntary basis and some teachers did not participate, some teachers reported
adjusting their second semester classes based on the FA outcomes. Specifically, two A2teachers added word parts self-study and weekly quizzes to their classes based on students
scoring low on that skill in some of their classes, and one B2 teacher added review of phrasal
verbs and idiomatic expressions to one class due to low scores from a particular class.
Although Gardner (2012) argues that it is best for students to be personally responsible forachieving the learning outcomes and this is a point that should be considered in the future, theFAs did seem to help at least some of the teachers to encourage students to acquire the skills
that they could not in their first semester.
Relatedly, another area in which the Tohoku University FAs need improvement isteacher outreach. Although several teachers gave their students the links for the FAs, themajority did not. Perhaps some teachers did not see the advantage to the FAs or did not want
to be held responsible for reviewing material from the previous semester, but this is beyond
the scope of this study and currently unclear. However, this issue should be addressed, as it
could also be seen as unfair to students who would have like to have taken the FAs but were77

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

not given the opportunity. In coming years, we hope to convince more teachers to consider
FA scores and create a fairer system for providing students with links for the FAs.
Based on the above results and discussion, the question of whether or not the Tohoku
University FAs are meaningful requires some personal interpretation. On the one hand, theycaused a statistically significant impact on TOEFL ITP® scores, but on the other hand, theamount it seemed to help was quite small; a few points overall with a small effect size.
Creating and administering the FAs was quite time consuming, and some might argue that thesmall impact is not worth the effort. However, we believe that anything that we can do thatwill help students even a little bit, or that will help even just a few students is meaningful.
Furthermore, since we have confirmed that the FAs are helpful to at least some students, wefeel that continuing to improve them to increase their impact is a worthwhile effort, and wewill endeavor to do so based on the results of this study.
Conclusion
Based on the above results and discussion, we can conclude that the Tohoku University
FAs were successful in their intended purpose of improving EGAP learning. Specifically, the
TOEFL ITP® scores of learners who took the FAs improved more than their peers who did
not, and the FAs improved the targeted skill areas of the TOEFL ITP® test that they wereintended to. Furthermore, some students and teachers revised their teaching and learning
strategies based on the results of the FAs, which likely helped students to achieve higher testscores. However, we did find that the impact of the FAs was quite minimal as only someteachers and students used them, and the effect size of the difference in scores was small,albeit statistically significant. Furthermore, we discovered several points for improvement to
the FAs in the future: i.e., teacher outreach to ensure more students can take the FAs,
including lists of questions and answers with results, and providing more specific andmonitorable practice for areas that students had difficulty with. Overall, we feel that the FAs

78

JACET Selected Papers Vol.9 (2023) 65-83

were meaningful on a curriculum-wide scale and hope that we can increase their impact in the
future.
Acknowledgements
We would like to thank the anonymous reviewers for their comments and suggestions,which helped us to improve the quality of this manuscript.
References
Alharbi, A. S., Alhebshi, A. A., & Meccawy, Z. (2021). EFL students’ and teachers’
perceptions of Google Forms as a digital formative assessment tool in Saudi secondary
schools. AWEJ Special Issue on CALL, 7, 140–154.
https://dx.doi.org/10.24093/awej/call7.10
Bardovi-Harlig, K., & Stringer, D. (2010). Variables in second language attrition: Advancing
the state of the art. Studies in Second Language Acquisition, 32(1), 1–45.
https://doi.org/10.1017/S0272263109990246
Black, P. & Wiliam, D. (2009). Developing the theory of formative assessment. Educational
Assessment, Evaluation and Accountability, 21(1), 5–31.
https://doi.org/10.1007/s11092-008-9068-5
Dalby, D., & Swan, M. (2018). Using digital technology to enhance formative assessment inmathematics classrooms. British Journal of Educational Technology, 50(2), 832–845.
https://doi.org/10.1111/bjet.12606
Dick, W., & Carey, L. (1996). The systematic design of instruction (4th edition). Harper
Collins.
Educational Testing Service. (2019). Official guide to the TOEFL ITP® assessment series.
ETS.

79

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

Gao, J., & Ma, S. (2020). Instructor feedback on free writing and automated corrective
feedback in drills: Intensity and efficacy. Language Teaching Research.
https://doi.org/10.1177%2F1362168820915337
Gardner, J. (2012). Assessment and learning (2nd edition). SAGE.
Guskaroska, A. (2020). ASR-dictation on smartphones for vowel pronunciation practice.
Journal of Contemporary Philology, 3(2), 45–61.
https://doi.org/10.37834/JCP2020045g
Harris, M., & McCann, P. (1994). Assessment: Handbook for the English classroom.
Heinemann Publishers.
Kato, F. (2009). Student preferences: Goal-setting and self-assessment activities in a tertiary
education environment. Language Teaching Research, 13(2), 177–200.
Kato, F. (2020). Strategies for growing and enhancing university-level Japanese programs(Ed. R. Spring). Routledge.
Ketabi, S., & Ketabi, S. (2014). Classroom and formative assessment in second/foreign
language teaching and learning. Theory and Practice in Language Studies, 4(2), 435–
440. https://doi.org/10.4304/tpls.4.2.435-440
Kluger, A.N., & De Nisi, A. (1996). The effects of feedback interventions on performance: A
historical review, a meta-analysis, and a preliminary feedback intervention theory.
Psychological Bulletin, 119(2), 254–284.
Link, S., Dursun, A., Karakaya, K., & Hegelheimer, V. (2014). Towards better ESL practices
for implementing automated writing evaluation. CALICO Journal, 31(3), 323–344.
Mahshanian, A., Shogi, R., & Bahrami, M. (2019). Investigating the differential effects offormative and summative assessment on EFL learners’ end-of-term achievements.
Journal of Language and Teaching Research, 10(5), 1055–1066.
https://dx.doi.org/10.17507/jltr.1005.19

80

JACET Selected Papers Vol.9 (2023) 65-83

Sakurai, S., Spring, R., & Takebayashi, S. (2021). The Tohoku method: A new academic
English curriculum at Tohoku University and its preliminary evaluation. The JACET
60th Commemorative International Convention Proceedings (p.53–54). Available athttps://jacet.org/jacet60th/wp-content/uploads/2021/08/60th_proceedings.pdf
Shang, H. (2022). Exploring online peer feedback and automated corrective feedback on EFL
writing performance. Interactive Learning Environments, 30(1), 4–16.
https://doi.org/10.1080/10494820.2019.1629601
Sheard, M. K., & Chambers, B. (2014). A case of technology-enhanced formative assessment
and achievement in primary grammar: How is quality assurance of formative
assessment assured? Studies in Educational Evaluation, 43, 14–23.
https://doi.org/10.1016/j.stueduc.2014.02.001
Spring, R. (ed.) (2021). Pathways to academic English. Tohoku University Press.
Spring, R., & Tabuchi, R. (2022). The role of ASR Training in EFL pronunciation
improvement: An in-depth look at the impact of treatment length and guided practiceon specific pronunciation points. Computer Assisted Language Learning Electronic
Journal (CALL-EJ), 23(3), 163-185. http://callej.org/journal/23-3/SpringTabuchi2022.pdf
Widiastuti, I. A. M. S., Murkminatien, N., Prayogo, J. A., & Irawati, E. (2020). Dissonancebetween teachers’ beliefs and practices of formative assessment in EFL classes.
International Journal of Instruction, 13(1), 71–84.
Wu, X., Liu, X., & Chen, L. (2022). Reducing EFL learners’ error of sound deletion with
ASR-based peer feedback. In W. Jia, Y. Tang, R. S. T. Lee, M. Herzog, H. Zhang, T.
Hao, & T. Wang (Eds.) Emerging technologies for education. SETE 2021 (pp. 178–
189). Springer. https://doi.org/10.1007/978-3-030-92836-0_16

81

Ryan Spring, Shizuka Sakurai
The Impact of a Curriculum-based Formative Assessment on EGAP

Appendices
Appendix A: Survey Questions Used in this Study (translated from Japanese)
Question 1: Why did you take the FAs? (open-ended)

Question 2: Choose only one item that best matches the difficulty of the FAs:
1. Very difficult, 2. Difficult, 3. Neither, 4. Easy, 5. Very easy

Question 3: I received useful information about my English proficiency by reading the FA
feedback.
1. Strongly disagree, 2. Disagree, 3. Neither, 4. Agree, 5. Strongly agree

Question 4: I want to improve my English by working on my weak skills based on the FA
feedback.
1. Strongly disagree, 2. Disagree, 3. Neither, 4. Agree, 5. Strongly agree

Question 5: Please write your comments or requests in detail about the FAs. (open-ended)

Question 6: The English skills in Pathways (the textbook for English classes, A1, A2, B1, and
B2) are helpful in acquiring EGAP. (Example skills: Word Parts, Synonym Vocabulary,
Skimming & Scanning, Note-taking while listening, Phrasal Verbs & Idiomatic Expressions,
Collocations, and Tone of Voice)
1. Strongly disagree, 2. Disagree, 3. Neither, 4. Agree, 5. Strongly agree

Question 7: The English skills in Pathways (the textbook for English classes, A1, A2, B1, and
B2) are helpful for the TOEFL® test. (Example skills: the same as Question 6)

82

JACET Selected Papers Vol.9 (2023) 65-83

1. Strongly disagree, 2. Disagree, 3. Neither, 4. Agree, 5. Strongly agree

Question 8: Taking the TOEFL ITP® test is effective in acquiring EGAP.
1. Strongly disagree, 2. Disagree, 3. Neither, 4. Agree, 5. Strongly agree

83Submission Guidelines

JACET International Convention Selected Papers, Vol. 9
SUBMISSION GUIDELINES
Manuscripts for the JACET International Convention Selected Papers (JACET Selected Papers) will
only be accepted by online submission. Please read the following guidelines carefully.
Important Dates:
Submission Form Open: September 20, 2022
Submission Deadline: 11:59 PM Japan Standard Time, November 1, 2022
Submission Form:
https://www.jacet.org/selected-papers-submission2022/
A.

Requirements

1.

The paper must be based on a presentation (oral or regular poster presentation) given at the
JACET 61st International Convention (Online, 2022) and the first contributor must be a memberof JACET. All other contributors must have also presented the work at the JACET 61st
International Convention.

2.

A paper based on a plenary lecture may be submitted as an Invited Paper (by invitation only).

B.

Editorial Policy

1.

JACET Selected Papers, a refereed, open-access electronic journal, encourages submission of thefollowing:-Research Articles on pedagogy and topics of significance to teachers of English-Symposium Papers on relevant issues to teachers of English (one per symposium)-Practitioner Reports to share findings and insights

2.

Manuscripts submitted to JACET Selected Papers must not have been previously published, norshould they be under consideration for publication elsewhere.

3.

Manuscripts which do not conform to the guidelines will not be considered for review.

4.

Only one paper can be submitted by each contributor.

5.

The Editorial Board of JACET Selected Papers reserves the right to make editorial changes inany manuscript accepted for publication to enhance clarity or style. The corresponding author
will be consulted if the changes are substantial.

6.

Paper offprints will not be provided.

C.

Guidelines

1.

Manuscripts, including abstract, references, figures, tables, and appendices, should be formatted
to A4 size and not exceed 8,000 words for Research articles, 5,500 words for Symposium Papers,
and 5,000 words for Practitioner Reports.

2.

All manuscripts must be written in English.
86

3.

All submissions to JACET Selected Papers must conform to the requirements of the Publication
Manual of the American Psychological Association, 7th edition.
3.1 Prepare manuscripts using the template provided by the Editorial Board.
3.2 Use 12-point Times New Roman font.
3.3 Leave margins of 2.5 cm on all sides of every page (A4 size, 210 mm × 297 mm or 8.27 in
× 11.7 in). There are 26 lines to a page.
3.4 Do not justify right margins.
3.5 Do not use running heads.
3.6 For anonymity in the peer review process, submit papers without the author name(s).
3.7 Include the title, an abstract (no more than 200 words), and keywords (no more than five
keywords; a multiple-word key phrase can be counted as one keyword).
3.8 Acknowledgements should not be included at the time of submission.
3.9 For pagination, use Arabic numerals placed in the upper right-hand corner of each page.
3.10 In order to guarantee anonymity in the review process, both the author name(s) and their
publication information should be substituted with ***** throughout the entire manuscript
including references.

D.

Submission Procedure1.

All contributors must complete a submission form on the JACET website, which can be accessedfrom the Submission Guidelines.

2.

Contributors must follow the instructions below.
2.1 Convert the Word file manuscript into PDF format, saving it under the author’s full name asin the following examples: suzukikaoru or smithkerry. Submit the PDF file by clicking“choose file” on the submission form.
2.2 If there are more than four authors, write all authors’ information on a separate file and send
to the JACET office by e-mail: Author names, affiliation, membership number, postal code,
address, telephone number, and e-mail address.
(JACET e-mail address: jacet@zb3.so-net.ne.jp)
2.3 Do not include a cover sheet.

E.

Contributor’s Responsibility

1.

Contributors are responsible for the content of their manuscripts.

2.

Contributors are responsible for obtaining permission to reproduce any material such as figures
and tables for which they do not own the copyright, and for ensuring that the appropriate
acknowledgements are included in their manuscript.

F.

Copyright

1.

JACET holds the copyright of the articles published in JACET Selected Papers.

2.

Anyone, including the author(s), who wishes to reproduce or republish an article, must obtain
permission from JACET. Also, it should be clearly stated that JACET holds the copyright.
87