Developing a Smartphone-based
Integrated Student Assessment Data
Collection System to Enhance
Yamagata University's Education
Katsumi Senyo
Shigeru Asano
Koji Fujiwara
Takao Hashizume
October 24, 20172017 Assessment InstituteSlides are available!
Go to Google and search “Yamagata OIRE”

HereOr type https://ir.yamagata-u.ac.jp/AI2017.pdf
Or scan with QR Code Reader

2Agenda1. About Us
2. Educational Stages in Japan
3. Key Characteristics ofHigher Education in Japan
4. Assessment Planat Yamagata University (YU)
5. Assessment via Smartphonea. YU-Portal Demonstration

6. Conclusions

KojiShigeruKatsumiTakao

3

Where is Yamagata Prefecture?

• 2.5 hours by bullet train• 70 min by airplane4

Yamagata University (YU)■Type: National University• multi-campus (4 campuses)■Students: 8,800• 7,500 undergraduates• 1,700 new incoming/year• 1,300 graduates■Full-time Faculty: 900■Six Distinct Colleges:• Agriculture• Literature and Social Science• Education, Arts and Science• Sciences• Medicine• Engineering••

Yamagata City – Tsuruoka City: 60 miles
Yamagata City – Yonezawa City: 30 miles

5

Educational Stages in JapanStageYearAges

13-4

24-5

35-6

Elementary16-7(Compulsory)

27-8

38-9

49-10

510-11611-12Junior High112-13(Compulsory)

213-14314-15115-16

216-17317-18Kindergarten

Senior HighNotes:• Continuation Rates

Junior High → Senior High: 97%+
Senior High → College*: 57%• Graduation Rates

Senior High: 99%
College (in 4 Years): 90%+

• No Grade Skipping• No Dual Enrollment

Higher Education↓* Degree-Granting Institutions

6Assessment in Japanese Higher Education
 Required by accrediting agencies••••••

Quality assurance(continuous quality improvement in education)
Program review
General education assessment
How to collect
Program-level assessment
the data?
「Direct」 vs. 「Indirect」
Struggling with “closing the loop”

 Research Grant Program by MEXT*•

Our proposal is accepted!!
($700,000/year)* Ministry of Education, Culture, Sports, Science and Technology

7Key Characteristics of Higher Education in Japan Enrollment caps are set by MEXT.


BudgetTuition# of Fac/Staff

 Students must decide on their major when
they apply for admission.
••••

Can’t change the major after enrollment
No double majors
No pre- program (e.g., Pre-Med, Pre-Vet, etc.)
Almost impossible to transfer to another school8Assessment at Yamagata University

9Assessment Plan at Yamagata UniversityGeneral EdLMS

Specialized EdGoal

Are students
learning properly?

Do students
meet our Goal?

FocusInterview

Freshman

Sophomore

JuniorSeniorAlumni
SurveyEmployer
Survey Combination of Direct (Own developed test carry out bysmartphone App 3 times) and Indirect (Course Evaluation,GPA, Surveys, etc.) Measurement for SLOA.
SLOA: Student Learning Outcome Assessment

10

Assessment Plan (contd.) Focus more on Direct Assessment• Internally Developed Tests:• Quantitative andLogical Reasoning• Mathematics• Physics• Chemistry• Biology

 Multiple-choice Tests
with IRT Models
 For each subject:• Up to 5 questions• 5 min. to complete• Memorizing vs. Understanding• Memorizing facts does not guarantee
understanding of a concept.
• e.g., Force Concept Inventory (FCI,Physics/Mechanics )※ IRT = Item Response Theory11

What do we assess?
Naive Conception: frequently misunderstoodconcepts in physics field
Which is the correct force direction in a parabolic motion?

Typical mistake
Force 1

Correct Answer
Force 412

What do we assess?
Naive Conception: frequently misunderstoodconcepts in physics field
Which path is observed in a horizontal plane?

Typical mistakes
Trajectory 2 and 3

Correct Answer
Trajectory 413

What do we assess?
Naive Conception: frequently misunderstoodconcepts in physics field
Which is the fastest at ground level with the same initial velocity?

Typical mistake5

Correct Answer
All the same

14

Assessment Plan (contd.) Big Five Personality Test• 70 Yes-No Questions• Localized version developed byProfessor Chieko and Yoshihiro Murakami• Openness to Experience• Conscientiousness• Extraversion• Agreeableness• Emotional Stability15

Assessment via Smartphone
 FACT: 99+% of YU students own asmartphone with Bluetooth4.0.
 We would like to…

a. Collect assessment data more effectively.
b. Provide the personalized info. more efficiently.
•
At YU, the data/info. is not centralized.
c. Track classroom attendance more cost-effectively.
•Previously:
NFC-based reader &IC chip-embedded student ID card•
Ongoing:
Bluetooth4.0(BLE) based beacon &students’ smartphone

Much info. collected & provided
16

Smartphone Application: YU Portal
 Important Info. is now centralized!!

17

Main Features



Test
SurveyPersonalized Class Schedule
Class Attendance Record
Personalized Notification•ClassSchedulee.g., Class Cancellations

Semester Course
Schedule

Test

Attendance
Record18

YU Portal Demonstration

19

Preliminary Results: Biology
Studied at HS
生物選択No formal study
生物非選択

500

No. of Students

400

300

200

100

0-2.1

-1.5

-0.9-0.3

0.3

0.9

1.5

2.1

Level of Understanding:θ

20

Conclusions
 First Japanese University to implement direct
assessment via smartphone for all studentbody Ongoing development with initial results
positive Results help with continuous improvement

21

Acknowledgements
The smartphone App isdeveloped withThis project is
supported byMEXT, Japan.

Special Thanks (in alphabetical order) to:••••••••

Shinobu Araki
Douglas Gloag
Takahiro Iijima
Nobuhiro Matsuzaka
Koto Sato
Tatsuya Suzuki
Eriko Watanabe
Junichiro Yasuda

andAll Students, Faculty,
Staff & Administration
at Yamagata University22