Jun.-Prof. Dr.-Ing. Shoya Ishimaru
https://shoya.io· Trippstadter Str. 122, 67663 Kaiserslautern, Germany · Born on 21.09.1991 in Ehime, JapanSummary
Shoya Ishimaru is a Junior Professor in Computer Science at the University of Kaiserslautern and the directorof Psybernetics Lab, an interdisciplinary research group
investigating Human-Computer Interaction, Machine
Learning, and Cognitive Psychology toward amplifying human intelligence. He has co-founded Immersive
Quantified Learning Lab and Alphaben in order to distribute practical applications using his research results.

Shoya received his Bachelor’s and Master’s degree inEngineering from Osaka Prefecture University in 2014and 2016, respectively. He received his PhD in Engineering with an award summa cum laude from the Universityof Kaiserslautern in 2019. He has also received the titleof MITOU Super Creator, which is given to outstanding
young developers (around 10 people per year) by the
Ministry of Economy, Trade, and Industry in Japan.

Background
Human-Computer Interaction, Learning Analytics, Wearable Computing, and Pattern Recognition

Academic Work Experience
Visiting Associate Professor, Osaka Metropolitan University

2022 – current

Junior Professor (PI), University of Kaiserslautern

2021 – current

Researcher, Keio Media Design Research Institute2014 – current

Visiting Researcher, Osaka Prefecture University

2016 – 2022

Research Associate, University of Kaiserslautern

2016 – 2019

Industrial Work Experience
Co-Founder Chief Research Officer (CRO), Alphaben GbR

2021 – current

Senior Researcher, German Research Center for Artificial Intelligence (DFKI)

2019 – current

Researcher, German Research Center for Artificial Intelligence (DFKI)

2016 – 2019

Internship, Part-Time Job, and Service
Mentor (PM), Mitou Junior2020 – current

Engineering Mentor, Life is Tech, Inc.

2013 – 2015

Engineering Internship, Recruit Holdings Co.

2015

Engineering Internship, Livesense Inc.

2014

Engineering Internship, Cookpad Inc.

2014

Software Engineer, Campus. Inc.

2012 – 2013

Part-Time Software Engineer, Paperboy&co. Inc

2012

Engineering Internship, Hatena Co.

2012

Education
PhD in Engineering, University of Kaiserslautern, Germany

2016 – 2019

Meta-Augmented Human: From Physical to Cognitive Towards Affective State Recognition
Reviewed by Prof. Andreas Dengel, Prof. Paul Lukowicz, and Prof. Koichi Kise
Master of Engineering, Graduate School of Engineering, Osaka Prefecture University, Japan

2014 – 2016

Eyewear Computing for Cognitive Activity Recognition
Reviewed by Prof. Koichi Kise, Prof. Tomoharu Nakashima, and Prof. Hisao Ishibuchi
Bachelor of Engineering, Osaka Prefecture University, Japan
Activity Recognition with Google Glass: Combining Head Motion and Eye Blink Frequency
Reviewed by Prof. Koichi Kise and Prof. Kai Kunze

2010 – 2014

Acquired Grants as PI
JSPS Grant-in-Aid for Young Scientists (B), 31 K EUR (4.16 M JPY)

FY 2017 – 2019

Eyetifact: Platform for Eyewear Data Conversion and Its Application to Activity Recognition in the Wild
IPA MITOU Exploratory IT Human Resources Project, 18 K EUR (2.30 M JPY)

FY 2015

Shin’ on kei: Development of a System for Visualizing the Mental State

Acquired Grants as Co-PI
DFG ANR JST International Call on Artificial Intelligence, 757 K EUR (93 M JPY)

2020 – 2022

Learning Cyclotron
JSPS Fostering Joint International Research (B), 142 K EUR (18.82 M JPY)

FY 2020 – 2022

International Collaborative Research on Intelligence Augmentation for Human Learning
JSPS Grant-in-Aid for Scientific Research (B), 142 K EUR (17.81 M JPY)

FY 2020 – 2022

Sensing and Actuation to Accelerate Human Knowledge I/O
JSPS Grant-in-Aid for Scientific Research (C), 37 K EUR (4.68 M JPY)

FY 2017 – 2019

Generating Artificial Eye Movements on a Document for an Objective Readability Measurement

Teaching
Collaborative Intelligence, University of Kaiserslautern (Co-lecturing with Prof. Dengel)

SS 2019 - 2022

Completed Supervision
11 Master Theses

Hussain, Jacob, Brishtel, Santhosh, Holub, Baitemirov, Steinert, Sinha, Vadiraja, Garg, Rohde

1 Bachelor Thesis

Steinert

6 Project/Seminar Students
7 Internship Students

Roy, Herurkar, Liu, Ghosh, Kumar, Garg
Yamanda, Ohbayashi, Watanabe, Maruichi, Allison, Gomez, Inal

Conference Organization
Co-Organizer for LeCycl 2022 (UbiComp Satellite Workshop)

2022

Program Committee for Augmented Humans 2022

2022

Program Committee for Augmented Humans 2021

2021

Program Co-Chair for Augmented Humans 20202020

Co-Organizer for EyeWild 2019 (UbiComp Satellite Tutorial)

2019

Program Committee for PerPersuasion 2019 (PerCom Satellite Workshop)

2019

Program Committee for READ 20182018

Co-Organizer for Application Developer Festival 2015

2015

Reviewer for ACM IMWUT, IEEE Access, IJDAR, MDPI Mathematics, IPSJ, CHI ’22 ’21 ’20, IUI ’21, AHs ’21,
ISWC ’20, MobileHCI ’20, PerCom-PerPersuasion ’19, READ ’18, ICDAR-HDI ’17

Selected Awards and Honorable Activities
Best Presentation Award at Asian CHI Symposium 2020

2020

PhD with an award summa cum laude (with highest praise)

2019

Poster Track Honorable Mention at UbiComp/ISWC 20182018

Gave an invited talk “Cognitive Cyborgs” at TEDxNagouaU

2017

Certificated as MITOU Super Creator by Ministry of Economy, Trade, and Industry in Japan

2016

President’s Honor in Osaka Prefecture University, Japan

2012, 2015

References
Prof. Dr. Prof. h.c. Andreas Dengel
Prof. Dr. Koichi Kise

Executive Director, DFKI. Andreas.Dengel@dfki.deProfessor, Osaka Prefecture University. kise@cs.osakafu-u.ac.jp

Journal Papers
1. Chenhao Chen, Yutaka Arakawa, Ko Watanabe and Shoya Ishimaru. “Quantitative Evaluation System
for Online Meetings Based on Multimodal Microbehavior Analysis”. Sensors and Materials 34 (8), pp.
3017–3027, 2022.
2. Andrew Vargo, Shoya Ishimaru, Md. Rabiul Islam, Benjamin Tag and Koichi Kise. “Obtaining Labels forIn-the-Wild Studies: Using Visual Cues and Recall”. IEEE Pervasive Computing 21 (1), pp. 9–17, 2022.
3. Ko Watanabe, Yusuke Soneda, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa, Andreas Dengel andShoya Ishimaru. “DisCaaS: Micro Behavior Analysis on Discussion by Camera as a Sensor”. Sensors 21(17), p. 5719, 2021..
4. Iuliia Brishtel, Anam Ahmad Khan, Thomas Schmidt, Tilman Dingler, Shoya Ishimaru and Andreas Dengel.
“Mind Wandering in Multimodal Reading Setting: Behavior Analysis & Automatic Detection using EyeTracking and EDA Sensor”. Sensors, 20 (9), p. 2546, 2020.
5. Oliver Amft, Florian Wahl, Shoya Ishimaru and Kai Kunze. “Making Regular Eyeglasses Smart”. IEEE
Pervasive Computing, 14 (3), pp. 32–43, 2015.

Book Chapter
1. Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, Pascal Klein, Jochen Kuhn andAndreas Dengel. “Augmented Learning on Anticipating Textbooks with Eye Tracking”. Positive Learningin the Age of Information (PLATO), pp. 387–398, 2018.

Conference Papers
1. Kanta Yamaoka, Ko Watanabe, Koichi Kise, Andreas Dengel and Shoya Ishimaru. ‘Experience is the Best
Teacher: Personalized Vocabulary Building Within the Context of Instagram Posts and Sentences fromGPT-3”. In Proceedings of the 2022 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication (UbiComp ’22 Adjunct), 2022, to appear.
2. Haruki Suzawa, Ko Watanabe, Masakazu Iwamura, Koichi Kise, Andreas Dengel and Shoya Ishimaru.
“Supporting Smooth Interruption in a Video Conference by Dynamically Changing Background Music
Depending on the Amount of Utterance”. In Proceedings of the 2022 ACM International Joint Conference
on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’22 Adjunct), 2022, to appear.
3. Andrew Vargo, Motoi Iwata, Mathilde Hutin, Sofiya Kobylyanskaya, Ioana Vasilescu, Olivier Augereau, KoWatanabe, Shoya Ishimaru, Benjamin Tag, Tilman Dingler, Koichi Kise, Laurence Devillers and AndreasDengel. “Learning Cyclotron: An Ecosystem of Knowledge Circulation”. In Proceedings of the 2022 ACM
International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp’22 Adjunct), 2022, to appear.
4. Ko Watanabe, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa and Shoya Ishimaru. “How do Programmers Use the Internet? Discovering Domain Knowledge from Browsing and Coding Behaviors”. In
Proceedings of the 8th IEEE International Conference on Smart Data (SmartData ’22), 2022, to appear.
5. David Dzsotjan, Kim Ludwig-Petsch, Sergey Mukhametov, Shoya Ishimaru, Stefan Küchemann and JochenKuhn. “The Predictive Power of Eye-Tracking Data in an Interactive AR Learning Environment”. In
Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing:Adjunct Publication (UbiComp ’21 Adjunct), pp. 467-471, 2021.
6. Pramod Vadiraja, Jayasankar Santhosh, Hanane Moulay, Andreas Dengel and Shoya Ishimaru. “Effectsof Counting Seconds in the Mind while Reading”. In Proceedings of the 2021 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’21 Adjunct), pp.
486-490, 2021.
7. Prerna Garg, Jayasankar Santhosh, Andreas Dengel and Shoya Ishimaru. “Stress Detection by Machine
Learning and Wearable Sensors”. In Proceedings of the 26th International Conference on Intelligent User
Interfaces Companion (IUI ’21), pp. 237–240, 2021.
8. Pramod Vadiraja, Andreas Dengel and Shoya Ishimaru. “Text Summary Augmentation for Intelligent
Reading Assistant”. In Proceedings of the 2nd Augmented Humans International Conference (AHs ’21),
pp. 319–321, 2021.

9. Shoya Ishimaru, Takanori Maruichi, Koichi Kise and Andreas Dengel. “Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback”. In Extended Abstracts of the 2020 CHI Conferenceon Human Factors in Computing Systems (Asian CHI Symposium ’20), 2020.
Best Presentation Award
10. Jayasankar Santhosh, Shoya Ishimaru and Andreas Dengel. “Generating Heatmap for Unknown Documents
Towards Readability Measurement”. In Proceedings of the 25th International Conference on Intelligent
User Interfaces Companion (IUI ’20), pp. 47–48, 2020.
11. Takanori Maruichi, Shoya Ishimaru and Koichi Kise. “Self-confidence Estimation on Vocabulary Tests withStroke-level Handwriting Logs”. In Proceedings of the 15th IAPR International Conference on Document
Analysis and Recognition (ICDAR HDI ’19), pp. 18–22, 2019.
12. Shoya Ishimaru, Takanori Maruichi, Manuel Landsmann, Koichi Kise and Andreas Dengel. “Electrooculography Dataset for Reading Detection in the Wild”. In Proceedings of the 2019 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’19 Adjunct), pp.
85–88, 2019.
13. Shoya Ishimaru, Ko Watanabe, Nicolas Großmann, Carina Heisel, Pascal Klein, Yutaka Arakawa, Jochen
Kuhn and Andreas Dengel. “HyperMind Builder: Pervasive User Interface to Create Intelligent InteractiveDocuments”. In Proceedings of the 2018 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication (UbiComp ’18 Adjunct), pp. 357–360, 2018.
14. Soumy Jacob, Shoya Ishimaru and Andreas Dengel. “Interest Detection While Reading Newspaper Articles
by Utilizing a Physiological Sensing Wristband”. In Proceedings of the 2018 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’18 Adjunct), pp.
78–81, 2018.
Poster Track Honorable Mention
15. Dayananda Herurkar, Shoya Ishimaru and Andreas Dengel. “Combining Software-Based Eye Tracking anda Wide-Angle Lens for Sneaking Detection”. In Proceedings of the 2018 ACM International Joint Conference
on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’18 Adjunct), pp. 54–57, 2018.
16. Yuya Ohbayashi, Shoya Ishimaru, Andreas Dengel and Koichi Kise. “Investigating Gaze and PhysiologicalFeatures to Estimate Comprehension on E-learning Video Lectures”. In Proceedings of the 1st international
interdisciplinary Symposium on Reading Experience and Analysis of Documents (READ ’18), 2018.
17. Jayasankar Santhosh, Shoya Ishimaru and Andreas Dengel. “Estimating Fixation Durations for Each Wordin Documents towards Readability Measurement”. In Proceedings of the 1st international interdisciplinary
Symposium on Reading Experience and Analysis of Documents (READ ’18), 2018.
18. Nicolas Großmann, Iuliia Brishtel, Shoya Ishimaru, Andreas Dengel, Carina Heisel, Pascal Klein and Jochen
Kuhn. “iQL - Immersive Quantified Learning Lab”. In Proceedings of the 1st international interdisciplinary
Symposium on Reading Experience and Analysis of Documents (READ ’18), 2018.
19. Soumy Jacob, Shoya Ishimaru, Syed Saqib Bukhari and Andreas Dengel. “Gaze-Based Interest Detection on
Newspaper Articles”. In Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-BasedInteraction (ETRA ’18), p. 4, 2018.
20. Iuliia Brishtel, Shoya Ishimaru, Olivier Augereau, Koichi Kise and Andreas Dengel. “Assessing Cognitive
Workload on Printed and Electronic Media using Eye-Tracker and EDA Wristband”. In Proceedings of the23rd International Conference on Intelligent User Interfaces Companion (IUI ’18), p. 45, 2018.
21. Shoya Ishimaru, Soumy Jacob, Apurba Roy, Syed Saqib Bukhari, Carina Heisel, Nicolas Großmann, MichaelThees, Jochen Kuhn and Andreas Dengel. “Cognitive State Measurement on Learning Materials by Utilizing
Eye Tracker and Thermal Camera”. In Proceedings of the 14th IAPR International Conference on Document
Analysis and Recognition (ICDAR HDI ’17), pp. 32–36, 2017.
22. Shoya Ishimaru, Kensuke Hoshika, Kai Kunze, Koichi Kise and Andreas Dengel. “Towards Reading Trackers
in the Wild: Detecting Reading Activities by EOG Glasses and Deep Neural Networks”. In Proceedings ofthe 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication(UbiComp ’17 Adjunct), pp. 704–711, 2017.
23. Koichi Kise, Olivier Augereau, Yuzuko Utsumi, Masakazu Iwamura, Kai Kunze, Shoya Ishimaru andAndreas Dengel. “Quantified Reading and Learning for Sharing Experiences”. In Proceedings of the2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication(UbiComp ’17 Adjunct), pp. 724–731, 2017.

24. Shoya Ishimaru and Andreas Dengel. “ARFLED: Ability Recognition Framework for Learning and Education”. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication (UbiComp ’17 Adjunct), pp. 339–343, 2017.
25. Sabine Hoffmann, Helga Tauscher, Andreas Dengel, Shoya Ishimaru, Sheraz Ahmed, Jochen Kuhn, CarinaHeisel and Yutaka Arakawa. “Sensing Thermal Stress at Office Workplaces”. In Proceedings of the 5thInternational Conference on Human-Environment Systems (ICHES ’16), 2016.
26. Shoya Ishimaru. “Cognitive State Recognition for Developing Anticipating Textbook”. In Proceedings ofthe 9th International Conference on Mobile Computing and Ubiquitous Networking (ICMU ’16), pp. 1–2,
2016.
27. Shoya Ishimaru, Syed Saqib Bukhari, Carina Heisel, Jochen Kuhn and Andreas Dengel. “Towards anIntelligent Textbook: Eye Gaze Based Attention Extraction on Materials for Learning and Instruction inPhysics”. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication (UbiComp ’16 Adjunct), pp. 1041–1045, 2016.
28. Shoya Ishimaru, Kai Kunze, Koichi Kise and Andreas Dengel. “The Wordometer 2.0: Estimating theNumber of Words You Read in Real Life using Commercial EOG Glasses”. In Proceedings of the 2016 ACMInternational Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp’16 Adjunct), pp. 293–296, 2016.
29. Shoya Ishimaru, Tilman Dingler, Kai Kunze, Koichi Kise and Andreas Dengel. “Reading Interventions:
Tracking Reading State and Designing Interventions”. In Proceedings of the 2016 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’16 Adjunct), pp.
1759–1764, 2016.
30. Christophe Rigaud, Thanh-Nam Le, J-C Burie, Jean-Marc Ogier, Shoya Ishimaru, Motoi Iwata and Koichi
Kise. “Semi-Automatic Text and Graphics Extraction of Manga using Eye Tracking Information”. InProceedings of the 2016 12th IAPR Workshop on Document Analysis Systems (DAS ’16), pp. 120–125, 2016.
31. Shoya Ishimaru and Koichi Kise. “Quantifying the Mental State on the Basis of Physical and Social Activities”. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and UbiquitousComputing: Adjunct Publication (UbiComp ’15 Adjunct), pp. 1217–1220, 2015.
32. Kai Kunze, Katsuma Tanaka, Shoya Ishimaru, Yuji Uema, Koichi Kise and Masahiko Inami. “MEME:Eye Wear Computing to Explore Human Behavior”. In Proceedings of the 2015 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’15 Adjunct), pp.
361–363, 2015.
33. Kai Kunze, Katsutoshi Masai, Masahiko Inami, Ömer Sacakli, Marcus Liwicki, Andreas Dengel, Shoya Ishimaru
and Koichi Kise. “Quantifying Reading Habits: Counting How Many Words You Read”. In Proceedings ofthe 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15), pp.
87–96, 2015.
34. Kai Kunze, Kazutaka Inoue, Katsutoshi Masai, Yuji Uema, Sean Shao-An Tsai, Shoya Ishimaru, KatsumaTanaka, Koichi Kise and Masahiko Inami. “MEME: Smart Glasses to Promote Healthy Habits for KnowledgeWorkers”. In ACM SIGGRAPH 2015 Emerging Technologies (SIGGRAPH ’15), p. 17, 2015.
35. Katsuma Tanaka, Shoya Ishimaru, Koichi Kise, Kai Kunze and Masahiko Inami. “Nekoze!: Monitoringand Detecting Head Posture while Working with Laptop and Mobile Phone”. In Proceedings of the 9thInternational Conference on Pervasive Computing Technologies for Healthcare (Pervasive Health ’15), pp.
237–240, 2015.
36. Shoya Ishimaru, Kai Kunze, Katsuma Tanaka, Yuji Uema, Koichi Kise and Masahiko Inami. “Smart Eyewearfor Interaction and Activity Recognition”. In Extended Abstracts of the 2015 CHI Conference on Human
Factors in Computing Systems (CHI ’15 EA), pp. 307–310, 2015.
37. Shoya Ishimaru, Kai Kunze, Koichi Kise and Masahiko Inami. “Position Paper: Brain Teasers - Toward
Wearable Computing That Engages Our Mind”. In Proceedings of the 2014 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’14 Adjunct), pp.
1405–1408, 2014.

38. Shoya Ishimaru, Kai Kunze, Yuji Uema, Koichi Kise, Masahiko Inami and Katsuma Tanaka. “SmarterEyewear: Using Commercial EOG Glasses for Activity Recognition”. In Proceedings of the 2014 ACM
International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp’14 Adjunct), pp. 239–242, 2014.
39. Shoya Ishimaru, Jens Weppner, Andreas Poxrucker, Paul Lukowicz, Kai Kunze and Koichi Kise. “Shiny:
An Activity Logging Platform for Google Glass”. In Proceedings of the 2014 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’14 Adjunct), pp.
283–286, 2014.
40. Shoya Ishimaru, Jens Weppner, Kai Kunze, Andreas Bulling, Koichi Kise, Andreas Dengel and Paul Lukowicz. “In the Blink of an Eye: Combining Head Motion and Eye Blink Frequency for Activity Recognition
with Google Glass”. In Proceedings of the 5th Augmented Human International Conference (AH ’14), pp.
150–153, 2014.
Cited by 150 times
41. Shoya Ishimaru, Kai Kunze, Yuzuko Utsumi, Masakazu Iwamura and Koichi Kise. “Where Are You Looking
At? - Feature-Based Eye Tracking on Unmodified Tablets”. In Proceedings of the 2013 2nd Asian Conference
on Pattern Recognition (ACPR ’13), pp. 738–739, 2013.
42. Kai Kunze, Shoya Ishimaru, Yuzuko Utsumi and Koichi Kise. “My Reading Life: Towards Utilizing Eyetracking on Unmodified Tablets and Phones”. In Proceedings of the 2013 ACM International Joint Conference on
Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp ’13 Adjunct), pp. 283–286, 2013.
43. Kai Kunze, Yuki Shiga, Shoya Ishimaru and Koichi Kise. “Reading Activity Recognition Using an Off-theShelf EEG – Detecting Reading Activities and Distinguishing Genres of Documents”. In Proceedings of the2013 12th International Conference on Document Analysis and Recognition (ICDAR ’13), pp. 96–100, 2013.

Miscellaneous (without peer review)
1. Shoya Ishimaru, Takanori Maruichi, Andreas Dengel and Koichi Kise. “Confidence-Aware Learning Assistant”. In arXiv preprint arXiv:2102.07312, 2021.
CV compiled on September 1st, 2022.