Gaze-based minimal virtual reality paradigm for tracking developing
sensitivity to dyadic interactions

Sho Tsuji, IRCN Babylab, The University of Tokyo
Part 1: Establishing the interactive learning advantage
In the first part of our project, we established that infants learn better from interactive than
non-interactive agents, and that interactivity is an especially important cue for young learners.
Using gaze-contingent eye-tracking, we designed a virtual agent that reacted (or did notreact) to infants’ gaze, and who taught them new words (Figure 1).

Figure 1. Virtual agent experimental design

Our findings show that young learners are able to learn these new words from the reactiveagent, but no the non-reactive agent. We also established that this effect is especiallypronounced around the age of 1 year and becomes less relevant with increasing age. Moreover,the interactivity of an agent was more important than another important cue to agency, thehuman-likeness of its face.
Part 2: Exploring the mechanisms underlying the interactive learning advantage
After establishing the importance of interactivity for language learning, the second part of ourproject sought to assess the mechanisms behind this interactive learning advantage. To thisend, we developed a two-way gaze-contingent eye-tracking paradigm, in which infant andexperimenter mutually track each other’s eye movements in an abstract, screen-mediatedsetting.
This setup allows to measure the complex
dynamics of interaction in real time, but in atightly controlled experimental settings. We
Figure 2. Two-way eyetracking experimental setup

have established that infants can distinguishbetween interactive and non-interactive events based on the on-screen dynamics alone, andwithout any other cue to agency. Interactive events lead to more frequent and longer
encounters, which is a potential mechanism underlying the social learning advantage. Future
study will link these findings to language learning outcomes and provide important insights
on the design of interactive AI agents for maximizing learning success.
Part 3: Improving remote online assessment of infant gaze
Our project fell into the pandemic, which prevented in-person testing for over a year. In thisperiod, we worked on improving the methods surrounding automatic, web-cam basedassessment of infant gaze. Different from lab studies, remote, at-home assessment of infantgaze is noisy, since environmental settings are less controlled, no expert experimenter ispresent, and gaze needs to be extracted from webcam data instead of a high-quality eyetracking device. We therefore identified a wide range of sources of noise from infant webcam
data, replicated these sources in an adult lab study, and identified which of these sources
impacted the performance of automatic coding algorithms the most. Based on these insights,
we are now developing ways to mitigate noise in infant online studies.
Future outlook
We are planning to expand this project on a multi-modal scale, by not only tracking gaze, butalso gestures and vocal signals during child-adult interaction. This will provide theoreticalinsights into the mechanisms sustaining communication as well as a basis for designing infantfocused conversational AI agents.
Peer-reviewed journal publications

Issard, C., Tsuji, S., & Cristia, A. (2022). Infants’ preference for speech is stable acrossthe first year of life: Meta-analytic evidence. Accepted for publication in Infancy.
Phillipsen, A., Tsuji, S., & Nagai, Y. (2022). Quantifying developmental and individualdifferences in spontaneous drawing completion among children. Frontiers inPsychology, 13, 783446. doi: 10.3389/fpsyg.2022.783446
Philippsen, A. K., Tsuji, S., & Nagai, Y. (2022). Simulating developmental andindividual differences of drawing behavior in children using a predictive-coding model.

Frontiers in Neurorobotics, 16, 856184.
Havron, N., Lovcevic, I., Kee, M. Z. L., Chen, H., Chong, Y. S., Daniel, M., Broekman,B. F. P., & Tsuji, S. (2022). The effect of older sibling, postnatal maternal stress, andhousehold factors on language development in two- to four-year-old children.

Developmental Psychology. Developmental Psychology, 58(11), 2096–2113.
Zaadnoordijk, L., Buckler, H., Cusack, R., Tsuji, S., & Bergmann, C. (2021). A global
perspective on testing infants online: introducing ManyBabies-AtHome. Frontiers inPsychology, 3811.
Gasparini, L., Tsuji, S., & Bergmann, C. (2022). Ten easy steps to conducting
transparent, reproducible meta-analyses for infant researchers. Infancy, 1-29.