# Abstract thematic roles in infants' representation of social events

Liuba Papeo, Sofie Vettori, Emilie Serraille, Catherine Odin, Farzad Rostami, Jean-Rémy Hochmann

► To cite this version:

Liuba Papeo, Sofie Vettori, Emilie Serraille, Catherine Odin, Farzad Rostami, et al.. Abstract thematic roles in infants' representation of social events. Current Biology, 2024, 34 (18), pp.4294-4300.e4. 10.1016/j.cub.2024.07.081. hal-04872279

HAL Id: hal-04872279

https://hal.science/hal-04872279v1

Submitted on 7 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Abstract thematic roles in infants' representation of social events

Liuba Papeo*, Sofie Vettori, Emilie Serraille,
Catherine Odin, Farzad Rostami & Jean-Rémy Hochmann*

Institut des Sciences Cognitives *Marc Jeannerod* -UMR5229, CNRS & Université Claude Bernard Lyon1, 67 Boulevard Pinel, 69675, Bron, France

*Lead contact: hochmann@isc.cnrs.fr

Twitter: @JR_Hochmann

**Please cite as:**

Papeo, L., Vettori, S., Serraille, E., Odin, C., Rostami, F., & Hochmann, J. R. (2024). Abstract thematic roles in infants’ representation of social events. *Current Biology*, *34*(18), 4294-4300.

10.1016/j.cub.2024.07.081
## Summary

Infants' thoughts are classically characterized as iconic, perceptual-like representations¹⁻³. Less clear is whether preverbal infants also possess a propositional language of thought, where mental symbols are combined according to syntactic rules, very much like words in sentences⁴⁻¹⁷. Because it is rich, productive and abstract, a language of thought would provide a key to explaining impressive achievements in early infancy, from logical inference to representation of false beliefs¹⁸⁻³¹. A propositional language—including a language of thought⁵—implies thematic roles that, in a sentence, indicate the relation between noun and verb phrases, defining *who acts on whom*; i.e., who is the *agent* and who is the *patient*³²⁻³⁹. Agent and patient roles are abstract in that they generally apply to different situations: whether *A* kicks, helps or kisses *B*, *A* is the agent, *B* is the patient. Do preverbal infants represent abstract agent and patient roles? We presented 7-month-olds (n=143) with sequences of scenes where the posture or relative positioning of two individuals indicated that, across different interactions, *A acted on B*. Results from habituation (Experiment 1) and pupillometry paradigms (Experiments 2-3) demonstrated that infants showed surprise when roles eventually switched (*B acted on A*). Thus, while encoding social interactions, infants fill in an abstract relational structure that marks the roles of *agent* and *patient*, and that can be accessed *via* different event scenes and properties of the event participants (body postures or positioning). This mental process implies a combinatorial capacity that lays the foundations for productivity and compositionality in language and cognition.

## Results

### *Experiment 1 – Effects of role assignment in a habituation paradigm*

We created renderings of a girl and a boy in various postures that could belong to one of two different types: agent-like or patient-like⁴⁰⁻⁴². In the agent-like type, the body leaned forward with limbs forward or outstretched, resulting in a dynamic posture; in the patient-like type, the body leaned backward, with limbs close to, or along the trunk, in a less dynamic posture. Stimuli were created by pairing one individual in an agent-like posture, with the other in a patient-like posture (Figure S1). Seven-month-old infants (n = 63) were habituated to images featuring the same dyad (e.g., the same boy in blue clothes and the same girl in orange clothes), where body postures changed from one trial to another but one individual (e.g., the boy) was always in an agent-like posture, and the other (e.g., the girl) was in a patient-like posture (Figure 1). Role assignment
(agent/patient) and side assignment (left/right) to the girl and the boy were counterbalanced across participants. Moreover, since role assignment implies the representation of a relation between the participants in the event (e.g., one acting on another), we predicted that role assignment would preferentially happen when the two appear to interact. To test this, 32 participants were presented with face-to-face dyads; the remaining 31 participants were presented with back-to-back dyads. Since infants represent face-to-face individuals as parts of a structured unit and back-to-back individuals as unrelated<sup>43-44</sup>, we predicted a stronger effect of role assignment (i.e., surprise) for role switch in the face-to-face condition.

The same number of infants reached the habituation criterion (a decrease by 50% of the average looking time across three consecutive trials) in the face-to-face (21 out of 32) and in the back-to-back condition (21 out of 31), with no difference between conditions in the number of trials required to reach habituation (face-to-face: 7.14 trials ±1.74 SD; back-to-back: 6.95 ±1.69; t<sub>40</sub> = 0.36, p = 0.72). After reaching the habituation criterion or after completing the 12 habituation trials, participants were tested on three pairs of trials, alternating between same-role trials, where role assignment was the same as in the habituation phase, and role-switch trials, where roles were swapped relative to the habituation phase. Half of the infants began the test phase with a same-role trial, half, with a role-switch trial.

As looking time data in test trials were not normally distributed (Shapiro-Wilk test; W = 0.72, p < 0.0001), raw values were log-transformed before entering a repeated-measures ANOVA with Role Assignment (same-role or role-switch relative to habituation trials) and Test-trial Pair (first, second, or third pair) as within-subjects factors, and Position (face-to-face or back-to-back) as a between-subjects factor. As shown in Figure 1, in the face-to-face condition only, infants looked significantly longer when there was a role switch than when role assignment was identical to the habituation, in the first test-trial pair that followed habituation. Once the switch was introduced with the first test-trial pair, there was no difference in looking times between same-role and role-switch in the remaining two test-trial pairs.

The statistics confirmed this pattern, showing a significant interaction between Role Assignment, Test-trial Pair and Position, *F*<sub>2,122</sub>= 3.16, *p* = .046, *η*<sub>p</sub><sup>2</sup> = 0.049. No other effect or interaction was significant (Role Assignment: *F*<sub>1,61</sub> = .34; *p* = 0.56; *η*<sub>p</sub><sup>2</sup> = .005; Test-trial Pair: *F*<sub>2,122</sub> = .030, *p* = .97, *η*<sub>p</sub><sup>2</sup> = <0.001, Position: *F*<sub>1,61</sub> = 3.34; *p* = .072; *η*<sub>p</sub><sup>2</sup> = .052; Role Assignment by Test-trial Pair:
$F_{2,122} = .16, p = .86, \eta_p^2 = .003$; Role Assignment by Position: $F_{1,61} = .017; p = .90, \eta_p^2 < .001$; Position by Test-trial Pair: $F_{2,122} = 1.03, p = .36, \eta_p^2 = .017$). To explain the interaction, we analysed the performance in each test-trial pair separately, and found a significant interaction between Role Assignment and Position for the first test-trial pair, $F_{1,61} = 5.92, p = .018, \eta_p^2 = 0.088$, but not for the second pair, $F_{1,61} = .21, p = .65, \eta_p^2 = 0.003$, or the third pair, $F_{1,61} = 1.60, p = .21, \eta_p^2 = 0.026$. In the first test-trial pair, infants looked longer at the stimuli in role-switch than in same-role trials, in the face-to-face condition ($t_{31} = 1.87, p = .035, M_{diff} = 0.31$, Cohen's *d* = 0.35), but not in the back-to-back condition ($t_{30} = -1.58, p = .94, M_{diff} = -0.29$, Cohen's *d* = -0.30). This difference was not significant for the other two test-trial pairs, for either the face-to-face or the back-to-back condition (all *p*s > .19).

## *Experiment 2 – Effects of role assignment in a pupillometry paradigm*

Seven-month-old infants (*n* = 48) were presented with sequences of images showing the same two individuals in various postures. In 83% of those images (standard trials), each individual appeared in the same role, as defined by the posture (e.g., the boy was the agent, the girl was the patient; Figure 2). In the remaining 17% of the images (deviant trials), roles switched (e.g., the boy was the patient, the girl was the agent). Half of the participants were presented with face-to-face dyads; the remaining saw the same dyads presented back-to-back.

As shown in Figure 2, pupil dilation in infants increased in the deviant trials that introduced a role switch relative to the more frequent standard trials. This effect only emerged when face-to-face – seemingly interacting– individuals were presented, but not when the two appeared to be unrelated (i.e., back-to-back). The statistics confirmed this observation. A cluster mass permutation test comparing face-to-face and back-to-back conditions did not reveal significant interaction (*p* > .17) between Position (face-to-face, back-to-back) and Trial Type (standard, deviant). However, a cluster mass permutation test analyzing infants tested in the face-to-face condition did reveal larger pupil dilation in deviant, compared to standard trials, between 183 ms and 1950 ms (*p* = .015). The same test analyzing infants tested in the back-to-back condition revealed no difference between deviant and standard trials (*p* > .21 at all timepoints).
Experiment 3 – Generalization from posture-based to position-based role assignment

Experiments 1-2 showed that roles are assigned on the basis of a conjunction of posture and
position: in a face-to-face configuration, infants encoded and distinguished between agent-like and
patient-like postures; disruption of such effect in the back-to-back configuration suggests that the
effects did not reflect mere violation of a posture-to-individual mapping (e.g., a violation of the
expectation, given previous trials, that the girl is more dynamic than the boy), as the same mapping-
process would apply to both the face-to-face and back-to-back conditions. To further prove that
infants’ role representations abstract away from the perceptual (e.g., postural) information that
triggers those representations, we tested the infants’ capacity to generalize role representations
based on postures to role representations based on positioning.

To test this, seven-month-old infants (n = 32) were presented with sequences of images showing the same two individuals in various postures. In 83% of those images (standard trials), the two individuals appeared in a face-to-face arrangement, just like in the face-to-face condition of Experiment 2, and each individual kept the same role, as defined by the posture (e.g., the boy was the agent and the girl was the patient). In the remaining 17% of the trials (deviant trials), the two individuals assumed the same neutral posture but their relative positioning changed: one faced the other who faced away (face-to-back; Figure 3). In this face-to-back configuration, only one faced the other and, therefore, only one was in the position to address/act on the other –in sum, to be the agent. In half of deviant trials (*same role*), the individual who was in the agent-like posture in standard trials faced toward the other (agent-like position), who turned away (patient-flip). In the other half of deviant trials (*role switch*), the individual who was in the patient-like posture in standard trials faced toward the other (agent-like position), who turned away (agent-flip). Thus, in deviant trials, while postures became uninformative (both were in the same neutral posture), role assignment could still be made based on the asymmetric positioning of the two individuals.

As shown in Figure 3, pupil dilation increased in the role-switch deviant trials (agent-flip) relative
to the same-role deviant trials (patient-flip). The effect in the time window of interest, selected a
*priori* based on the results of Experiment 2 (i.e., effect in the face-to-face condition), showed that
pupil dilation was larger for role-switch (agent-flip) compared to same-role (patient-flip) trials, t<sub>31</sub>
= 2.19, *p* = 0.018, Cohen’s *d* = 0.39 (paired *t*-tests). A cluster mass permutation test within the time
window of interest further revealed larger pupil dilation in role-switch (agent-flip) compared to
same-role (patient-flip) trials, between 300 and 1950 ms ($p = .009$).

Alternative accounts of the present results based on a different deployment of attention to face-to-face *vs*. back-to-back dyads (Experiments 1-2) or to the agent-like *vs*. the patient-like individual (Experiment 1-3), were excluded based on additional analyses showing no indication of such differences (see Figures S2 and S3).

Discussion

A key ingredient of a propositional language of thought is the capacity to represent the abstract relational structure of events, by attributing to entities abstract roles such as *agent* and *patient*, beyond situation-specific roles (e.g., *puncher* and *punchee*)⁵. Here we showed such capacity in young preverbal infants. We showed that, in dyadic social scenes, 7-month-old infants encoded visuo-spatial cues such as posture and relative positioning (where one stands relative to another), indicating *who acts on whom*; in doing so, they assigned the two actors to the categories of *agent* and *patient*. These categories showed a hallmark of abstraction, that is generalization, the property of a representation to apply to a range of variable instances, exemplars or cases⁴⁵⁻⁴⁶. Generalization here was evident from the fact that role assignment in one scene affected the same process in another scene. In particular, we presented infants with body dyads that varied constantly in terms of postures and implied movements and actions, but maintained consistent role assignment (e.g., the girl *looked like* the agent; the boy *looked like* the patient). Infants showed surprise (i.e., longer looking times in Experiment 1, and larger pupil dilation in Experiments 2-3) when roles eventually switched in test trials after habituation to a given agent-patient schema (Experiment 1), or in rare trials among frequent trials with the same agent-patient schema (Experiments 2-3).

These effects were especially found when two individuals appeared face-to-face (*vs*. back-to-back), a positioning that is spontaneously and consistently associated with social interaction⁴³⁻⁴⁴, ⁴⁷⁻⁴⁹. This circumstance supports the interpretation that infants assigned the participants to roles in a relation (i.e., in the interaction, the girl is the agent, the boy is the patient), besides a mere categorization by posture (i.e., the girl is in more dynamic and the boy in less dynamic postures). If surprise was solely the reaction to a change of posture, infants would have reacted to the change in every trial. Moreover, if surprise only reflected categorization by posture, the same reaction
would have been observed irrespective of the spatial relation between the two individuals (face-to-face or back-to-back). Thus, the sorting of postures in two categories only for seemingly interacting (face-to-face) individuals reveals that infants encoded one as the agent not just because it *looked like* an agent but because it looked like an agent in relation to another, occupying a specific place in a relational structure.

Experiment 3 goes further, showing that infants' representation of roles was abstract enough to be accessible not only across different types of agent-patient events (i.e., the different social-interaction events presented sequentially during an experiment, building up the surprise response), but also through different types of perceptual signals (postural and spatial relations). Here, infants showed surprise when the agent-like and the patient-like individuals in standard trials switched roles but in ways defined by relative positioning (the actor that previously assumed patient-like postures appears in the back agent-like position and the actor that previously assumed agent-like postures appears in the front patient-like position), relative to when they kept the same roles but in ways defined by relative position (the actor that previously assumed agent-like postures appears in the back agent-like position and the actor that previously assumed patient-like postures appears in the front patient-like position). In sum, role assignment based on postures generalized to role assignment based on relative positioning.

It remains to be studied whether the effects of role assignment observed here would generalize to non-social relations (an agent acting on an object or a physical interaction between inanimate objects), and to other roles. In processing visual scenes, preverbal infants are sensitive to agency⁵⁰⁻⁵², goals⁵²⁻⁵⁴, instruments³¹ and other roles (e.g., in causal events³⁷, ⁵⁵⁻⁵⁹), but it is unknown how abstract those roles are. The present findings pave the way to further research involving different types of events (social and nonsocial), participants (intentional animate agents and inanimate objects) and roles (*agent*, *patient*, *goal*, *recipient*, *instrument*, etc.).

In conclusion, remarkably congruent results, across three experiments and two independent methodologies, demonstrated that 7-month-old infants represent visual events according to a *who-to-whom* structure that encodes the abstract thematic roles of *agent* and *patient*. The ability to represent entities and relations in an abstract format in terms of thematic roles is key to the productivity and compositionality of language and reasoning⁶, ⁴⁵, ⁶⁰⁻⁶¹. As we showed here, this capacity is available before an external language develops. Representations of abstract thematic
roles such as *agent* and *patient* exist outside and beyond language⁶²⁻⁶³, reflecting a core aspect of human cognition that precedes the acquisition of a lexicon and morphosyntax markers, and may generally bias how individuals represent events across languages and cultures.

**Acknowledgements.** Funding for this project was provided by a European Research Council Starting Grant awarded to L.P. (Project: THEMPO, Grant Agreement 758473). S.V. was supported by a Marie Skłodowska-Curie action individual fellowship (MSCA-IF 101108756). F.R. was supported by the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon within the program "Investissements d'Avenir".

**Author contributions.** L.P. and J-R.H. designed and supervised research; L.P. provided funding; E.S. and C.O. performed research; S.V., C.O., F.R. and J-R.H. contributed analytic tools and data analysis; L.P., S.V. and J-R.H. wrote the paper.

**Declaration of interests.** The authors declare no competing interest.

## Figure legends

**Figure 1. Stimuli, procedure and results of Experiment 1. Top:** Examples of stimuli and design of Experiment 1 in the face-to-face and back-to-back condition. In both conditions, each stimulus stayed on screen until the infant looked away for longer than 2s. Stimuli were interleaved by a screen displaying an animated colorful shape, to attract the infant's attention to the screen. This screen remained until a fixation of 100 ms occurred, then the next stimulus was shown. Each infant was presented with stimuli showing the same agent-patient structure (in the example: the boy is the agent, the girl is the patient, based on their postures). In the habituation phase, stimuli were shown until the infant habituated or saw all 12 habituation-trials. Then, six test trials were shown, alternating between same-role trials (trials with the same structure as in the habitation: e.g., boy is the agent and girl is the patient) and role-switch trials (e.g., girl is the agent and boy is the patient).

**Bottom:** Results of Experiment 1. A dishabituation effect (i.e., longer looking times for the role-switch vs. same-role test-trial) was observed in the first test-pair and only in the condition with face-to-face stimuli. *p* < 0.05. See also Figures S2 and S3.
**Figure 2. Stimuli, procedure and results of Experiment 2.** *Top:* Extract of stimuli and design of Experiment 2 in the face-to-face and back-to-back condition. In both conditions, each infant was presented with a sequence where standard (i.e., more frequent) trials showed the same agent-patient structure (in the example: the boy is the agent and the girl is the patient, based on their postures) and deviant (less frequent) trials introduced a role switch (e.g., the girl is the agent and the boy is the patient). Each stimulus appeared in a frame for 3000 ms, then disappeared. The next stimulus was shown when the infant fixed the area within the frame for 100 ms. *Bottom:* Results of Experiment 2. A larger increase in pupil diameter, relative to the baseline, was found in deviant trials between 183 and 1950 ms, than in standard trials, when dyads were presented face-to-face. No difference between standard and deviant trials was observed in response to back-to-back dyads. The horizontal black bar denotes a significant effect with *p* < 0.05. See also Figures S2 and S3.

**Figure 3. Stimuli, procedure and results of Experiment 3.** *Top:* Extract of stimuli and design of Experiment 3. Standard trials showed the same agent-patient face-to-face structure, with roles defined by postures. Deviant trials introduced a role switch (8.5% of all trials) or maintained the same role assignment (8.5% of all trials), based on position (to act on the patient, the agent must face towards them). *Bottom:* Results of Experiment 3. Pupil variations were analyzed in the time window, where an effect was observed in the face-to-face condition of Experiment 2. There was a larger increase in pupil diameter (compared to baseline) for role-switch deviant trials, relative to same-role deviant trials, between 300 ms and 1950 ms. The horizontal black bar denotes a significant effect with *p* < 0.05. See also Figure S3.
# STAR★METHODS

## RESOURCE AVAILABILITY

### Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Liuba Papeo (liuba.papeo@isc.cnrs.fr) or Jean-Rémy Hochmann (hochmann@isc.cnrs.fr).

### Materials availability

Stimuli can be found at https://osf.io/zcwng/.

### Data and code availability

* All data have been deposited and are publicly available as of the date of publication at https://osf.io/zcwng/.

* All original codes have been deposited and are publicly available as of the date of publication as of the date of publication at https://osf.io/zcwng/.

* Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

## EXPERIMENTAL MODEL AND SUBJECT DETAILS

*Experiment 1.* Sixty-four infants (age range (months, days): 7m 0d – 7m 30d; average: 7m 15d) participated in Experiment 1: 32 in the face-to-face condition and 32 in the back-to-back condition. Without directly comparable pilot studies, the number of participants was chosen to be larger than the minimal sample size (n = 17 for α = .05 and 1-β = .80; G*Power) required to obtain an effect size comparable to a similar previous habituation study³⁷ (Experiment 1: *d* = .7252). One infant in the back-to-back condition was excluded from the final analyses because of an outlier performance: their average looking time over all test trials (42.13 s) was above 3 *SD* from the mean (M = 7.06 s, SD = 10.88 s). Twenty additional infants were tested but their data were not analyzed as they did not complete the experiment (8 in the face-to-face condition; 12 in the back-to-back condition). Participants in this and the following experiments were recruited and tested in the *Babylab* of Institut des Sciences Cognitives *Marc Jeannerod* (Bron, France). Parents gave informed consent
before participation and received travel reimbursement. The study was approved by the ethics
committee CPP sud-est II.

*Experiment 2.* Forty-eight infants (age range: 7m 0d – 7m30d; average: 7m 13d) participated in Experiment 2. Without directly comparable pilot studies, a sample size of 24 per group was chosen as it is larger than the minimal sample size (n = 17 for *d* = .74, *α* = .05 and 1-β = .80; G*Power) to obtain a difference in pupil dilation between standard *vs*. deviant trials in a study using a similar paradigm²⁷. Two additional infants were tested but excluded for not providing a sufficient number of trials (see analysis).

*Experiment 3.* Thirty-two infants (age range: 7m 0d – 7m30d; average: 7m 17d) participated in Experiment 3. Nine additional infants were tested but excluded for not providing a sufficient number of trials (see analysis). The sample size of 32 was estimated with a power analysis based on the results of Experiment 2 (face-to-face condition). For this analysis, the averaged pupil-dilation values in the significant cluster (183-1950 ms) for the deviant and standard trials were compared with one-tail pairwise t-test (t(23) = 2.21, p = 0.01, *d* = 0.4502). Thirty-two is the minimal sample size to detect an effect of standard *vs*. deviant trials similar to Experiment 2 (*α* = .05; 1-β = .80; G*Power*).

METHOD DETAILS

Stimuli

*Face-to-face dyads:* 192 stimuli were created, each depicting two human bodies facing each other (Figures 1 and 2). Individual bodies were created with Daz3D (Daz Productions) and the image-processing toolbox of MATLAB (The MathWorks). To create these stimuli, we selected four models: a male dressed in blue, a male dressed in orange, a female dressed in blue, and a female dressed in orange. Following Hafri and colleagues⁴², postures were defined as agent-like (leaning forward, more dynamic) or patient-like (leaning backward, less dynamic). For each model, we created 24 unique agent-like postures and 24 patient-like postures, as well as their horizontal mirror-flipped version, resulting in 384 individuals (4 models, 48 postures, original and mirror-flipped). By randomly pairing agent-like and patient-like postures, we obtained 192 unique facing dyads (Figure S1). In creating the stimuli for this and the subsequent experiments, we favored the
control of visuospatial cues of agent and patient, with scant consideration of the semantic coherence
of the resulting action-event, considering that the young participants in this study had limited action
and event knowledge, but were rather sensitive to spatiotemporal cues and contingencies⁶⁴⁻⁶⁵. Each
posture was used eight times, one for each combination of color (blue or orange), side (left or right)
and gender (male or female). Each stimulus occupied an area of 379 x 500 pixels on the screen. At
a distance of 60 cm, the visual angle of the stimuli was 11'x11' with the two bodies at an equal
distance (25 pixels) from the center of the image. Each infant only saw a subset of the stimuli: the
boy in blue and the girl in orange, or *vice versa*.

*Back-to-back dyads:* Stimuli were identical to the facing dyads except for the relative positioning of the bodies: each body was flipped along the vertical axis so that the two appeared back-to-back. The mean distance between the two bodies in a dyad was matched across face-to-face and back-to-back stimuli. The centers of the two minimal bounding boxes that contained each figure of a dyad were placed at an equal distance from the center of the image. In addition, the distance between the closest points (extremities) of the two bodies was matched across face-to-face and back-to-back dyads (mean for face-to-face dyads: 95.33 pixels; SD = 10.0; mean for back-to-back dyads: 94.77 pixels ±10.55 SD; *t*(382) = 0.52, *p* = 0.60, two-tailed).

*Face-to-back dyads:* Four additional body-stimuli were created for Experiment 3 (a blue male, an orange male, a blue female, and an orange female), to be used in the deviant trials. The four new bodies were in the same neutral posture as one another (a standing posture with no cues of agent-like or patient-like role). One male and one female body dressed in different colors were combined in pairs where both faced in the same direction (leftward or rightward), yielding a face-to-back configuration.

## Procedure

*Experiment 1.* Infants sat on their parent's lap 60 cm away from a Tobii T60XL eye tracker (Tobi Technology, Stockholm, Sweden) in a dark room. We asked the parent to close their eyes. Stimulus presentation and recording of the eye-tracking data at 60Hz were controlled by the Psychtoolbox running in Matlab (The Mathworks, Natick MA). The experiment began after the calibration for eye-tracking, and consisted of up to 12 habituation trials and 6 test trials.

Each habituation and test trial began with a small central colorful animation to draw infant's attention to the center of the screen. After 200 ms of central fixation, the animation disappeared
and the experimental stimulus appeared. Display duration was controlled by the infant's behavior: the stimulus disappeared and the trial ended, when the participant looked away from the screen for more than 2 seconds consecutively.

In the habituation phase, each infant saw a randomly selected subset of 12 dyads with the same two characters and the same agent-patient structure. The role-gender mapping was counterbalanced across infants so that half saw the boy in agent-like postures, the others saw the girl in agent-like postures (see Figure 1). The side of the boy and the girl remained the same across all trials, and was counterbalanced across infants. After 6 trials, the average looking time on the last three trials was calculated to verified whether the infant had reached the habituation criterion, corresponding to an average looking time smaller than half the average looking time of the first three trials. The habituation phase ended when the habituation criterion was reached or after all 12 habituation trials were shown.

The test phase consisted of 6 trials alternating between trials with the same-role assignment as in the habituation phase, and trials with a role switch (i.e., the character that was in agent-like postures during habituation, now was in patient-like postures, and the character that was in patient-like postures during habituation, now was in agent-like postures). Half of the infants began the test phase with a “same role” trial; the other half began with a “role-switch” trial. The experiment lasted about 5 minutes.

*Experiment 2.* Each infant sat on their parent's lap 60 cm away from a Tobii T60XL eye tracker (Tobi Technology, Stockholm, Sweden) in a dark room. We asked the parent to close their eyes. Stimulus presentation and recording of the eye-tracking data at 60Hz were controlled by the Psychtoolbox running in Matlab. The experiment began after the calibration for eye-tracking and consisted of 60 trials. In each trial, a stimulus appeared within a central rectangular area delimited by a black perimeter, for the fixed duration of 3 sec. Stimulus presentation started automatically when the infant looked inside the rectangle for 100 ms consecutively.

For a participant, the gender and position of the agent-like and patient-like postures was the same in 5 out of 6 trials (standard trials: 83%). In standard trials, half of the infants saw the boy in agent-like postures and the girl in patient-like postures; half saw the opposite. Agent-like postures were on the left side for half of the participants and on the right for the other half. In deviant trials (1 out of 6; 17% of trials), the roles switched, so that the body in agent-like postures in standard trials,
now was in a patient-like posture, and the body in patient-like postures in standard trials now was
in an agent-like posture. Standard and deviant trials were presented in a pseudo-random order to
avoid two consecutive deviant trials.

Half of the participants (n = 24) were tested with face-to-face stimuli, and the other half (n = 24) with back-to-back stimuli. The experiment ended when all 60 trials were shown (~ 5 min), or when the infant expressed discomfort, or stopped looking at the screen.

*Experiment 3.* The set-up and procedure were identical to Experiment 2 for the standard trials. The experiment began after the calibration for eye-tracking and consisted of 96 trials. In each trial, a stimulus appeared within a central rectangular area delimited by a black perimeter, for a fixed duration of 3 sec. Stimulus presentation started automatically when the infant looked inside the rectangular areas, for 100 ms consecutively.

For each infant, the gender and position of the agent-like and patient-like postures was the same in 5 out of 6 trials (standard trials: 83%). In standard trials, half of the infants saw the boy in agent-like postures and the girl in patient-like postures; half saw the opposite. Agent-like postures were presented on the left for half of the participants and on the right for the other half. While all deviant trials presented face-to-back stimuli (17%), in half of those trials (*same role*), roles did not change as the individual who was in the agent-like posture in standard trials kept facing the other (agent-like position), who turned away (patient-like position). In the remaining deviant trials (*role switch*), there was a role switch as the individual who was in the patient-like posture in standard trials kept facing the other (agent-like position), who turned away (patient-like position). Standard and deviant trials were presented in a pseudo-random order to avoid two consecutive deviant trials. The experiment ended when all 96 trials were shown (~9 min), or when the infant expressed discomfort, or stopped looking at the screen.

## QUANTIFICATION AND STATISTICAL ANALYSIS

*Experiment 1.* Log-transformed looking times to test trials were analyzed in a repeated-measures ANOVA with Role Assignment (same role, role switch) and Test-Trial Pair (first, second, third) as within-subject factors and Condition (face-to-face, back-to-back) as a between-subjects factor.

*Experiment 2.* We defined as the area of interest, the central rectangle in which stimuli appeared. Trials for which pupil-diameter information was available for at least 60% of the total trial duration
and at least 100 ms in the baseline time window (see below) were considered “good”. To be included in the analysis, infants had to provide at least one good standard trial and one good deviant trial. In the face-to-face condition, the mean number of good trials was 30.25 ±14.36 *SD* (standard trials: 25.08 ±12.68; deviant trials: 5.17 ±2.62). In the back-to-back condition, the mean number of good trials was 25.13 ±12.58 (standard trials: 21.13 ±10.64; deviant trials: 4.00 ±2.22).

We analyzed changes in the right pupil size in each trial relative to a 100-ms baseline period that began with the stimulus onset. In each condition, a cluster-mass permutation test<sup>66-67</sup> was performed to analyze the time course of the difference in pupil dilation between deviant and standard trials. We compared the pupil dilation for deviant *vs*. standard trials at each time-point with a paired-sample *t*-test. Neighbouring timepoints showing an effect with a *t* value larger than a threshold corresponding to *P* < .10 were clustered and tested for significance using a permutation test, with 1,000 random permutations of the trial-type labels on the original data. The significance probability of the original clusters was computed as the number of times the shuffled data produced clusters with higher summed *t* values than the real data. A relatively low threshold was chosen in order to detect possibly weaker but long-lasting effects.

*Experiment 3*. The area of interest was the central rectangle in which stimuli appeared. The time-window of interest corresponded to the interval, in which the significant effect of standard *vs*. deviant trials was observed in the face-to-face condition of Experiment 2 (183-1950 ms). Trials for which information about the pupil diameter was available for at least 60% of the time window of interest and at least 100 ms in the baseline time window (see below) were considered “good”. To be included in the analysis, infants had to provide at least one good trial for each type of deviant trials: the *role-switch* type, in which the standard agent turned away (agent-flip), and the *same-role* type in which the standard patient turned away (patient-flip). The mean number of good trials was 3.56 ± 1.93 for *same-role* trials and 2.48 ±1.48 for *role-switch* trials.

We analyzed the right pupil-size change in each trial relative to a 100-ms baseline period that began with the stimulus onset. Changes in the pupil size relative to the baseline were averaged across time-points within the time-window of interest, separately for each type of deviant trial. A paired *t*-test (one-tail) tested whether role-switch deviant trials elicited larger pupil dilation than same-role deviant trials. Like in Experiment 2, a cluster-mass permutation test<sup>66-67</sup> was performed in the
time window of interest to analyze the time-course of the difference in pupil dilation between the
two type of deviant trials.

References

1. Bruner, J. S. (1964). The course of cognitive growth, American Psychologist, 1-15, 19.

2. Mandler, J. M. (1992). How to build a baby: II. Conceptual primitives. *Psychol. Rev.* **99**, 587.

3. Mandler, J. M., and Cánovas, C. P. (2014). On defining image schemas. *Lang. Cogn. 6*, 510–532 (2014).

4. Cesana-Arlotti, N., Martín, A., Téglás, E., Vorobyova, L., Cetnarski, R. and Bonatti, L.L. (2018). Precursors of logical reasoning in preverbal human infants. *Science* **359**, 1263–1266.

5. Quilty-Dunn, J., Porot, N., and Mandelbaum, E. (2022). The best game in town: The re-emergence of the language of thought hypothesis across the cognitive sciences. *Behav. Brain Sci.*, 1–55.

6. Fodor, J. A. (1975). *The language of thought* (Harvard university press).

7. Fodor, J. A. (2008). *LOT 2: The language of thought revisited* (Oxford University Press, USA).

8. Leahy, B. P., and Carey, S.E. (2020). The acquisition of modal concepts. *Trends Cogn. Sci.* **24**, 65–78.

9. Leahy, B., Huemer, M., Steele, M., Alderete, S., and Carey, S. (2022). Minimal representations of possibility at age 3. *Proc. Natl. Acad. Sci.* **119**, e2207499119.

10. Feiman, R., Mody, S., and Carey, S. (2022). The development of reasoning by exclusion in infancy. *Cognit. Psychol.* **135**, 101473.

11. Mody, S., and Carey, S. (2016). The emergence of reasoning by the disjunctive syllogism in early childhood. *Cognition* **154**, 40–48.

12. Hochmann, J.R., Tuerk, A.S., Sanborn, S., Zhu, R., Long, R., Dempster, M. and Carey, S. (2017). Children’s representation of abstract relations in relational/array match-to-sample tasks. *Cognit. Psychol.* **99**, 17–43.

13. Hochmann, J-R. (2022). Representations of abstract relations in infancy. *Open Mind* **6**, 291–310.

14. De Villiers, J. G., and Pyers, J. E. (2002). Complements to cognition: A longitudinal study of the relationship between complex syntax and false-belief-understanding. *Cogn. Dev.* **17**, 1037–1060.

15. Carey, S. (2023). Do nonlinguistic creatures deploy mental symbols for logical connectives in reasoning?. *Behavioral & Brain Sciences*, 46.

16. Hochmann, J-R. (2023). Incomplete Language of Thought in infancy. *Behavioral & Brain Sciences*, 46.

17. Canudas-Grabolosa, I., Martin-Salguero, A., and Bonatti, L. L. (2023). Natural logic and baby LOTH. *Behavioral & Brain Sciences*, 46.

18. Cesana-Arlotti, N., Kovács, Á. M., and Téglás, E. (2020). Infants recruit logic to learn about the social world. *Nat. Commun.* **11**, 5999.

19. Cesana-Arlotti, N., Varga, B., and Téglás, E. (2022). The pupillometry of the possible: an investigation of infants’ representation of alternative possibilities. *Philos. Trans. R. Soc. B* **377**, 20210343.
20. Ekramnia, M., Mehler, J., and Dehaene-Lambertz, G. (2021). Disjunctive inference in preverbal infants. *IsScience* **24**.

21. Téglás, E., Girotto, V., Gonzalez, M., and Bonatti, L. L. (2007). Intuitions of probabilities shape expectations about the future at 12 months and beyond. *Proc. Natl. Acad. Sci.* **104**, 19156–19159.

22. Téglás, E., Vul, E., Girotto, V., Gonzalez, M., Tenenbaum, J.B. and Bonatti, L.L. (2011). Pure reasoning in 12-month-old infants as probabilistic inference. *science* **332**, 1054–1059.

23. Téglás, E., and Bonatti, L. L. (2016). Infants anticipate probabilistic but not deterministic outcomes. *Cognition* **157**, 227–236.

24. Denison, S., Reed, C., and Xu, F. (2013). The emergence of probabilistic reasoning in very young infants: evidence from 4.5-and 6-month-olds. *Dev. Psychol.* **49**, 243.

25. Denison, S., and Xu, F. (2010). Twelve-to 14-month-old infants can predict single-event probability with large set sizes. *Dev. Sci.* **13**, 798–803.

26. Denison, S., and Xu, F. (2014). The origins of probabilistic inference in human infants. *Cognition* **130**, 335–347.

27. Hochmann, J-R., and Toro, J. M. (2021). Negative mental representations in infancy. *Cognition* **213**, 104599.

28. Stahl, A. E., and Feigenson, L. (2015). Observing the unexpected enhances infants’ learning and exploration. *Science* **348**, 91–94.

29. Baillargeon, R., Scott, R. M., and He, Z. (2010). False-belief understanding in infants. *Trends Cogn. Sci.* **14**, 110–118.

30. Kovács, Á. M., Téglás, E., and Endress, A. D. (2010). The social sense: Susceptibility to others’ beliefs in human infants and adults. *Science* **330**, 1830–1834.

31. Adibpour, P., and Hochmann, J-R. (2023). Infants’ understanding of the causal power of agents and tools. *Proceedings of the National Academy of Sciences*, 120(50), e2309669120.

32. Fillmore, C. J. (1968). “The case for case.” in *Universals in Linguistic Theory*, (New York: Holt, Rinehart and Winston), p. 210.

33. Calvin, W. H., and Bickerton, D. (2000). *Lingua ex machina: Reconciling Darwin and Chomsky with the human brain* (MIT press).

34. Wilson, V. A., Sauppe, S., Brocard, S., Ringen, E. J., Daum, M. M., Wermelinger, S., ... and Zuberbuehler, K. (2023). Primate origins of human event cognition. *bioRxiv*, 2023-11.

35. Wilson, V. A., Zuberbühler, K., and Bickel, B. (2022). The evolutionary origins of syntax: Event cognition in nonhuman primates. *Science Advances*, 8(25), eabn8464.

36. S. Pinker, 1984. Language Learnability and Language Development, Cambridge, MA, USA: Harvard University Press.

37. Leslie, A. M., and Keeble, S. (1987). Do six-month-old infants perceive causality? *Cognition* **25**, 265–288.

38. Golinkoff, R. M. (1975). A comparison of reading comprehension processes in good and poor comprehenders. *Read. Res. Q.*, 623–659.

39. Golinkoff, R. M., and Kerr, J. L. (1975). Infants’ Perception of Semantically Defined Action Role Changes in Filmed Events. *Merrill-Palmer Quarterly of Behavior and Development*, 24(1), 53-61.

40. Hafri, A., Trueswell, J. C., and Strickland, B. (2018). Encoding of event roles from visual scenes is rapid, spontaneous, and interacts with higher-level visual processing. *Cognition* **175**, 36–52.
41. Vettori, S., Odin, C., Hochmann, J-R., and Papeo, L. (2024) Who to whom? Perceptual grounding of thematic/semantic role assignment in social events. *Journal of Experimental Psychology: General*.

42. Hafri, A., Papafragou, A., and Trueswell, J. C. (2013). Getting the Gist of Events: Recognition of Two-Participant Actions from Brief Displays. *J. Exp. Psychol. Gen.* **142**, 880–905 (2013).

43. Goupil, N., Papeo, L., and Hochmann, J-R. (2022). Visual perception grounding of social cognition in preverbal infants. *Infancy* **27**, 210–231.

44. Goupil, N., Rayson, H., Serraille, E., Massera, A., Ferrari, P. F., Hochmann, J-R., and Papeo, L. (2023). Visual preference for socially relevant spatial relations in humans and monkeys, *Psychological Science*, 35, 10.1177/09567976241242

45. Hummel, J. E., and Holyoak, K. J. (2003). A symbolic-connectionist theory of relational inference and generalization. *Psychol. Rev.* **110**, 220–264.

46. Lambon Ralph, M. A., and Patterson, K. (2008). Generalization and differentiation in semantic memory: insights from semantic dementia. *Ann. N. Y. Acad. Sci.* **1124**, 61–76.

47. Papeo, L. (2020). Twos in human visual perception. *Cortex* **132**, 473–478.

48. Papeo, L., Stein, T., and Soto-Faraco, S. (2017). The Two-Body Inversion Effect. *Psychol. Sci.* **28**, 369–379.

49. Papeo, L., Goupil, N., and Soto-Faraco, S. (2019). Visual Search for People Among People. *Psychol. Sci.* **30**, 1483–1496.

50. Carey, S. (2009). *The Origin of Concepts* (Oxford University Press).

51. Spelke, E. S. (2022). *What Babies Know: Core Knowledge and Composition Volume 1* (Oxford University Press).

52. Csibra, G. Gergely, G., Bíró, S., Koós, O., and Brockbank, M. (1999). Goal attribution without agency cues: the perception of ‘pure reason’ in infancy. *Cognition* **72**, 237–267.

53. Csibra, G. (2003)Teleological and referential understanding of action in infancy. *Philos. Trans. R. Soc. Lond. B. Biol. Sci.* **358**, 447–458.

54. Woodward, A. L. (1998). Infants selectively encode the goal object of an actor’s reach. *Cognition* **69**, 1–34.

55. Leslie, A. M. (1982). The Perception of Causality in Infants. *Perception* **11**, 173–186.

56. Muentener, P., and Carey, S. (2010). Infants’ causal representations of state change events. *Cognit. Psychol.* **61**, 63–86.

57. Hamlin, J. K., Wynn, K., and Bloom, P. (2007). Social evaluation by preverbal infants. *Nature* **450**, 557–559.

58. Tatone, D., Geraci, A., and Csibra, G. (2015). Giving and taking: Representational building blocks of active resource-transfer events in human infants. *Cognition* **137**, 47–62.

59. Baillargeon, R. (2004). Infants’ Physical World. *Curr. Dir. Psychol. Sci.* **13**, 89–94.

60. Holyoak, K. J., and Lu, H. (2021). Emergence of relational reasoning. *Curr. Opin. Behav. Sci.* **37**, 118–124.

61. Marcus, G. F. (2003). *The Algebraic Mind: Integrating Connectionism and Cognitive Science* (MIT Press).

62. Rissman, L., and Majid, A. (2019). Thematic roles: Core knowledge or linguistic construct? *Psychon. Bull. Rev.* **26**, 1850–1869.

63. Strickland, B., Fisher, M., and Knobe, J. (2012). Moral structure falls out of general event structure. *Psychological Inquiry*, 23(2), 198-205.

64. Michotte, A. (1963). *The perception of causality* (Basic Books).

65. Scholl, B. J., and Tremoulet, P. D. (2000). Perceptual causality and animacy. *Trends Cogn. Sci.* **4**, 299–309.
66. Hochmann, J-R., and Papeo, L. (2014). The Invariance Problem in Infancy: A Pupillometry Study. *Psychol. Sci.* **25**, 2038–2046.

67. Maris, E., and Oostenveld, R. (2007). Nonparametric statistical testing of EEG- and MEG-data. *J. Neurosci. Methods* **164**, 177–190.