# The 7th Workshop on Cognitive Aspects of the Lexicon

Michael Zock, Chersoni Emmanuele, Hsu Yu-Yin, Santus Enrico

## ► To cite this version:

Michael Zock, Chersoni Emmanuele, Hsu Yu-Yin, Santus Enrico. The 7th Workshop on Cognitive Aspects of the Lexicon. The 7th Workshop on Cognitive Aspects of the Lexicon, pp.109, 2022, 978-1-959429-01-2. 10.26615/978-1-959429-01-2. hal-04570169

HAL Id: hal-04570169

https://hal.science/hal-04570169

Submitted on 6 May 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Public Domain
# The 7th Workshop on Cognitive Aspects of the Lexicon

Michael Zock, Chersoni Emmanuele, Hsu Yu-Yin, Santus Enrico

► To cite this version:

Michael Zock, Chersoni Emmanuele, Hsu Yu-Yin, Santus Enrico. The 7th Workshop on Cognitive Aspects of the Lexicon. The 7th Workshop on Cognitive Aspects of the Lexicon, pp.109, 2022, 978-1-959429-01-2. 10.26615/978-1-959429-01-2. hal-04570169

HAL Id: hal-04570169

https://hal.science/hal-04570169v1

Submitted on 6 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
CogALex-VII

The Workshop on Cognitive Aspects of the Lexicon

Proceedings of the Workshop

November 20, 2022
CogALex-VII

# The Workshop on Cognitive Aspects of the Lexicon

Proceedings of the Workshop

November 20, 2022
Order copies of this and other ACL proceedings from:

Association for Computational Linguistics (ACL)
209 N. Eighth Street
Stroudsburg, PA 18360
USA
Tel: +1-570-476-8006
Fax: +1-570-476-0860
acl@aclweb.org

ISBN 978-1-959429-01-2
Order copies of this and other ACL proceedings from:

Association for Computational Linguistics (ACL)
209 N. Eighth Street
Stroudsburg, PA 18360
USA
Tel: +1-570-476-8006
Fax: +1-570-476-0860
acl@aclweb.org

ISBN 978-1-959429-01-2
Preface

Being ubiquitous and versatile, words are important for many tasks: sharing, storing, processing and accessing information. They are vital not only for communication, the acquisition, structuring (categorization), maintenance and access of knowledge, but also for thinking: problem-solving, link discovery, detection of similarities, metaphors and analogies. This being so, one may wonder how to build tools supporting the learning of words (memorization) and the various types of usage (access/navigation). Alas, the answer is not easy, as it depends on many factors: the task or goal (production/reception), the user's background (age, expertise), the momentary cognitive state (information available at the onset of the search), the material support (book, computer), etc. Obviously, words in books, computers, and the human brain are not the same. Being aware of this, different communities (linguists, lexicographers, psychologists) have focused on different aspects: representation and organization of words in dictionaries, creation of tools supporting navigation or conceptual search (thesauri), time course of word access, etc. The situation is complex, as the respective views, methods and research goals are not quite the same, all the more as they have changed considerably over time.

For example, rather than considering the lexicon as a static entity, where discrete units (words) are organized alphabetically (database view), dictionaries are now viewed dynamically, i.e., as lexical graphs, whose entities are linked in various ways (topical relations; associations) and whose link weights may vary over time. Also, lexicographers view words as products, i.e., holistic entities, while psychologists and neuroscientists view them as processes. Words are decomposed, and their synthesis requires activation of various parts of our brain, each one of them being dedicated to a specific part (meaning, form, sound) of the final form. The normal time course starts with some input (concrete object in the real world, or, more or less clear ideas, abstract concepts) leading then, 'little by little', to an output (word form). All this is achieved in milliseconds, as normal discourse consists in the production of two to three words per second.

Computational linguists have their own ways to look at words, which also have changed quite a bit over time. Discrete count-based vector representations have successively been replaced by continuous vectors (i.e., word embeddings), and then by language-model-based contextualized representations which outperform the static models (including word-embeddings) in a broad range of tasks.

As one can see, different communities look at words from different angles, which can be an asset and a problem. It is an asset, as complementary views may help us to broaden and deepen our understanding of this fundamental cognitive resource. Yet, this diversity of perspectives can also be a problem, in particular in a dynamic field like ours that is so rapidly moving on. Hence, it becomes harder and harder for everyone, including experts, to remain fully informed about the latest changes (state of the art). This is one of the reasons why we organize this workshop. More precisely, our goal is not only to keep people informed without getting them crushed by the information glut, but also to help them to perceive clearly what is new, relevant, and hence important. Last, but not least, we would like to connect people from different communities in the hope that this may help them to gain new insights or inspiration.

This is the 7th edition of CogALex, the first one not to be associated with COLING, as CogALex-VII is part of AACL-IJCNLP 2022. We have received 14 submissions, out of which we have selected 4 for oral presentation, and 6 for posters. This amounts to a submission rate of 71% (28,5% for the papers, 42,5% for the posters). In addition, we have an invited speaker, Massimo Stella from the CogNosco Lab of the University of Exeter (UK). His talk — Multiplex networks and AI unveil the influence of the mental lexicon on picture naming and its failures by people struck with aphasia.— fits our goal perfectly well, as it demonstrates the potential of graph theory to shed some light on the structure and evolution of the mental lexicon. We would like to express our thanks to him for having accepted to be our invited speaker. Also, sincerest thanks to all the members of the Programme Committee. Their expertise was invaluable
Preface

Being ubiquitous and versatile, words are important for many tasks: sharing, storing, processing and accessing information. They are vital not only for communication, the acquisition, structuring (categorization), maintenance and access of knowledge, but also for thinking: problem-solving, link discovery, detection of similarities, metaphors and analogies. This being so, one may wonder how to build tools supporting the learning of words (memorization) and the various types of usage (access/navigation). Alas, the answer is not easy, as it depends on many factors: the task or goal (production/reception), the user's background (age, expertise), the momentary cognitive state (information available at the onset of the search), the material support (book, computer), etc. Obviously, words in books, computers, and the human brain are not the same. Being aware of this, different communities (linguists, lexicographers, psychologists) have focused on different aspects: representation and organization of words in dictionaries, creation of tools supporting navigation or conceptual search (thesauri), time course of word access, etc. The situation is complex, as the respective views, methods and research goals are not quite the same, all the more as they have changed considerably over time.

For example, rather than considering the lexicon as a static entity, where discrete units (words) are organized alphabetically (database view), dictionaries are now viewed dynamically, i.e., as lexical graphs, whose entities are linked in various ways (topical relations; associations) and whose link weights may vary over time. Also, lexicographers view words as products, i.e., holistic entities, while psychologists and neuroscientists view them as processes. Words are decomposed, and their synthesis requires activation of various parts of our brain, each one of them being dedicated to a specific part (meaning, form, sound) of the final form. The normal time course starts with some input (concrete object in the real world, or, more or less clear ideas, abstract concepts) leading then, 'little by little', to an output (word form). All this is achieved in milliseconds, as normal discourse consists in the production of two to three words per second.

Computational linguists have their own ways to look at words, which also have changed quite a bit over time. Discrete count-based vector representations have successively been replaced by continuous vectors (i.e., word embeddings), and then by language-model-based contextualized representations which outperform the static models (including word-embeddings) in a broad range of tasks.

As one can see, different communities look at words from different angles, which can be an asset and a problem. It is an asset, as complementary views may help us to broaden and deepen our understanding of this fundamental cognitive resource. Yet, this diversity of perspectives can also be a problem, in particular in a dynamic field like ours that is so rapidly moving on. Hence, it becomes harder and harder for everyone, including experts, to remain fully informed about the latest changes (state of the art). This is one of the reasons why we organize this workshop. More precisely, our goal is not only to keep people informed without getting them crushed by the information glut, but also to help them to perceive clearly what is new, relevant, and hence important. Last, but not least, we would like to connect people from different communities in the hope that this may help them to gain new insights or inspiration.

This is the 7th edition of CogALex, the first one not to be associated with COLING, as CogALex-VII is part of AACL-IJCNLP 2022. We have received 14 submissions, out of which we have selected 4 for oral presentation, and 6 for posters. This amounts to a submission rate of 71% (28,5% for the papers, 42,5% for the posters). In addition, we have an invited speaker, Massimo Stella from the CogNosco Lab of the University of Exeter (UK). His talk — Multiplex networks and AI unveil the influence of the mental lexicon on picture naming and its failures by people struck with aphasia.— fits our goal perfectly well, as it demonstrates the potential of graph theory to shed some light on the structure and evolution of the mental lexicon. We would like to express our thanks to him for having accepted to be our invited speaker. Also, sincerest thanks to all the members of the Programme Committee. Their expertise was invaluable
to ensure a good selection of papers despite the tight schedule. Their reviews were helpful not only for us to make the decisions, but also for the authors, helping them to strengthen their work. We hope that the work presented here will inspire you, generate fruitful discussions, and possibly lead to new ideas, insights, and collaborations.

*The CogALex-VII Workshop Chairs*
to ensure a good selection of papers despite the tight schedule. Their reviews were helpful not only for us to make the decisions, but also for the authors, helping them to strengthen their work. We hope that the work presented here will inspire you, generate fruitful discussions, and possibly lead to new ideas, insights, and collaborations.

*The CogALex-VII Workshop Chairs*
**Organizing Committee**

Michael Zock (Aix-Marseille University)

Emmanuele Chersoni (The Hong Kong Polytechnic University)

Yu-Yin Hsu (The Hong Kong Polytechnic University)

Enrico Santus (Bloomberg LP)

**Program Committee**

Chris Biemann (University of Hamburg)

Marc Brysbaert (Ghent University)

Nichol Castro (University at Buffalo)

Yan Cong (Feinstein Institute for Medical Research)

Simon De Deyne (University of Melbourne)

Rodolfo Delmonte (University Ca' Foscari of Venice)

Luis Espinosa Anke (Cardiff University)

Olivier Ferret (CEA List)

Jinghang Gu (The Hong Kong Polytechnic University)

Eduard Hovy (Carnegie Mellon University)

Shu-Kai Hsieh (National Taiwan University)

Philippe Langlais (University of Montréal)

Alessandro Lenci (University of Pisa)

Vito Pirrelli (ILC-CNR Pisa)

Alain Polguère (University of Lorraine)

Jakob Prange (The Hong Kong Polytechnic University)

Alessandro Raganato (University of Milan-Bicocca)

Lavinia Salicchi (The Hong Kong Polytechnic University)

Sabine Schulte im Walde (University of Stuttgart)

Didier Schwab (Grenoble Alpes University)

Cynthia Siew (National University of Singapore)

Gilles Sérasset (Grenoble Alpes University)
**Organizing Committee**

Michael Zock (Aix-Marseille University)

Emmanuele Chersoni (The Hong Kong Polytechnic University)

Yu-Yin Hsu (The Hong Kong Polytechnic University)

Enrico Santus (Bloomberg LP)

**Program Committee**

Chris Biemann (University of Hamburg)

Marc Brysbaert (Ghent University)

Nichol Castro (University at Buffalo)

Yan Cong (Feinstein Institute for Medical Research)

Simon De Deyne (University of Melbourne)

Rodolfo Delmonte (University Ca' Foscari of Venice)

Luis Espinosa Anke (Cardiff University)

Olivier Ferret (CEA List)

Jinghang Gu (The Hong Kong Polytechnic University)

Eduard Hovy (Carnegie Mellon University)

Shu-Kai Hsieh (National Taiwan University)

Philippe Langlais (University of Montréal)

Alessandro Lenci (University of Pisa)

Vito Pirrelli (ILC-CNR Pisa)

Alain Polguère (University of Lorraine)

Jakob Prange (The Hong Kong Polytechnic University)

Alessandro Raganato (University of Milan-Bicocca)

Lavinia Salicchi (The Hong Kong Polytechnic University)

Sabine Schulte im Walde (University of Stuttgart)

Didier Schwab (Grenoble Alpes University)

Cynthia Siew (National University of Singapore)

Gilles Sérasset (Grenoble Alpes University)
<figure><img src="image_1.png" /></figure>
<figure><img src="image_1.png" /></figure>
Table of Contents

<table><tr><td>Patterns of Text Readability in Human and Predicted Eye Movements<br/>Nora Hollenstein, Itziar Gonzalez-Dios, Lisa Beinborn and Lena Jäger</td><td>1</td></tr><tr><td>(In)Alienable Possession in Mandarin Relative Clauses<br/>Deran Kong and Yu-Yin Hsu</td><td>16</td></tr><tr><td>Do Age of Acquisition and Orthographic Transparency Have the Same Effects in Different Modalities?<br/>Mohammad Momenian</td><td>25</td></tr><tr><td>CAT ManyNames: A New Dataset for Object Naming in Catalan<br/>Mar Domínguez Orfila, Maite Melero Nogués and Gemma Boleda Torrent</td><td>31</td></tr><tr><td>Finetuning Latin BERT for Word Sense Disambiguation on the Thesaurus Linguae Latinae<br/>Piroska Lendvai and Claudia Wick</td><td>37</td></tr><tr><td>Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis<br/>Fatemah Almeman and Luis Espinosa Anke</td><td>42</td></tr><tr><td>Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings<br/>Chenxin LIU and Emmanuele Chersoni</td><td>49</td></tr><tr><td>A Frame-Based Model of Inherent Polysemy, Copredication and Argument Coercion<br/>Chen Long, Laura Kallmeyer and Rainer Osswald</td><td>58</td></tr><tr><td>VISCOSE - a Kanji Dictionary Enriched with VISual, COmpositional, and SEMantic Information<br/>Werner Winiwarter and Bartholomäus Wloka</td><td>68</td></tr><tr><td>Compositionality as an Analogical Process: Introducing ANNE<br/>Giulia Rambelli, Emmanuele Chersoni, Philippe Blache and Alessandro Lenci</td><td>78</td></tr></table>
Table of Contents

<table><tbody><tr><td>Patterns of Text Readability in Human and Predicted Eye Movements<br>Nora Hollenstein, Itziar Gonzalez-Dios, Lisa Beinborn and Lena Jäger</td><td>1</td></tr><tr><td>(In)Alienable Possession in Mandarin Relative Clauses<br>Deran Kong and Yu-Yin Hsu</td><td>16</td></tr><tr><td>Do Age of Acquisition and Orthographic Transparency Have the Same Effects in Different Modalities?<br>Mohammad Momenian</td><td>25</td></tr><tr><td>CAT ManyNames: A New Dataset for Object Naming in Catalan<br>Mar Domínguez Orfila, Maite Melero Nogués and Gemma Boleda Torrent</td><td>31</td></tr><tr><td>Finetuning Latin BERT for Word Sense Disambiguation on the Thesaurus Linguae Latinae<br>Piroska Lendvai and Claudia Wick</td><td>37</td></tr><tr><td>Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis<br>Fatemah Almeman and Luis Espinosa Anke</td><td>42</td></tr><tr><td>Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings<br>Chenxin LIU and Emmanuele Chersoni</td><td>49</td></tr><tr><td>A Frame-Based Model of Inherent Polysemy, Copredication and Argument Coercion<br>Chen Long, Laura Kallmeyer and Rainer Osswald</td><td>58</td></tr><tr><td>VISCOSE - a Kanji Dictionary Enriched with VISual, COmpositional, and SEMantic Information<br>Werner Winiwarter and Bartholomäus Wloka</td><td>68</td></tr><tr><td>Compositionality as an Analogical Process: Introducing ANNE<br>Giulia Rambelli, Emmanuele Chersoni, Philippe Blache and Alessandro Lenci</td><td>78</td></tr></tbody></table>
<figure><img src="image_2.png" /></figure>
<figure><img src="image_2.png" /></figure>
Conference Program

November 20, 2022

17:00–17:15 (UTC +8, Taipei time) Introduction

17:15–18:15 Keynote Talk

Multiplex Networks and AI Unveil the Influence of the Mental Lexicon on Picture Naming and its Failures by People Struck with Aphasia
Massimo Stella (University of Exeter)

18:15–18:30 Short Break

18:30–19:30 Oral Presentations 1: Cognitive Modeling and Psycholinguistics

Patterns of Text Readability in Human and Predicted Eye Movements
Nora Hollenstein, Itziar Gonzalez-Dios, Lisa Beinborn and Lena Jäger

(In)Alienable Possession in Mandarin Relative Clauses
Deran Kong and Yu-Yin Hsu

19:30–20:30 Long Break
Conference Program

November 20, 2022

17:00–17:15 (UTC +8, Taipei time) Introduction

17:15–18:15 Keynote Talk

Multiplex Networks and AI Unveil the Influence of the Mental Lexicon on Picture Naming and its Failures by People Struck with Aphasia
Massimo Stella (University of Exeter)

18:15–18:30 Short Break

18:30–19:30 Oral Presentations 1: Cognitive Modeling and Psycholinguistics

Patterns of Text Readability in Human and Predicted Eye Movements
Nora Hollenstein, Itziar Gonzalez-Dios, Lisa Beinborn and Lena Jäger

(In)Alienable Possession in Mandarin Relative Clauses
Deran Kong and Yu-Yin Hsu

19:30–20:30 Long Break
20:30–21:45 Poster Session

*Do Age of Acquisition and Orthographic Transparency Have the Same Effects in Different Modalities?*

Mohammad Momenian

*CAT ManyNames: A New Dataset for Object Naming in Catalan*

Mar Domínguez Orfila, Maite Melero Nogués and Gemma Boleda Torrent

*Finetuning Latin BERT for Word Sense Disambiguation on the Thesaurus Linguae Latinae*

Piroska Lendvai and Claudia Wick

*Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis*

Fatemah Almeman and Luis Espinosa Anke

*Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings*

Chenxin LIU and Emmanuele Chersoni

*A Frame-Based Model of Inherent Polysemy, Copredication and Argument Coercion*

Chen Long, Laura Kallmeyer and Rainer Osswald

21:45–22:45 Oral Presentations 2: Lexical and Distributional Semantics

*VISCOSE - a Kanji Dictionary Enriched with VISual, COmpositional, and SEman-tic Information*

Werner Winiwarter and Bartholomäus Wloka

*Compositionality as an Analogical Process: Introducing ANNE*

Giulia Rambelli, Emmanuele Chersoni, Philippe Blache and Alessandro Lenci
20:30–21:45 Poster Session

*Do Age of Acquisition and Orthographic Transparency Have the Same Effects in Different Modalities?*

Mohammad Momenian

*CAT ManyNames: A New Dataset for Object Naming in Catalan*

Mar Domínguez Orfila, Maite Melero Nogués and Gemma Boleda Torrent

*Finetuning Latin BERT for Word Sense Disambiguation on the Thesaurus Linguae Latinae*

Piroska Lendvai and Claudia Wick

*Putting WordNet's Dictionary Examples in the Context of Definition Modelling: An Empirical Analysis*

Fatemah Almeman and Luis Espinosa Anke

*Exploring Nominal Coercion in Semantic Spaces with Static and Contextualized Word Embeddings*

Chenxin LIU and Emmanuele Chersoni

*A Frame-Based Model of Inherent Polysemy, Copredication and Argument Coercion*

Chen Long, Laura Kallmeyer and Rainer Osswald

21:45–22:45 Oral Presentations 2: Lexical and Distributional Semantics

*VISCOSE - a Kanji Dictionary Enriched with VISual, COmpositional, and SEman-tic Information*

Werner Winiwarter and Bartholomäus Wloka

*Compositionality as an Analogical Process: Introducing ANNE*

Giulia Rambelli, Emmanuele Chersoni, Philippe Blache and Alessandro Lenci
November 20, 2022 (continued)

22:45–23:00 Conclusive Remarks
November 20, 2022 (continued)

22:45–23:00 Conclusive Remarks
<figure><img src="image_3.png" /></figure>
<figure><img src="image_3.png" /></figure>
# Patterns of Text Readability in Human and Predicted Eye Movements

**Nora Hollenstein**

University of Copenhagen
nora.hollenstein@hum.ku.dk

**Itziar Gonzalez-Dios**

HiTZ Center - IXA (UPV/EHU)
itziar.gonzalezd@ehu.eus

**Lisa Beinborn**

CLTL Lab, VU Amsterdam
l.beinborn@vu.nl

**Lena Jäger**

University of Zurich; University of Potsdam
jaeger@cl.uzh.ch

## Abstract

It has been shown that multilingual transformer models are able to predict human reading behavior when fine-tuned on small amounts of eye tracking data. As the cumulated prediction results do not provide insights into the linguistic cues that the model acquires to predict reading behavior, we conduct a deeper analysis of the predictions from the perspective of readability. We try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for English. We compare a range of model configurations to multiple baselines. We show that the models exhibit difficulties with function words and that pre-training only provides limited advantages for linguistic generalization.

## 1 Introduction

Eye movement data of reading provides rich insights into cognitive processes of language understanding. The signal can be used to modulate the inductive bias of machine learning models towards more cognitively plausible processing which can increase model performance (Mathias et al., 2020; Hollenstein et al., 2019). It has been shown that large multilingual pre-trained language models are able to accurately predict eye tracking patterns when fine-tuned on small amounts of eye tracking data (Hollenstein et al., 2021; Takmaz, 2022; Salicchi et al., 2022).

Generally, transformer-based language models seem to be better at predicting cognitive signals of human language comprehension (e.g., self-paced reading times, eye movements, or brain activity) than language models based on other architectures (Merkx and Frank, 2021; Schrimpf et al., 2020). However, as prediction is not explanation (Demberg and Keller, 2019; Hale et al., 2022), we aim to dissect the predicted reading patterns and

analyze them in more detail to gain clearer insights into the underlying representation of processing complexity. Eye tracking data can be very informative to evaluate sentence comprehension strategies, however, the interdependencies between the eye tracking measures need to be taken into account (Vasishth et al., 2013). We propose to use the relation between eye movements in reading and text readability in terms of linguistic complexity to better understand procedural patterns of English sentence comprehension in language models. We provide interpretable insights into the prediction errors to investigate the following two questions:

(1) What is the impact of pre-training on the performance of language models predicting human eye movements?

(2) Is the relationship between human reading patterns and English text readability preserved in the reading patterns predicted by the investigated language models?

We focus on multilingual pre-trained language models (mBERT and XLM), fine-tuned on a range of eye tracking features from reading in multiple languages (English, German, Dutch, and Russian). We build upon the approach by Hollenstein et al. (2021) and provide strong baselines and a series of model configurations to answer the first question in Section 2. Subsequently, we address the second question in Section 3, by performing an extensive readability analysis based on various aspects of English text complexity. We propose to evaluate the predicted gaze features by analyzing whether their correlation with a range of readability measures is similar to the correlation observed in human eye movement data.<fnref n="1" /> These two contributions allow us to better interpret the ability of language models to predict human reading behaviour.

<footnote n="1">¹Our code is available here: https://github.com/norahollenstein/readability-patterns</footnote>
# Patterns of Text Readability in Human and Predicted Eye Movements

Nora Hollenstein

University of Copenhagen

nora.hollenstein@hum.ku.dk

Itziar Gonzalez-Dios

HiTZ Center - IXA (UPV/EHU)

itziar.gonzalezd@ehu.eus

Lisa Beinborn

CLTL Lab, VU Amsterdam

l.beinborn@vu.nl

Lena Jäger

University of Zurich; University of Potsdam

jaeger@cl.uzh.ch

## Abstract

It has been shown that multilingual transformer models are able to predict human reading behavior when fine-tuned on small amounts of eye tracking data. As the cumulated prediction results do not provide insights into the linguistic cues that the model acquires to predict reading behavior, we conduct a deeper analysis of the predictions from the perspective of readability. We try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for English. We compare a range of model configurations to multiple baselines. We show that the models exhibit difficulties with function words and that pre-training only provides limited advantages for linguistic generalization.

## 1 Introduction

Eye movement data of reading provides rich insights into cognitive processes of language understanding. The signal can be used to modulate the inductive bias of machine learning models towards more cognitively plausible processing which can increase model performance (Mathias et al., 2020; Hollenstein et al., 2019). It has been shown that large multilingual pre-trained language models are able to accurately predict eye tracking patterns when fine-tuned on small amounts of eye tracking data (Hollenstein et al., 2021; Takmaz, 2022; Salicchi et al., 2022).

Generally, transformer-based language models seem to be better at predicting cognitive signals of human language comprehension (e.g., self-paced reading times, eye movements, or brain activity) than language models based on other architectures (Merkx and Frank, 2021; Schrimpf et al., 2020). However, as prediction is not explanation (Demberg and Keller, 2019; Hale et al., 2022), we aim to dissect the predicted reading patterns and

analyze them in more detail to gain clearer insights into the underlying representation of processing complexity. Eye tracking data can be very informative to evaluate sentence comprehension strategies, however, the interdependencies between the eye tracking measures need to be taken into account (Vasishth et al., 2013). We propose to use the relation between eye movements in reading and text readability in terms of linguistic complexity to better understand procedural patterns of English sentence comprehension in language models. We provide interpretable insights into the prediction errors to investigate the following two questions:

1. What is the impact of pre-training on the performance of language models predicting human eye movements?

2. Is the relationship between human reading patterns and English text readability preserved in the reading patterns predicted by the investigated language models?

We focus on multilingual pre-trained language models (mBERT and XLM), fine-tuned on a range of eye tracking features from reading in multiple languages (English, German, Dutch, and Russian). We build upon the approach by Hollenstein et al. (2021) and provide strong baselines and a series of model configurations to answer the first question in Section 2. Subsequently, we address the second question in Section 3, by performing an extensive readability analysis based on various aspects of English text complexity. We propose to evaluate the predicted gaze features by analyzing whether their correlation with a range of readability measures is similar to the correlation observed in human eye movement data.<fnref n="1" /> These two contributions allow us to better interpret the ability of language models to predict human reading behaviour.

<footnote n="1">¹Our code is available here: https://github.com/norahollenstein/readability-patterns</footnote>
<table><caption>Table 1: Prediction accuracy aggregated across all eight eye tracking features (with standard deviation across three runs in parentheses). Fine-tuned models: last layer ($M_{\star}$), all layers ($M_{\star\dagger}$), all layers without pre-training ($M_{\dagger}$).</caption><thead><tr><th>Models</th><th>EN</th><th>NL</th><th>DE</th><th>RU</th><th>ALL</th></tr></thead><tbody><tr><td>RANDOM BL</td><td>78.66 (0.06)</td><td>84.30 (0.11)</td><td>74.11 (4.6)</td><td>65.83 (2.55)</td><td>86.15 (0.43)</td></tr><tr><td>MEAN BL</td><td>89.94</td><td>90.15</td><td>84.98</td><td>85.35</td><td>92.54</td></tr><tr><td>MBERT</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>M<sub>☆</sub></td><td>90.95 (0.11)</td><td>90.51 (0.31)</td><td>75.68 (3.99)</td><td>70.64 (2.38)</td><td>92.93 (0.13)</td></tr><tr><td>M<sub>♡</sub></td><td>93.73 (0.08)</td><td>91.91 (0.23)</td><td>77.41 (3.65)</td><td>77.30 (4.17)</td><td>94.68 (0.05)</td></tr><tr><td>M<sub>♀</sub></td><td>93.30 (0.03)</td><td>91.60 (0.36)</td><td>77.85 (2.85)</td><td>77.38 (1.85)</td><td>94.35 (0.13)</td></tr><tr><td>XLM-100</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>M<sub>☆</sub></td><td>92.94 (0.05)</td><td>91.80 (0.40)</td><td>77.31 (2.75)</td><td>76.54 (1.92)</td><td>94.19 (0.10)</td></tr><tr><td>M<sub>♡</sub></td><td><b>93.92</b> (0.07)</td><td>92.26 (0.33)</td><td><b>86.38</b> (0.27)</td><td><b>94.65</b> (0.88)</td><td><b>94.89</b> (0.12)</td></tr><tr><td>M<sub>♀</sub></td><td><b>93.92</b> (0.16)</td><td><b>92.32</b> (0.36)</td><td>86.04 (0.28)</td><td>94.62 (0.84)</td><td>94.15 (1.20)</td></tr></tbody></table>

## 2 Multilingual Prediction of Eye Movements in Reading

Hollenstein et al. (2021) showed that language models can predict a range of eye tracking features in multiple languages. The prediction setup has been made widely available as a shared task to facilitate comparisons between models and the analysis of their inner workings (Hollenstein et al., 2022). In this work, we use a similar setup which we summarize below before we present the results of the eye tracking prediction.

### 2.1 Data

We use eye tracking corpora for sentences in four languages: English, Dutch, German, and Russian. Full sentences or longer naturally occurring text spans were read by multiple native speakers (see Appendix A.1 for detailed statistics) and tracked by high-precision eye trackers. The datasets report the following eye tracking features for each token of the stimulus text: (NFIX), mean fixation duration (MFD), fixation proportion (FPROP), first fixation duration (FFD), first pass duration (FPD), total reading time (TRT), number of re-fixations (NREFIX), and re-read proportion (REPROP). All features are first computed for each subject separately by aggregating over the fixations, and then averaged over all subjects.

These features arguably reflect the complete reading process at the various stages of linguistic integration, from early lexical access for word recognition (e.g., FFD) up to subsequent syntactic integration taking into account regression move-

ments (e.g., NREFIX). For a review of which eye movement feature reflects which linguistic level, see Clifton et al. (2007).

For more detailed information about the data and the training procedure, see Hollenstein et al. (2021).

### 2.2 Model Configurations

The model is optimized to predict eye tracking features from reading as accurately as possible. For each token $w$ in the input text, we predict a vector containing the eight eye tracking features listed above. We focus on the transformer-based models multilingual BERT (Devlin et al., 2019) and cross-lingual XLM-100 (Lample and Conneau, 2019). We use pre-trained checkpoints from the Hugging-Face repository.<fnref n="2" />

We propose the following baselines to benchmark model performance. First, we compare with a random baseline (RANDOM BL), which presents model predictions made from a randomly initialized regression layer. Second, we use a mean baseline averaged across all eye tracking features (MEAN BL), which calculates the mean value for each eye tracking feature from the training data and uses it as a prediction for all words in the test data.

Since one of our goals is to evaluate the gains from fine-tuning a pre-trained LM on eye tracking data, we also compare fine-tuning all layers (we call these models $M_{\star\dagger}$), to fine-tuning only the final regression layer ($M_{\star\dagger}$). Finally, we investigate the benefits of pre-training on large language corpora

<footnote n="2">²xlm-mlm-100-1280 and bert-base-multilingual-cased</footnote>
<table><caption>Table 1: Prediction accuracy aggregated across all eight eye tracking features (with standard deviation across three runs in parentheses). Fine-tuned models: last layer ($M_{\star}$), all layers ($M_{\star\dagger}$), all layers without pre-training ($M_{\dagger}$).</caption><thead><tr><th>Models</th><th>EN</th><th>NL</th><th>DE</th><th>RU</th><th>ALL</th></tr></thead><tbody><tr><td>RANDOM BL</td><td>78.66 (0.06)</td><td>84.30 (0.11)</td><td>74.11 (4.6)</td><td>65.83 (2.55)</td><td>86.15 (0.43)</td></tr><tr><td>MEAN BL</td><td>89.94</td><td>90.15</td><td>84.98</td><td>85.35</td><td>92.54</td></tr><tr><td>MBERT</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>M<sub>☆</sub></td><td>90.95 (0.11)</td><td>90.51 (0.31)</td><td>75.68 (3.99)</td><td>70.64 (2.38)</td><td>92.93 (0.13)</td></tr><tr><td>M<sub>♡</sub></td><td>93.73 (0.08)</td><td>91.91 (0.23)</td><td>77.41 (3.65)</td><td>77.30 (4.17)</td><td>94.68 (0.05)</td></tr><tr><td>M<sub>♀</sub></td><td>93.30 (0.03)</td><td>91.60 (0.36)</td><td>77.85 (2.85)</td><td>77.38 (1.85)</td><td>94.35 (0.13)</td></tr><tr><td>XLM-100</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>M<sub>☆</sub></td><td>92.94 (0.05)</td><td>91.80 (0.40)</td><td>77.31 (2.75)</td><td>76.54 (1.92)</td><td>94.19 (0.10)</td></tr><tr><td>M<sub>♡</sub></td><td><b>93.92</b> (0.07)</td><td>92.26 (0.33)</td><td><b>86.38</b> (0.27)</td><td><b>94.65</b> (0.88)</td><td><b>94.89</b> (0.12)</td></tr><tr><td>M<sub>♀</sub></td><td><b>93.92</b> (0.16)</td><td><b>92.32</b> (0.36)</td><td>86.04 (0.28)</td><td>94.62 (0.84)</td><td>94.15 (1.20)</td></tr></tbody></table>

## 2 Multilingual Prediction of Eye Movements in Reading

Hollenstein et al. (2021) showed that language models can predict a range of eye tracking features in multiple languages. The prediction setup has been made widely available as a shared task to facilitate comparisons between models and the analysis of their inner workings (Hollenstein et al., 2022). In this work, we use a similar setup which we summarize below before we present the results of the eye tracking prediction.

### 2.1 Data

We use eye tracking corpora for sentences in four languages: English, Dutch, German, and Russian. Full sentences or longer naturally occurring text spans were read by multiple native speakers (see Appendix A.1 for detailed statistics) and tracked by high-precision eye trackers. The datasets report the following eye tracking features for each token of the stimulus text: (NFIX), mean fixation duration (MFD), fixation proportion (FPROP), first fixation duration (FFD), first pass duration (FPD), total reading time (TRT), number of re-fixations (NREFIX), and re-read proportion (REPROP). All features are first computed for each subject separately by aggregating over the fixations, and then averaged over all subjects.

These features arguably reflect the complete reading process at the various stages of linguistic integration, from early lexical access for word recognition (e.g., FFD) up to subsequent syntactic integration taking into account regression move-

ments (e.g., NREFIX). For a review of which eye movement feature reflects which linguistic level, see Clifton et al. (2007).

For more detailed information about the data and the training procedure, see Hollenstein et al. (2021).

### 2.2 Model Configurations

The model is optimized to predict eye tracking features from reading as accurately as possible. For each token $w$ in the input text, we predict a vector containing the eight eye tracking features listed above. We focus on the transformer-based models multilingual BERT (Devlin et al., 2019) and cross-lingual XLM-100 (Lample and Conneau, 2019). We use pre-trained checkpoints from the Hugging-Face repository.<fnref n="2" />

We propose the following baselines to benchmark model performance. First, we compare with a random baseline (RANDOM BL), which presents model predictions made from a randomly initialized regression layer. Second, we use a mean baseline averaged across all eye tracking features (MEAN BL), which calculates the mean value for each eye tracking feature from the training data and uses it as a prediction for all words in the test data.

Since one of our goals is to evaluate the gains from fine-tuning a pre-trained LM on eye tracking data, we also compare fine-tuning all layers (we call these models $M_{\star\dagger}$), to fine-tuning only the final regression layer ($M_{\star\dagger}$). Finally, we investigate the benefits of pre-training on large language corpora

<footnote n="2">²xlm-mlm-100-1280 and bert-base-multilingual-cased</footnote>
by comparing the fully fine-tuned pre-trained language models ($M_♀$) to a model trained from scratch on randomly initialized weights ($M_♂$).

## 2.3 Results

Since we scale all gaze features to values between 0–100, we evaluate the models using the mean absolute error (MAE). For better readability, we report the results as *prediction accuracy*, defined as 100–MAE. The results are presented in Table 1.

**Baseline comparison** The performance of the random baseline (RANDOM BL) is much lower than that of the mean baseline (MEAN BL), and therefore is not suitable for comparison. XLM-100 outperforms the mean baseline for all languages, but mBERT does not reach it for German (DE) and Russian (RU).

**The impact of pre-training** The results of the pre-trained and fine-tuned language models ($M_♀$) and the transformer models trained from scratch ($M_♂$) show a very similar performance. This demonstrates that the advantage of pre-training language models on large text corpora is only minimal for the task of predicting human eye movements. When fine-tuning only the regression layer ($M_☆$), the models yield only modest (if any) improvements over the MEAN BL. However, when all layers of a model are fine-tuned ($M_♀$), the differences become more notable, especially for languages where less eye tracking data is available (DE and RU).

Generally, XLM-100 yields better results than mBERT for all languages, and especially for the ones with smaller datasets (DE and RU). Our results are in line with previous work showing that XLM models perform better at zero-shot eye tracking prediction for an unseen language than mBERT (Srivastava, 2022). Similarly, Hollenstein et al. (2021) find that mBERT is outperformed by monolingual models for languages with small eye tracking training datasets.

This indicates that the architecture and training objective of a model might be more important for eye tracking prediction than pre-training on large amounts of text. Transformer architectures are promising for predicting reading times, but the extensive pre-training on text input might be superfluous, as the models learn more from the fine-tuning on psychometric features. This could mean that not much linguistic knowledge is required for eye tracking prediction. Alternately, the choice of metric (MAE) might not be the most appropriate to

capture the subtleties of the task. However, as we will see in Section 3, the pre-trained models show an advantage over randomly initialized models in their correlation with text readability measures.

## Evaluation of individual eye tracking features

The aggregated mean baseline across all eye tracking features can be misleading because it conceals the model's prediction performance for individual features. A model that yields a superior performance on the aggregated level does not necessarily outperform the mean baseline for all features. Therefore, we zoom in on individual eye tracking features and compare the performance of the fine-tuned mBERT and XLM-100 with the mean baseline in Figure 2. The results show that mBERT fails to predict MFD, FPROP and REPROP for German and Russian, while XLM-100 outperforms the aggregate mean baseline for all languages. Some features are more strongly affected by the large degree of individual variability in human eye movements (Kidd et al., 2018). We additionally visualize the feature ranges of the predicted eye tracking features compared to the real eye tracking data in Figure 1.

## 3 Readability Analysis

Eye movement patterns during reading are known to be influenced by the readability of texts (Rayner et al., 2006). Singh et al. (2016) assess text readability with automatically predicted eye tracking features. Although their readability assessment model was based only on predicted reading times, it yielded results comparable to models that use extensive syntactic features to compute linguistic complexity. Wiechmann et al. (2022) find that, for English, the accuracy of eye tracking prediction is systematically linked to sentence-level text features that approximate readability.

We try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for English. We analyze the correlation between model predictions and readability measures to better understand the processing patterns that the model picks up.

### 3.1 Measuring Readability

The readability of a text is affected by variation at all levels of linguistic processing (Beinborn et al., 2012). Feng et al. (2009) introduce a large range of
by comparing the fully fine-tuned pre-trained language models ($M_♀$) to a model trained from scratch on randomly initialized weights ($M_♂$).

## 2.3 Results

Since we scale all gaze features to values between 0–100, we evaluate the models using the mean absolute error (MAE). For better readability, we report the results as *prediction accuracy*, defined as 100–MAE. The results are presented in Table 1.

**Baseline comparison** The performance of the random baseline (RANDOM BL) is much lower than that of the mean baseline (MEAN BL), and therefore is not suitable for comparison. XLM-100 outperforms the mean baseline for all languages, but mBERT does not reach it for German (DE) and Russian (RU).

**The impact of pre-training** The results of the pre-trained and fine-tuned language models ($M_♀$) and the transformer models trained from scratch ($M_♂$) show a very similar performance. This demonstrates that the advantage of pre-training language models on large text corpora is only minimal for the task of predicting human eye movements. When fine-tuning only the regression layer ($M_☆$), the models yield only modest (if any) improvements over the MEAN BL. However, when all layers of a model are fine-tuned ($M_♀$), the differences become more notable, especially for languages where less eye tracking data is available (DE and RU).

Generally, XLM-100 yields better results than mBERT for all languages, and especially for the ones with smaller datasets (DE and RU). Our results are in line with previous work showing that XLM models perform better at zero-shot eye tracking prediction for an unseen language than mBERT (Srivastava, 2022). Similarly, Hollenstein et al. (2021) find that mBERT is outperformed by monolingual models for languages with small eye tracking training datasets.

This indicates that the architecture and training objective of a model might be more important for eye tracking prediction than pre-training on large amounts of text. Transformer architectures are promising for predicting reading times, but the extensive pre-training on text input might be superfluous, as the models learn more from the fine-tuning on psychometric features. This could mean that not much linguistic knowledge is required for eye tracking prediction. Alternately, the choice of metric (MAE) might not be the most appropriate to

capture the subtleties of the task. However, as we will see in Section 3, the pre-trained models show an advantage over randomly initialized models in their correlation with text readability measures.

**Evaluation of individual eye tracking features**

The aggregated mean baseline across all eye tracking features can be misleading because it conceals the model's prediction performance for individual features. A model that yields a superior performance on the aggregated level does not necessarily outperform the mean baseline for all features. Therefore, we zoom in on individual eye tracking features and compare the performance of the fine-tuned mBERT and XLM-100 with the mean baseline in Figure 2. The results show that mBERT fails to predict MFD, FPROP and REPROP for German and Russian, while XLM-100 outperforms the aggregate mean baseline for all languages. Some features are more strongly affected by the large degree of individual variability in human eye movements (Kidd et al., 2018). We additionally visualize the feature ranges of the predicted eye tracking features compared to the real eye tracking data in Figure 1.

## 3 Readability Analysis

Eye movement patterns during reading are known to be influenced by the readability of texts (Rayner et al., 2006). Singh et al. (2016) assess text readability with automatically predicted eye tracking features. Although their readability assessment model was based only on predicted reading times, it yielded results comparable to models that use extensive syntactic features to compute linguistic complexity. Wiechmann et al. (2022) find that, for English, the accuracy of eye tracking prediction is systematically linked to sentence-level text features that approximate readability.

We try to disentangle the three-fold relationship between human eye movements, the capability of language models to predict these eye movement patterns, and sentence-level readability measures for English. We analyze the correlation between model predictions and readability measures to better understand the processing patterns that the model picks up.

### 3.1 Measuring Readability

The readability of a text is affected by variation at all levels of linguistic processing (Beinborn et al., 2012). Feng et al. (2009) introduce a large range of
<figure><img src="image_4.png" /><figcaption>Figure 1: Feature ranges of the true eye tracking values for MFD and FPROP compared to the predicted eye tracking data (mBERT and XLM-100) for English and all four languages together.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 2: Improvements on prediction accuracy of the fine-tuned models mBERT and XLM-100 compared to the mean baseline across all four languages.</figcaption></figure>

cognitively motivated readability measures that can be extracted using a standard natural language processing pipeline. Machine learning models trained on these measures can reliably predict the readability of texts in multiple languages (Vajjala Balakrishna, 2015). We explore a subset of 11 measures that are likely to affect eye movement patterns.

**Flesch score (FLESCH):** Flesch (1948) introduced the most renowned readability formula that takes the surface structure of a text into account, which is measured by the number of syllables, words and sentences. The Flesch reading ease score has been found to provide only a shallow readability estimation for English texts because it ignores deeper linguistic levels of text processing (Collins-Thompson, 2014; Bengoetxea and Gonzalez-Dios, 2021). Nahatame (2021) show that readability measures that quantify lexical and syntactic characteristics provide better approximations for predicting eye movement patterns than the Flesch score. We include it in our analysis mainly for the sake of comparison and completeness.

**Word frequency (WF, ZIPF):** The influence of lexical frequency on fixation duration is one of the most studied phenomena in psycholinguistic reading research. It is well established that readers tend to look longer at infrequent words (Rayner,

1977). We use the lexical frequency values provided by the `wordfreq` Python library (Speer et al., 2018) and its Zipfian variant on a logarithmic scale (Van Heuven et al., 2014).

**Word length (WL):** Longer words (measured in terms of number of characters) are generally fixed for longer periods. Nearly 70% of the variance in mean fixation duration can be explained by word length and word frequency (Just and Carpenter, 1980).

**Sentence length (SL):** We include sentence length, calculated as the number of tokens of each sentence, since the readability measures highlighted in our analysis are strongly related to the length of a sentence. Sarti et al. (2021) confirm that for all text complexity metrics, sentence length exhibits the highest correlation.

**Distance to head (D2H):** Sarti et al. (2021) find a strong correlation between readability measures related to dependency parsing (e.g., parse depth) and perceived complexity. Dependency features also correlate well with eye tracking patterns and can predict regressive eye movements (Lopopolo et al., 2019). In the opposite direction, Strzyz et al. (2019) show that eye tracking information can improve dependency parsing. We therefore measure
<figure><img src="image_4.png" /><figcaption>Figure 1: Feature ranges of the true eye tracking values for MFD and FPROP compared to the predicted eye tracking data (mBERT and XLM-100) for English and all four languages together.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 2: Improvements on prediction accuracy of the fine-tuned models mBERT and XLM-100 compared to the mean baseline across all four languages.</figcaption></figure>

cognitively motivated readability measures that can be extracted using a standard natural language processing pipeline. Machine learning models trained on these measures can reliably predict the readability of texts in multiple languages (Vajjala Balakrishna, 2015). We explore a subset of 11 measures that are likely to affect eye movement patterns.

**Flesch score (FLESCH):** Flesch (1948) introduced the most renowned readability formula that takes the surface structure of a text into account, which is measured by the number of syllables, words and sentences. The Flesch reading ease score has been found to provide only a shallow readability estimation for English texts because it ignores deeper linguistic levels of text processing (Collins-Thompson, 2014; Bengoetxea and Gonzalez-Dios, 2021). Nahatame (2021) show that readability measures that quantify lexical and syntactic characteristics provide better approximations for predicting eye movement patterns than the Flesch score. We include it in our analysis mainly for the sake of comparison and completeness.

**Word frequency (WF, ZIPF):** The influence of lexical frequency on fixation duration is one of the most studied phenomena in psycholinguistic reading research. It is well established that readers tend to look longer at infrequent words (Rayner,

1977). We use the lexical frequency values provided by the `wordfreq` Python library (Speer et al., 2018) and its Zipfian variant on a logarithmic scale (Van Heuven et al., 2014).

**Word length (WL):** Longer words (measured in terms of number of characters) are generally fixed for longer periods. Nearly 70% of the variance in mean fixation duration can be explained by word length and word frequency (Just and Carpenter, 1980).

**Sentence length (SL):** We include sentence length, calculated as the number of tokens of each sentence, since the readability measures highlighted in our analysis are strongly related to the length of a sentence. Sarti et al. (2021) confirm that for all text complexity metrics, sentence length exhibits the highest correlation.

**Distance to head (D2H):** Sarti et al. (2021) find a strong correlation between readability measures related to dependency parsing (e.g., parse depth) and perceived complexity. Dependency features also correlate well with eye tracking patterns and can predict regressive eye movements (Lopopolo et al., 2019). In the opposite direction, Strzyz et al. (2019) show that eye tracking information can improve dependency parsing. We therefore measure
<table><caption>Table 2: Spearman correlation coefficients of observed eye tracking features and readability measures. All correlations are significant ($p < 0.01$).</caption><thead><tr><th>Feature</th><th>FLESCH</th><th>WL</th><th>WF</th><th>ZIPF</th><th>SL</th><th>D2H</th><th>AMB</th><th>AOA</th><th>FAM</th><th>CONC</th><th>IMAG</th></tr></thead><tbody><tr><td>NFIX</td><td>-0.55</td><td>0.94</td><td>-0.54</td><td>-0.82</td><td>0.96</td><td>-0.29</td><td>-0.28</td><td>0.29</td><td>-0.46</td><td>0.45</td><td>0.43</td></tr><tr><td>NREFIX</td><td>-0.58</td><td>0.92</td><td>-0.55</td><td>-0.82</td><td>0.81</td><td>-0.23</td><td>-0.25</td><td>0.26</td><td>-0.35</td><td>0.33</td><td>0.32</td></tr><tr><td>MFD</td><td>-0.47</td><td>0.84</td><td>-0.44</td><td>-0.68</td><td>0.96</td><td>-0.26</td><td>-0.19</td><td>0.18</td><td>-0.42</td><td>0.42</td><td>0.40</td></tr><tr><td>FFD</td><td>-0.47</td><td>0.82</td><td>-0.43</td><td>-0.63</td><td>0.96</td><td>-0.26</td><td>-0.19</td><td>0.18</td><td>-0.42</td><td>0.42</td><td>0.41</td></tr><tr><td>FPD</td><td>-0.50</td><td>0.91</td><td>-0.50</td><td>-0.78</td><td>0.96</td><td>-0.28</td><td>-0.25</td><td>0.25</td><td>-0.44</td><td>0.43</td><td>0.42</td></tr><tr><td>TRT</td><td>-0.52</td><td>0.95</td><td>-0.54</td><td>-0.82</td><td>0.95</td><td>-0.28</td><td>-0.26</td><td>0.25</td><td>-0.44</td><td>0.44</td><td>0.42</td></tr><tr><td>FPROP</td><td>-0.52</td><td>0.98</td><td>-0.48</td><td>-0.72</td><td>0.99</td><td>-0.30</td><td>-0.25</td><td>0.24</td><td>-0.45</td><td>0.45</td><td>0.44</td></tr><tr><td>REPROP</td><td>-0.58</td><td>0.96</td><td>-0.52</td><td>-0.83</td><td>0.86</td><td>-0.23</td><td>-0.25</td><td>0.26</td><td>-0.36</td><td>0.33</td><td>0.32</td></tr></tbody></table>

the distance to head as the number of words between the current word and its head according to the dependency tree. For example, in the sentence *She reads a mistery novel*, the distance from the word *novel* to its head *reads* is 2. We use the parser *Stanza* (Qi et al., 2020) for the dependency analysis.

**Ambiguity level (AMB):** The meaning of polysemous words can usually be disambiguated by processing the context. The effect of a high ambiguity level on eye movement patterns is therefore usually more pronounced for later processing measures such as NREFIX than for early gaze metrics reflecting lexical access (Foraker and Murphy, 2012; Shen and Li, 2016). We calculate the ambiguity level for each open class word (nouns, adjectives, adverbs, and verbs) as the number of possible senses (synsets) that can be found in the NLTK implementation (Bird and Loper, 2004) of WordNet (Fellbaum). The minimum ambiguity level is 1, which means that there is only one sense for a given wordform. For example, the noun *car* has an ambiguity level of five because it appears in 5 nominal synsets. For this analysis, we exclude words that do not appear in WordNet.

**Age of acquisition (AOA), familiarity (FAM), concreteness (CONC), and imageability (IMAG):** We include four cognitively motivated features of word complexity that are likely to affect fixation durations in reading (Juhasz and Rayner, 2006). Ratings for age of acquisition, familiarity, concreteness, and imageability (the intensity with which a word evokes a clear mental image) in the MRC Psycholinguistic Database (Wilson, 1988) are strongly associated with each other and with other read-

ability metrics.<fnref n="3" /> Paetzold and Specia (2016) find that word frequencies correlate with familiarity and AOA, while the depth of a word in a thesaurus hierarchy correlates with both its concreteness and its imageability. We exclude words that do not appear in the database when calculating the correlations.

## 3.2 Readability and Eye Movement Patterns

We calculate the Spearman correlation coefficients between the recorded eye tracking data and the readability measures for English (Table 2). The strongest correlations can be found for sentence length, word length, and lexical frequency, which confirms three widely studied effects in reading research (Sarti et al., 2021).

**Predictive Power** In a second step, we analyze the correlations between four of the readability measures (FLESCH, WL, ZIPF frequency, SL) and the predictions of the different models for fixation proportion and mean fixation duration for English (see Table 3).<fnref n="4" /> The results show that for fine-tuned mBERT, while yielding lower overall prediction accuracy when aggregating across all features, the correlation of the predicted eye movement values to word frequency and length is generally more similar to the correlation of real gaze features with word frequency and length than for the XLM-100 models. When comparing fine-tuned pre-trained models ($M_\hat{\omega}$) to models trained from scratch on eye tracking data ($M_\hat{z}$), the results presented in Table 3 also show that the correlation with word length and frequency is stronger in the predictions of the

<footnote n="3">³https://websites.psychology.uwa.edu.au/school/mrcdatabase/uwa_mrc.htm</footnote>

<footnote n="4">⁴Correlations to other gaze features show the same trends. We analyze FPROP because the models yield low prediction performance on this feature, compared to MFD, which yields high prediction results.</footnote>
<table><caption>Table 2: Spearman correlation coefficients of observed eye tracking features and readability measures. All correlations are significant (p < 0.01).</caption><thead><tr><th>Feature</th><th>FLESCH</th><th>WL</th><th>WF</th><th>ZIPF</th><th>SL</th><th>D2H</th><th>AMB</th><th>AOA</th><th>FAM</th><th>CONC</th><th>IMAG</th></tr></thead><tbody><tr><td>NFIX</td><td>-0.55</td><td>0.94</td><td>-0.54</td><td>-0.82</td><td>0.96</td><td>-0.29</td><td>-0.28</td><td>0.29</td><td>-0.46</td><td>0.45</td><td>0.43</td></tr><tr><td>NREFIX</td><td>-0.58</td><td>0.92</td><td>-0.55</td><td>-0.82</td><td>0.81</td><td>-0.23</td><td>-0.25</td><td>0.26</td><td>-0.35</td><td>0.33</td><td>0.32</td></tr><tr><td>MFD</td><td>-0.47</td><td>0.84</td><td>-0.44</td><td>-0.68</td><td>0.96</td><td>-0.26</td><td>-0.19</td><td>0.18</td><td>-0.42</td><td>0.42</td><td>0.40</td></tr><tr><td>FFD</td><td>-0.47</td><td>0.82</td><td>-0.43</td><td>-0.63</td><td>0.96</td><td>-0.26</td><td>-0.19</td><td>0.18</td><td>-0.42</td><td>0.42</td><td>0.41</td></tr><tr><td>FPD</td><td>-0.50</td><td>0.91</td><td>-0.50</td><td>-0.78</td><td>0.96</td><td>-0.28</td><td>-0.25</td><td>0.25</td><td>-0.44</td><td>0.43</td><td>0.42</td></tr><tr><td>TRT</td><td>-0.52</td><td>0.95</td><td>-0.54</td><td>-0.82</td><td>0.95</td><td>-0.28</td><td>-0.26</td><td>0.25</td><td>-0.44</td><td>0.44</td><td>0.42</td></tr><tr><td>FPROP</td><td>-0.52</td><td>0.98</td><td>-0.48</td><td>-0.72</td><td>0.99</td><td>-0.30</td><td>-0.25</td><td>0.24</td><td>-0.45</td><td>0.45</td><td>0.44</td></tr><tr><td>REPROP</td><td>-0.58</td><td>0.96</td><td>-0.52</td><td>-0.83</td><td>0.86</td><td>-0.23</td><td>-0.25</td><td>0.26</td><td>-0.36</td><td>0.33</td><td>0.32</td></tr></tbody></table>

the distance to head as the number of words between the current word and its head according to the dependency tree. For example, in the sentence *She reads a mistery novel*, the distance from the word *novel* to its head *reads* is 2. We use the parser *Stanza* (Qi et al., 2020) for the dependency analysis.

**Ambiguity level (AMB):** The meaning of polysemous words can usually be disambiguated by processing the context. The effect of a high ambiguity level on eye movement patterns is therefore usually more pronounced for later processing measures such as NREFIX than for early gaze metrics reflecting lexical access (Foraker and Murphy, 2012; Shen and Li, 2016). We calculate the ambiguity level for each open class word (nouns, adjectives, adverbs, and verbs) as the number of possible senses (synsets) that can be found in the NLTK implementation (Bird and Loper, 2004) of WordNet (Fellbaum). The minimum ambiguity level is 1, which means that there is only one sense for a given wordform. For example, the noun *car* has an ambiguity level of five because it appears in 5 nominal synsets. For this analysis, we exclude words that do not appear in WordNet.

**Age of acquisition (AOA), familiarity (FAM), concreteness (CONC), and imageability (IMAG):** We include four cognitively motivated features of word complexity that are likely to affect fixation durations in reading (Juhasz and Rayner, 2006). Ratings for age of acquisition, familiarity, concreteness, and imageability (the intensity with which a word evokes a clear mental image) in the MRC Psycholinguistic Database (Wilson, 1988) are strongly associated with each other and with other read-

ability metrics.<fnref n="3" /> Paetzold and Specia (2016) find that word frequencies correlate with familiarity and AOA, while the depth of a word in a thesaurus hierarchy correlates with both its concreteness and its imageability. We exclude words that do not appear in the database when calculating the correlations.

## 3.2 Readability and Eye Movement Patterns

We calculate the Spearman correlation coefficients between the recorded eye tracking data and the readability measures for English (Table 2). The strongest correlations can be found for sentence length, word length, and lexical frequency, which confirms three widely studied effects in reading research (Sarti et al., 2021).

**Predictive Power** In a second step, we analyze the correlations between four of the readability measures (FLESCH, WL, ZIPF frequency, SL) and the predictions of the different models for fixation proportion and mean fixation duration for English (see Table 3).<fnref n="4" /> The results show that for fine-tuned mBERT, while yielding lower overall prediction accuracy when aggregating across all features, the correlation of the predicted eye movement values to word frequency and length is generally more similar to the correlation of real gaze features with word frequency and length than for the XLM-100 models. When comparing fine-tuned pre-trained models ($M_{\hat{\omega}}$) to models trained from scratch on eye tracking data ($M_{\hat{z}}$), the results presented in Table 3 also show that the correlation with word length and frequency is stronger in the predictions of the

<footnote n="3">³https://websites.psychology.uwa.edu.au/school/mrcdatabase/uwa_mrc.htm</footnote>

<footnote n="4">⁴Correlations to other gaze features show the same trends. We analyze FPROP because the models yield low prediction performance on this feature, compared to MFD, which yields high prediction results.</footnote>
fine-tuned models and closer to the correlation of the real eye tracking features, showing that while pre-training might not strictly be needed for a high prediction accuracy, it does help the model to predict eye tracking features that are closer to human reading behavior in terms of text readability.

### 3.3 Prediction Errors

To systematically analyze the relationship between readability of the input and predictive power of the model, we focus on the prediction errors. We analyze a sample of 6,396 instances of the test set (20%) and calculate the percentage error (PE; Eq. 1) of the predictions compared to the observed scaled features.

$$PE = \frac{|Prediction - Observed|}{|Observed|} * 100 \quad (1)$$

In Table 4, we present the correlations of the readability measures with the prediction errors for eye tracking features. Imageability, familiarity, concreteness, function words, and Zipf scale frequency values show a moderate correlation to all eye tracking features, which is slightly more notable in the case of the mBERT model. Interestingly, the strongest correlation can be found to fixation proportion. This indicates that the prediction of whether a word will be fixated or not is strongly linked to its imageability and concreteness. Similar tendencies are observed for the correlations between all investigated eye tracking measures and readability measures.

## 4 Word-Level Analysis

As lexical aspects seem to be highly relevant, we additionally analyze the influence of the word class on prediction errors. We focus on words that cause a prediction error ≥ the third quartile value for that feature. Figure 3 shows the aggregated results for all predicted eye movement features.

### 4.1 Word Classes

It can be seen that the large majority of prediction errors can be attributed to function words. This tendency is consistent across all eight gaze features.<fnref n="5" /> Function words such as determiners, pronouns, prepositions and conjunctions, usually trigger low fixation duration and high skipping probability. It has been shown that distributional models

<figure><img src="image_6.png" /><figcaption>Figure 3: Percentage error per part-of-speech class, aggregated across all predicted gaze features.</figcaption></figure>

are generally not well suited for representing function words (Bernardi et al., 2015) and that their representation in transformer-based models such as BERT is highly context-sensitive (Ethayarajh, 2019; Kim et al., 2019; Atanasova et al., 2020). Of the content words, nouns are most often mispredicted and responsible for around 10% of the errors. This is in line with Furtner et al. (2009), who indicated in a reading study that the noun is the most influential word class for facilitating the comprehension of other words.

### 4.2 A Closer Look at Function Words

We have seen that most prediction errors are caused by function words. Function or closed category words are words that are short, frequent, ambiguous, and subject to pragmatic effects in English. They are critical for language understanding.

Reading research has shown that short function words can be identified in reading without a direct fixation (Rayner et al., 1989). Similarly, Barrett and Søgaard (2015) show a negative correlation between function word frequency and fixation probability. Schmauder et al. (2000) found increased processing times in phrases immediately following a low-frequency function word. Function and content words are likely stored and accessed similarly (Diaz and McCarthy, 2009), but have different roles in text processing and constructing discourse representations. Function words show frequency effects in first fixation and first pass duration that are similar to those seen for content words. However, clear differences in reading patterns in the online processing of function and content words emerged in later processing measures (Schmauder

<footnote n="5">⁵Detailed results per feature can be found in the Appendix in Table 7.</footnote>
fine-tuned models and closer to the correlation of the real eye tracking features, showing that while pre-training might not strictly be needed for a high prediction accuracy, it does help the model to predict eye tracking features that are closer to human reading behavior in terms of text readability.

### 3.3 Prediction Errors

To systematically analyze the relationship between readability of the input and predictive power of the model, we focus on the prediction errors. We analyze a sample of 6,396 instances of the test set (20%) and calculate the percentage error (PE; Eq. 1) of the predictions compared to the observed scaled features.

$$PE = \frac{|Prediction - Observed|}{|Observed|} * 100 \quad (1)$$

In Table 4, we present the correlations of the readability measures with the prediction errors for eye tracking features. Imageability, familiarity, concreteness, function words, and Zipf scale frequency values show a moderate correlation to all eye tracking features, which is slightly more notable in the case of the mBERT model. Interestingly, the strongest correlation can be found to fixation proportion. This indicates that the prediction of whether a word will be fixated or not is strongly linked to its imageability and concreteness. Similar tendencies are observed for the correlations between all investigated eye tracking measures and readability measures.

## 4 Word-Level Analysis

As lexical aspects seem to be highly relevant, we additionally analyze the influence of the word class on prediction errors. We focus on words that cause a prediction error ≥ the third quartile value for that feature. Figure 3 shows the aggregated results for all predicted eye movement features.

### 4.1 Word Classes

It can be seen that the large majority of prediction errors can be attributed to function words. This tendency is consistent across all eight gaze features.<fnref n="5" /> Function words such as determiners, pronouns, prepositions and conjunctions, usually trigger low fixation duration and high skipping probability. It has been shown that distributional models

<figure><img src="image_6.png" /><figcaption>Figure 3: Percentage error per part-of-speech class, aggregated across all predicted gaze features.</figcaption></figure>

are generally not well suited for representing function words (Bernardi et al., 2015) and that their representation in transformer-based models such as BERT is highly context-sensitive (Ethayarajh, 2019; Kim et al., 2019; Atanasova et al., 2020). Of the content words, nouns are most often mispredicted and responsible for around 10% of the errors. This is in line with Furtner et al. (2009), who indicated in a reading study that the noun is the most influential word class for facilitating the comprehension of other words.

### 4.2 A Closer Look at Function Words

We have seen that most prediction errors are caused by function words. Function or closed category words are words that are short, frequent, ambiguous, and subject to pragmatic effects in English. They are critical for language understanding.

Reading research has shown that short function words can be identified in reading without a direct fixation (Rayner et al., 1989). Similarly, Barrett and Søgaard (2015) show a negative correlation between function word frequency and fixation probability. Schmauder et al. (2000) found increased processing times in phrases immediately following a low-frequency function word. Function and content words are likely stored and accessed similarly (Diaz and McCarthy, 2009), but have different roles in text processing and constructing discourse representations. Function words show frequency effects in first fixation and first pass duration that are similar to those seen for content words. However, clear differences in reading patterns in the online processing of function and content words emerged in later processing measures (Schmauder

<footnote n="5">⁵Detailed results per feature can be found in the Appendix in Table 7.</footnote>
<table><caption>Table 3: Spearman correlation coefficients between real human eye tracking features or model predictions and word length, word frequency, and sentence length for **fixation proportion** (FPROP) on the left side and for **mean fixation duration** (MFD) on the right side. $M_z$ stands for models trained from scratch and $M_h$ for fine-tuned pre-trained models. Significant results are marked with * ($p$ < 0.01) and results in bold are closest to human eye tracking features.</caption><thead><tr><th rowspan="2">Eye tracking</th><th colspan="4">FPROP</th><th colspan="4">MFD</th></tr><tr><th>FLESCH</th><th>ZIPF</th><th>WL</th><th>SL</th><th>FLESCH</th><th>ZIPF</th><th>WL</th><th>SL</th></tr></thead><tbody><tr><td>Human</td><td>-0.52*</td><td>-0.72*</td><td>0.98*</td><td>0.99*</td><td>-0.47*</td><td>-0.68*</td><td>0.84*</td><td>0.96*</td></tr><tr><td>RANDOM BL</td><td>-0.03</td><td>-0.07</td><td>0.36</td><td>-0.16*</td><td>0.12*</td><td>-0.48</td><td>-0.20</td><td>-0.12*</td></tr><tr><td>MEAN BL</td><td>-0.61*</td><td>0.14*</td><td>-0.31</td><td>0.99*</td><td>-0.41*</td><td>-0.01</td><td>0.23</td><td>0.91*</td></tr><tr><td>M<sub>z</sub> mBERT</td><td>-0.50*</td><td>-0.53*</td><td>0.72*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.52*</td><td>0.61*</td><td><strong>0.98*</strong></td></tr><tr><td>M<sub>z</sub> XLM-100</td><td><strong>-0.52*</strong></td><td>-0.68*</td><td>0.67*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.65*</td><td>0.68*</td><td>0.97*</td></tr><tr><td>M<sub>h</sub> mBERT</td><td><strong>-0.52*</strong></td><td>-0.73*</td><td><strong>0.78*</strong></td><td><strong>0.99*</strong></td><td><strong>-0.48*</strong></td><td><strong>-0.68*</strong></td><td><strong>0.80*</strong></td><td>0.97*</td></tr><tr><td>M<sub>h</sub> XLM-100</td><td>-0.53*</td><td><strong>-0.72*</strong></td><td>0.68*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.62*</td><td>0.62*</td><td><strong>0.98*</strong></td></tr></tbody></table>

<table><caption>Table 4: Correlations between percentage error of the eye tracking predictions and the readability measures (imageability, familiarity, concreteness, function words and Zipf frequencies of words).</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">IMAG</th><th colspan="2">FAM</th><th colspan="2">CONC</th><th colspan="2">ZIPF</th><th colspan="2">FUNCT</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>-0.18</td><td>-0.17</td><td>0.19</td><td>0.14</td><td>-0.19</td><td>-0.17</td><td>0.22</td><td>0.18</td><td>0.20</td><td>0.19</td></tr><tr><td>MFD</td><td>-0.18</td><td>-0.09</td><td>0.16</td><td>0.10</td><td>-0.19</td><td>-0.09</td><td>0.24</td><td>0.13</td><td>0.23</td><td>0.10</td></tr><tr><td>FPROP</td><td>-0.21</td><td>-0.17</td><td>0.24</td><td>0.20</td><td>-0.21</td><td>-0.17</td><td>0.34</td><td>0.26</td><td>0.29</td><td>0.22</td></tr><tr><td>FFD</td><td>-0.17</td><td>-0.08</td><td>0.16</td><td>0.09</td><td>-0.18</td><td>-0.09</td><td>0.23</td><td>0.12</td><td>0.22</td><td>0.10</td></tr><tr><td>FPD</td><td>-0.16</td><td>-0.06</td><td>0.15</td><td>0.07</td><td>-0.17</td><td>-0.06</td><td>0.19</td><td>0.07</td><td>0.18</td><td>0.06</td></tr><tr><td>TRT</td><td>-0.18</td><td>-0.13</td><td>0.16</td><td>0.12</td><td>-0.19</td><td>-0.14</td><td>0.21</td><td>0.16</td><td>0.19</td><td>0.16</td></tr><tr><td>NREFIX</td><td>-0.19</td><td>-0.14</td><td>0.22</td><td>0.16</td><td>-0.21</td><td>-0.15</td><td>0.27</td><td>0.17</td><td>0.23</td><td>0.16</td></tr><tr><td>REPROP</td><td>-0.14</td><td>-0.14</td><td>0.15</td><td>0.16</td><td>-0.16</td><td>-0.16</td><td>0.19</td><td>0.20</td><td>0.16</td><td>0.18</td></tr></tbody></table>

et al., 2000). These findings can be taken as evidence of the different roles the two word types have in sentence processing beyond the lexical level.

We analyze the variance in the observed eye tracking features for prepositions, determiners, and conjunctions. We calculate the standard deviation of each eye tracking feature for the six most frequent words of each category. Generally, there is a high variation in the data across all of the gaze features, and the highest deviation can be observed for mean fixation duration. To illustrate this, in Figure 4 we show the graphics of the six most common prepositions. Determiners and conjunctions are shown in Figures 5 and 6 in the Appendix (standard deviation values can also be found in Table 8 in the Appendix).

The lower fixation proportion on shorter words, together with the immense variation on the fixation duration of function words, is likely the cause for

the difficulty in predicting reading times on this class accurately. The differences in skipping rate (that is, the ratio of words that are not fixated) have various origins. Skipping rate is regulated by word length (Drieghe et al., 2004), which therefore leads to differences across languages (see Fig. 1), and by proficiency, since highly proficient readers show a higher skipping rate (Eskenazi and Folk, 2015). We discuss these challenges in more detail in the next section.

## 5 Methodological Limitations

In this work, we make a few simplifying assumptions that are common in the field, but severely affect the interpretation of the results. We want to discuss these limitations explicitly and encourage methodological research to better address these open challenges.
<table><caption>Table 3: Spearman correlation coefficients between real human eye tracking features or model predictions and word length, word frequency, and sentence length for **fixation proportion** (FPROP) on the left side and for **mean fixation duration** (MFD) on the right side. $M_{z}$ stands for models trained from scratch and $M_{♀}$ for fine-tuned pre-trained models. Significant results are marked with * ($p < 0.01$) and results in bold are closest to human eye tracking features.</caption><thead><tr><th rowspan="2">Eye tracking</th><th colspan="4">FPROP</th><th colspan="4">MFD</th></tr><tr><th>FLESCH</th><th>ZIPF</th><th>WL</th><th>SL</th><th>FLESCH</th><th>ZIPF</th><th>WL</th><th>SL</th></tr></thead><tbody><tr><td>Human</td><td>-0.52*</td><td>-0.72*</td><td>0.98*</td><td>0.99*</td><td>-0.47*</td><td>-0.68*</td><td>0.84*</td><td>0.96*</td></tr><tr><td>RANDOM BL</td><td>-0.03</td><td>-0.07</td><td>0.36</td><td>-0.16*</td><td>0.12*</td><td>-0.48</td><td>-0.20</td><td>-0.12*</td></tr><tr><td>MEAN BL</td><td>-0.61*</td><td>0.14*</td><td>-0.31</td><td>0.99*</td><td>-0.41*</td><td>-0.01</td><td>0.23</td><td>0.91*</td></tr><tr><td>M<sub>z</sub> mBERT</td><td>-0.50*</td><td>-0.53*</td><td>0.72*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.52*</td><td>0.61*</td><td><strong>0.98*</strong></td></tr><tr><td>M<sub>z</sub> XLM-100</td><td><strong>-0.52*</strong></td><td>-0.68*</td><td>0.67*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.65*</td><td>0.68*</td><td>0.97*</td></tr><tr><td>M<sub>♀</sub> mBERT</td><td><strong>-0.52*</strong></td><td>-0.73*</td><td><strong>0.78*</strong></td><td><strong>0.99*</strong></td><td><strong>-0.48*</strong></td><td><strong>-0.68*</strong></td><td><strong>0.80*</strong></td><td>0.97*</td></tr><tr><td>M<sub>♀</sub> XLM-100</td><td>-0.53*</td><td><strong>-0.72*</strong></td><td>0.68*</td><td><strong>0.99*</strong></td><td>-0.49*</td><td>-0.62*</td><td>0.62*</td><td><strong>0.98*</strong></td></tr></tbody></table>

<table><caption>Table 4: Correlations between percentage error of the eye tracking predictions and the readability measures (imageability, familiarity, concreteness, function words and Zipf frequencies of words).</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">IMAG</th><th colspan="2">FAM</th><th colspan="2">CONC</th><th colspan="2">ZIPF</th><th colspan="2">FUNCT</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>-0.18</td><td>-0.17</td><td>0.19</td><td>0.14</td><td>-0.19</td><td>-0.17</td><td>0.22</td><td>0.18</td><td>0.20</td><td>0.19</td></tr><tr><td>MFD</td><td>-0.18</td><td>-0.09</td><td>0.16</td><td>0.10</td><td>-0.19</td><td>-0.09</td><td>0.24</td><td>0.13</td><td>0.23</td><td>0.10</td></tr><tr><td>FPROP</td><td>-0.21</td><td>-0.17</td><td>0.24</td><td>0.20</td><td>-0.21</td><td>-0.17</td><td>0.34</td><td>0.26</td><td>0.29</td><td>0.22</td></tr><tr><td>FFD</td><td>-0.17</td><td>-0.08</td><td>0.16</td><td>0.09</td><td>-0.18</td><td>-0.09</td><td>0.23</td><td>0.12</td><td>0.22</td><td>0.10</td></tr><tr><td>FPD</td><td>-0.16</td><td>-0.06</td><td>0.15</td><td>0.07</td><td>-0.17</td><td>-0.06</td><td>0.19</td><td>0.07</td><td>0.18</td><td>0.06</td></tr><tr><td>TRT</td><td>-0.18</td><td>-0.13</td><td>0.16</td><td>0.12</td><td>-0.19</td><td>-0.14</td><td>0.21</td><td>0.16</td><td>0.19</td><td>0.16</td></tr><tr><td>NREFIX</td><td>-0.19</td><td>-0.14</td><td>0.22</td><td>0.16</td><td>-0.21</td><td>-0.15</td><td>0.27</td><td>0.17</td><td>0.23</td><td>0.16</td></tr><tr><td>REPROP</td><td>-0.14</td><td>-0.14</td><td>0.15</td><td>0.16</td><td>-0.16</td><td>-0.16</td><td>0.19</td><td>0.20</td><td>0.16</td><td>0.18</td></tr></tbody></table>

et al., 2000). These findings can be taken as evidence of the different roles the two word types have in sentence processing beyond the lexical level.

We analyze the variance in the observed eye tracking features for prepositions, determiners, and conjunctions. We calculate the standard deviation of each eye tracking feature for the six most frequent words of each category. Generally, there is a high variation in the data across all of the gaze features, and the highest deviation can be observed for mean fixation duration. To illustrate this, in Figure 4 we show the graphics of the six most common prepositions. Determiners and conjunctions are shown in Figures 5 and 6 in the Appendix (standard deviation values can also be found in Table 8 in the Appendix).

The lower fixation proportion on shorter words, together with the immense variation on the fixation duration of function words, is likely the cause for

the difficulty in predicting reading times on this class accurately. The differences in skipping rate (that is, the ratio of words that are not fixated) have various origins. Skipping rate is regulated by word length (Drieghe et al., 2004), which therefore leads to differences across languages (see Fig. 1), and by proficiency, since highly proficient readers show a higher skipping rate (Eskenazi and Folk, 2015). We discuss these challenges in more detail in the next section.

## 5 Methodological Limitations

In this work, we make a few simplifying assumptions that are common in the field, but severely affect the interpretation of the results. We want to discuss these limitations explicitly and encourage methodological research to better address these open challenges.
<figure><img src="image_7.png" /><figcaption>Figure 4: Standard deviations of the eye tracking features of the most frequent English prepositions. Note that the y-axis scales change in each subplot.</figcaption></figure>

**Token-level alignment** Stimuli for eye tracking are usually pre-processed with high linguistic quality. Neural language models, on the other hand, are optimized for engineering objectives that sometimes compete with linguistic and cognitive plausibility. This discrepancy becomes obvious when inspecting the tokenization. The HuggingFace transformer-based models that we apply expect the use of subword tokenizers. Words such as `close-knit`! that are interpreted as a single unit in the eye tracking data are split into subword tokens (`'close'`, `'#-'`, `'#knit'`) and it remains an open challenge how to align the different units. In our implementation, we assign the same gaze features to all subtokens and choose to compute the loss only with respect to the first subtoken. It is unclear, however, if this is the best strategy as it complicates fine-grained error analyses. For example, in the eye tracking data, punctuation is not separated from the preceding token as they are usually fixated jointly. If we apply part-of-speech tagging on words with attached punctuation signs they might get assigned incorrect tags leading to skewed results. In this work, we resolved this by manually aligning tokenized text with the eye tracking stimuli for the readability analysis. However, this is laborious and limits the size of the analyzed data.

**Aggregating over participants** The second matter we address is the variability between readers. The negligence of individual differences is a well-known issue in cognitive science, leading to a picture of an idealized human that is largely invariant across individuals (Levinson, 2012), and the resulting insights underestimate the extent to which human sentence processing is affected by individual differences (Kidd et al., 2018). Currently, the captured individual differences are merely treated as a source of variance that is controlled for through aggregation. As in this work, most often NLP researchers aggregate across all readers due to evidence

showing that this leads to more robust results regarding model performance (Klerke and Plank, 2019). However, the high variability found in some features, such as FPROP, calls for more careful data preprocessing, possibly by considering additional cognitive tests performed during data collection and performing proper outlier detection. Alternatively, single-subject and cross-subject approaches should also be considered in eye tracking prediction for more practical applications.

**Cross-lingual differences** Finally, eye movements depend on the stimulus and therefore contain language-specific information (Liversedge et al., 2016). Reading patterns can be related to linguistic factors of the reader's native language. Berzak et al. (2017) found evidence that similar languages have more similar reading patterns. Siegelman et al. (2022) found that readers of different languages vary considerably in their skipping rate and that this variability is explained by cross-lingual differences in word length distributions. It is unclear yet to what extent these differences affect the predictions of pre-trained language models. Therefore, more research is required on multilingual models that predict eye tracking in typologically more diverse languages.

# 6 Conclusion

Our results show that transformer models yield high accuracy in predicting cognitive language processing signals which confirms tendencies observed in previous work (e.g., Schrimpf et al., 2020; Michaelov et al., 2021). We go beyond aggregated performance metrics and provide a detailed analysis of the linguistic text complexity factors that underlie the prediction of eye movement patterns.

In a detailed analysis of fine-tuned language models that predict eye tracking features from reading, we found that the models learn more from the fine-tuning on psychometric features than from pre-
<figure><img src="image_7.png" /><figcaption>Figure 4: Standard deviations of the eye tracking features of the most frequent English prepositions. Note that the y-axis scales change in each subplot.</figcaption></figure>

**Token-level alignment** Stimuli for eye tracking are usually pre-processed with high linguistic quality. Neural language models, on the other hand, are optimized for engineering objectives that sometimes compete with linguistic and cognitive plausibility. This discrepancy becomes obvious when inspecting the tokenization. The HuggingFace transformer-based models that we apply expect the use of subword tokenizers. Words such as `close-knit`! that are interpreted as a single unit in the eye tracking data are split into subword tokens (`'close'`, `'#-'`, `'#knit'`) and it remains an open challenge how to align the different units. In our implementation, we assign the same gaze features to all subtokens and choose to compute the loss only with respect to the first subtoken. It is unclear, however, if this is the best strategy as it complicates fine-grained error analyses. For example, in the eye tracking data, punctuation is not separated from the preceding token as they are usually fixated jointly. If we apply part-of-speech tagging on words with attached punctuation signs they might get assigned incorrect tags leading to skewed results. In this work, we resolved this by manually aligning tokenized text with the eye tracking stimuli for the readability analysis. However, this is laborious and limits the size of the analyzed data.

**Aggregating over participants** The second matter we address is the variability between readers. The negligence of individual differences is a well-known issue in cognitive science, leading to a picture of an idealized human that is largely invariant across individuals (Levinson, 2012), and the resulting insights underestimate the extent to which human sentence processing is affected by individual differences (Kidd et al., 2018). Currently, the captured individual differences are merely treated as a source of variance that is controlled for through aggregation. As in this work, most often NLP researchers aggregate across all readers due to evidence

showing that this leads to more robust results regarding model performance (Klerke and Plank, 2019). However, the high variability found in some features, such as FPROP, calls for more careful data preprocessing, possibly by considering additional cognitive tests performed during data collection and performing proper outlier detection. Alternatively, single-subject and cross-subject approaches should also be considered in eye tracking prediction for more practical applications.

**Cross-lingual differences** Finally, eye movements depend on the stimulus and therefore contain language-specific information (Liversedge et al., 2016). Reading patterns can be related to linguistic factors of the reader's native language. Berzak et al. (2017) found evidence that similar languages have more similar reading patterns. Siegelman et al. (2022) found that readers of different languages vary considerably in their skipping rate and that this variability is explained by cross-lingual differences in word length distributions. It is unclear yet to what extent these differences affect the predictions of pre-trained language models. Therefore, more research is required on multilingual models that predict eye tracking in typologically more diverse languages.

# 6 Conclusion

Our results show that transformer models yield high accuracy in predicting cognitive language processing signals which confirms tendencies observed in previous work (e.g., Schrimpf et al., 2020; Michaelov et al., 2021). We go beyond aggregated performance metrics and provide a detailed analysis of the linguistic text complexity factors that underlie the prediction of eye movement patterns.

In a detailed analysis of fine-tuned language models that predict eye tracking features from reading, we found that the models learn more from the fine-tuning on psychometric features than from pre-
training on textual input. However, the pre-trained
models show an advantage over the randomly ini-
tialized models in their correlation to text read-
ability measures. Although pre-training of large
language models is not required to obtain high accu-
racy in predicting eye tracking feature from reading,
it does contribute to a stronger correlation to text
readability measures, making the predictions more
similar to human reading behavior. The models
struggle most to predict accurate eye tracking val-
ues for function words, which are exactly the class
of words that exhibits large ranges in reading times
and skipping rates, together with extreme variabil-
ity between readers. The next step will be to extend
the readability analysis to the other languages and
to discern further between syntactic and semantic
text difficulty and purely structural complexity.

This line of work does not only advance our un-
derstanding of language models and allows to com-
pare their output to human language (Tuckute et al.,
2022), but it also furthers research on new readabil-
ity formulas supported by eye tracking data and ma-
chine learning methods (e.g., González-Garduño
and Søgaard, 2018; Baazeem et al., 2021). We hope
this work can serve as a stepping stone towards a
more detailed evaluation setup for eye movement
prediction from reading.

6.0.1 Acknowledgements

I. Gonzalez-Dios's research was partially funded
by the Basque Government (Excellence Research
Group, IT1570-22).

L. Beinborn's research is part of the VENI project
Vl.Veni.211C.039, which is financed by the Dutch
Research Council (NWO).

L. A. Jäger's research is partially funded by the Ger-
man Federal Ministry of Education and Research
(grant 01|S20043).

References

Pepa Atanasova, Jakob Grue Simonsen, Christina Li-
oma, and Isabelle Augenstein. 2020. A diagnostic
study of explainability techniques for text classifi-
cation. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing
(EMNLP), pages 3256–3274, Online. Association for
Computational Linguistics.

Ibtehal Baazeem, Hend Al-Khalifa, and Abdulmalik
Al-Salman. 2021. Cognitively driven Arabic text
readability assessment using eye-tracking. Applied
Sciences, 11(18):8607.

Maria Barrett and Anders Søgaard. 2015. Reading be-
havior predicts syntactic categories. In Proceedings
of the Nineteenth Conference on Computational Nat-
ural Language Learning, pages 345–349, Beijing,
China. Association for Computational Linguistics.

Lisa Beinborn, Torsten Zesch, and Iryna Gurevych. 2012. Towards fine-grained readability measures for self-directed language learning. In Proceedings of the SLTC 2012 Workshop on NLP for CALL, pages 11–19. Linköping University Electronic Press.

Kepa Bengoetxea and Itziar Gonzalez-Dios. 2021. Multiaztertest: A multilingual analyzer on multiple levels of language for readability assessment. *arXiv preprint arXiv:2109.04870*.

Raffaella Bernardi, Gemma Boleda, Raquel Fernández, and Denis Paperno. 2015. Distributional semantics in use. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 95–101, Lisbon, Portugal. Association for Computational Linguistics.

Yevgeni Berzak, Chie Nakamura, Suzanne Flynn, and Boris Katz. 2017. Predicting native language from gaze. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 541–551, Vancouver, Canada. Association for Computational Linguistics.

Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.

Charles Clifton, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. In Eye Movements: A Window on Mind and Brain, pages 341–371. Elsevier.

Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. *ITL-International Journal of Applied Linguistics*, 165(2):97–135.

Vera Demberg and Frank Keller. 2019. Cognitive models of syntax and sentence processing. In Human Language: From Genes and Brains to Behavior. MIT Press.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Michele T. Diaz and Gregory McCarthy. 2009. A comparison of brain activity evoked by single content and function words: An fMRI investigation of implicit word processing. *Brain Research*, 1282:38–49.
training on textual input. However, the pre-trained
models show an advantage over the randomly ini-
tialized models in their correlation to text read-
ability measures. Although pre-training of large
language models is not required to obtain high accu-
racy in predicting eye tracking feature from reading,
it does contribute to a stronger correlation to text
readability measures, making the predictions more
similar to human reading behavior. The models
struggle most to predict accurate eye tracking val-
ues for function words, which are exactly the class
of words that exhibits large ranges in reading times
and skipping rates, together with extreme variabil-
ity between readers. The next step will be to extend
the readability analysis to the other languages and
to discern further between syntactic and semantic
text difficulty and purely structural complexity.

This line of work does not only advance our un-
derstanding of language models and allows to com-
pare their output to human language (Tuckute et al.,
2022), but it also furthers research on new readabil-
ity formulas supported by eye tracking data and ma-
chine learning methods (e.g., González-Garduño
and Søgaard, 2018; Baazeem et al., 2021). We hope
this work can serve as a stepping stone towards a
more detailed evaluation setup for eye movement
prediction from reading.

6.0.1 Acknowledgements

I. Gonzalez-Dios's research was partially funded
by the Basque Government (Excellence Research
Group, IT1570-22).

L. Beinborn's research is part of the VENI project
Vl.Veni.211C.039, which is financed by the Dutch
Research Council (NWO).

L. A. Jäger's research is partially funded by the Ger-
man Federal Ministry of Education and Research
(grant 01|S20043).

References

Pepa Atanasova, Jakob Grue Simonsen, Christina Li-
oma, and Isabelle Augenstein. 2020. A diagnostic
study of explainability techniques for text classifi-
cation. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing
(EMNLP), pages 3256–3274, Online. Association for
Computational Linguistics.

Ibtehal Baazeem, Hend Al-Khalifa, and Abdulmalik Al-Salman. 2021. Cognitively driven Arabic text readability assessment using eye-tracking. *Applied Sciences*, 11(18):8607.

Maria Barrett and Anders Søgaard. 2015. Reading be-
havior predicts syntactic categories. In Proceedings
of the Nineteenth Conference on Computational Nat-
ural Language Learning, pages 345–349, Beijing,
China. Association for Computational Linguistics.

Lisa Beinborn, Torsten Zesch, and Iryna Gurevych. 2012. Towards fine-grained readability measures for self-directed language learning. In Proceedings of the SLTC 2012 Workshop on NLP for CALL, pages 11–19. Linköping University Electronic Press.

Kepa Bengoetxea and Itziar Gonzalez-Dios. 2021. Multiaztertest: A multilingual analyzer on multiple levels of language for readability assessment. *arXiv preprint arXiv:2109.04870*.

Raffaella Bernardi, Gemma Boleda, Raquel Fernández, and Denis Paperno. 2015. Distributional semantics in use. In Proceedings of the First Workshop on Linking Computational Models of Lexical, Sentential and Discourse-level Semantics, pages 95–101, Lisbon, Portugal. Association for Computational Linguistics.

Yevgeni Berzak, Chie Nakamura, Suzanne Flynn, and Boris Katz. 2017. Predicting native language from gaze. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 541–551, Vancouver, Canada. Association for Computational Linguistics.

Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics.

Charles Clifton, Adrian Staub, and Keith Rayner. 2007. Eye movements in reading words and sentences. In Eye Movements: A Window on Mind and Brain, pages 341–371. Elsevier.

Kevyn Collins-Thompson. 2014. Computational assessment of text readability: A survey of current and future research. *ITL-International Journal of Applied Linguistics*, 165(2):97–135.

Vera Demberg and Frank Keller. 2019. Cognitive models of syntax and sentence processing. In Human Language: From Genes and Brains to Behavior. MIT Press.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Michele T. Diaz and Gregory McCarthy. 2009. A comparison of brain activity evoked by single content and function words: An fMRI investigation of implicit word processing. *Brain Research*, 1282:38–49.
Denis Drieghe, Marc Brysbaert, Timothy Desmet, and Constantijn De Baecke. 2004. Word skipping in reading: On the interplay of linguistic and visual factors. *European Journal of Cognitive Psychology*, 16(1-2):79–103.

Michael A. Eskenazi and Jocelyn R. Folk. 2015. Reading skill and word skipping: Implications for visual and linguistic accounts of word skipping. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 41(6):1923.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 55–65, Hong Kong, China. Association for Computational Linguistics.

Christiane Fellbaum, editor. *WordNet: An Electronic Lexical Database*. MIT Press, Cambridge, MA.

Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readability assessment. In *Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)*, pages 229–237, Athens, Greece. Association for Computational Linguistics.

Rudolph Flesch. 1948. A new readability yardstick. *Journal of Applied Psychology*, 32(3):221.

Stephani Foraker and Gregory Murphy. 2012. Polyseny in sentence comprehension: Effects of meaning dominance. *Journal of Memory and Language*, 67(4):407–425.

Marco R Furtner, John F Rauthmann, and Pierre Sachse. 2009. Nomen est omen: Investigating the dominance of nouns in word comprehension with eye movement analyses. *Advances in Cognitive Psychology*, 5:91.

Ana Valeria González-Garduño and Anders Søgaard. 2018. Learning to predict readability using eye-movement data from natives and learners. In *Proceedings of the 32nd AAAI Conference on Artificial Intelligence*, pages 5118–5124.

John T Hale, Luca Campanelli, Jixing Li, Shohini Bhat-tasali, Christophe Pallier, and Jonathan R Brennan. 2022. Neurocomputational models of language processing. *Annual Review of Linguistics*, 8:427–446.

Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang. 2019. Advancing NLP with cognitive language processing signals. *arXiv preprint arXiv:1904.02682*.

Nora Hollenstein, Emmanuele Chersoni, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus. 2022. CMCL 2022 shared task on multilingual and crosslingual prediction of human reading behavior.

In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 121–129, Dublin, Ireland. Association for Computational Linguistics.

Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, and Lisa Beinborn. 2021. Multilingual language models predict human reading behavior. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 106–123, Online. Association for Computational Linguistics.

Barbara J Juhasz and Keith Rayner. 2006. The role of age of acquisition and word frequency in reading: Evidence from eye fixation durations. *Visual Cognition*, 13(7-8):846–863.

Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. *Psychological Review*, 87(4):329.

Evan Kidd, Seamus Donnelly, and Morten H Christiansen. 2018. Individual differences in language acquisition and processing. *Trends in Cognitive Science*, 22(2):154–169.

Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In *Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)*, pages 235–249, Minneapolis, Minnesota. Association for Computational Linguistics.

Sigrid Klerke and Barbara Plank. 2019. At a glance: The impact of gaze aggregation views on syntactic tagging. In *Proceedings of the Beyond Vision and Language: inTEgrating Real-world Knowledge (LANTERN)*, pages 51–61, Hong Kong, China. Association for Computational Linguistics.

Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In *Advances in Neural Information Processing Systems 32 (NeurIPS 2019)*.

Stephen C. Levinson. 2012. The original sin of cognitive science. *Topics in Cognitive Science*, 4(3):396–403.

Simon P Liversedge, Denis Drieghe, Xin Li, Guoli Yan, Xuejun Bai, and Jukka Hyönä. 2016. Universality in eye movements and reading: A trilingual investigation. *Cognition*, 147:1–20.

Alessandro Lopopolo, Stefan L. Frank, Antal van den Bosch, and Roel Willems. 2019. Dependency parsing with your eyes: Dependency structure predicts eye regressions during reading. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 77–85, Minneapolis, Minnesota. Association for Computational Linguistics.
Denis Drieghe, Marc Brysbaert, Timothy Desmet, and Constantijn De Baecke. 2004. Word skipping in reading: On the interplay of linguistic and visual factors. *European Journal of Cognitive Psychology*, 16(1-2):79–103.

Michael A. Eskenazi and Jocelyn R. Folk. 2015. Reading skill and word skipping: Implications for visual and linguistic accounts of word skipping. *Journal of Experimental Psychology: Learning, Memory, and Cognition*, 41(6):1923.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 55–65, Hong Kong, China. Association for Computational Linguistics.

Christiane Fellbaum, editor. *WordNet: An Electronic Lexical Database*. MIT Press, Cambridge, MA.

Lijun Feng, Noémie Elhadad, and Matt Huenerfauth. 2009. Cognitively motivated features for readability assessment. In *Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)*, pages 229–237, Athens, Greece. Association for Computational Linguistics.

Rudolph Flesch. 1948. A new readability yardstick. *Journal of Applied Psychology*, 32(3):221.

Stephani Foraker and Gregory Murphy. 2012. Polysemy in sentence comprehension: Effects of meaning dominance. *Journal of Memory and Language*, 67(4):407–425.

Marco R Furtner, John F Rauthmann, and Pierre Sachse. 2009. Nomen est omen: Investigating the dominance of nouns in word comprehension with eye movement analyses. *Advances in Cognitive Psychology*, 5:91.

Ana Valeria González-Garduño and Anders Søgaard. 2018. Learning to predict readability using eye-movement data from natives and learners. In *Proceedings of the 32nd AAAI Conference on Artificial Intelligence*, pages 5118–5124.

John T Hale, Luca Campanelli, Jixing Li, Shohini Bhattasali, Christophe Pallier, and Jonathan R Brennan. 2022. Neurocomputational models of language processing. *Annual Review of Linguistics*, 8:427–446.

Nora Hollenstein, Maria Barrett, Marius Troendle, Francesco Bigiolli, Nicolas Langer, and Ce Zhang. 2019. Advancing NLP with cognitive language processing signals. *arXiv preprint arXiv:1904.02682*.

Nora Hollenstein, Emmanuele Chersoni, Cassandra Jacobs, Yohei Oseki, Laurent Prévot, and Enrico Santus. 2022. CMCL 2022 shared task on multilingual and crosslingual prediction of human reading behavior.

In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 121–129, Dublin, Ireland. Association for Computational Linguistics.

Nora Hollenstein, Federico Pirovano, Ce Zhang, Lena Jäger, and Lisa Beinborn. 2021. Multilingual language models predict human reading behavior. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 106–123, Online. Association for Computational Linguistics.

Barbara J Juhasz and Keith Rayner. 2006. The role of age of acquisition and word frequency in reading: Evidence from eye fixation durations. *Visual Cognition*, 13(7-8):846–863.

Marcel A. Just and Patricia A. Carpenter. 1980. A theory of reading: From eye fixations to comprehension. *Psychological Review*, 87(4):329.

Evan Kidd, Seamus Donnelly, and Morten H Christiansen. 2018. Individual differences in language acquisition and processing. *Trends in Cognitive Science*, 22(2):154–169.

Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In *Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)*, pages 235–249, Minneapolis, Minnesota. Association for Computational Linguistics.

Sigrid Klerke and Barbara Plank. 2019. At a glance: The impact of gaze aggregation views on syntactic tagging. In *Proceedings of the Beyond Vision and Language: inTEgrating Real-world Knowledge (LANTERN)*, pages 51–61, Hong Kong, China. Association for Computational Linguistics.

Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In *Advances in Neural Information Processing Systems 32 (NeurIPS 2019)*.

Stephen C. Levinson. 2012. The original sin of cognitive science. *Topics in Cognitive Science*, 4(3):396–403.

Simon P Liversedge, Denis Drieghe, Xin Li, Guoli Yan, Xuejun Bai, and Jukka Hyönä. 2016. Universality in eye movements and reading: A trilingual investigation. *Cognition*, 147:1–20.

Alessandro Lopopolo, Stefan L. Frank, Antal van den Bosch, and Roel Willems. 2019. Dependency parsing with your eyes: Dependency structure predicts eye regressions during reading. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 77–85, Minneapolis, Minnesota. Association for Computational Linguistics.
Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharyya. 2020. A survey on using gaze behaviour for natural language processing. *Proceedings of IJCAI*.

Danny Merkx and Stefan L. Frank. 2021. Human sentence processing: Recurrence or attention? In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 12–22, Online. Association for Computational Linguistics.

James A Michaelov, Megan D Bardolph, Seana Coulson, and Benjamin Bergen. 2021. Different kinds of cognitive plausibility: Why are transformers better than RNNs at predicting N400 amplitude? In *Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 43.

Shingo Nahatame. 2021. Text readability and processing effort in second language reading: A computational and eye-tracking investigation. *Language learning*.

Gustavo Paetzold and Lucia Specia. 2016. Inferring psycholinguistic properties of words. In *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 435–440, San Diego, California. Association for Computational Linguistics.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 101–108, Online. Association for Computational Linguistics.

Keith Rayner. 1977. Visual attention in reading: Eye movements reflect cognitive processes. *Memory & Cognition*, 5(4):443–448.

Keith Rayner, Kathryn H Chace, Timothy J. Slattery, and Jane Ashby. 2006. Eye movements as reflections of comprehension processes in reading. *Scientific Studies of Reading*, 10(3):241–255.

Keith Rayner, Sara C Sereno, Robin K Morris, A Rene Schmauder, and Charles Clifton Jr. 1989. Eye movements and on-line language comprehension processes. *Language and Cognitive Processes*, 4(3-4):S121–S149.

Lavinia Salicchi, Rong Xiang, and Yu-Yin Hsu. 2022. HkAmsters at CMCL 2022 shared task: Predicting eye-tracking data from a gradient boosting framework with linguistic features. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 114–120, Dublin, Ireland. Association for Computational Linguistics.

Gabriele Sarti, Dominique Brunato, and Felice Dell'Orletta. 2021. That looks hard: Characterizing linguistic complexity in humans and language

models. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 48–60, Online. Association for Computational Linguistics.

A René Schmauder, Robin K Morris, and David V Poynor. 2000. Lexical processing and text integration of function and content words: Evidence from priming and eye fixations. *Memory & Cognition*, 28(7):1098–1108.

Martin Schrimpf, Idan A Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy G Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2020. Artificial neural networks accurately predict language processing in the brain. *BioRxiv*.

Wei Shen and Xingshan Li. 2016. Processing and representation of ambiguous words in Chinese reading: Evidence from eye movements. *Frontiers in Psychology*, 7:1713.

Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, et al. 2022. Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO). *Behavior Research Methods*, pages 1–21.

Abhinav Deep Singh, Poojan Mehta, Samar Husain, and Rajkumar Rajakrishnan. 2016. Quantifying sentence complexity based on eye-tracking measures. In *Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC)*, pages 202–212, Osaka, Japan. The COLING 2016 Organizing Committee.

Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. 2018. LuminosoInsight/wordfreq: v2.2.

Harshvardhan Srivastava. 2022. Poirot at CMCL 2022 shared task: Zero shot crosslingual eye-tracking data prediction using multilingual transformer models. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 102–107, Dublin, Ireland. Association for Computational Linguistics.

Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Towards making a dependency parser see. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 1500–1506, Hong Kong, China. Association for Computational Linguistics.

Ece Takmaz. 2022. Team DMG at CMCL 2022 shared task: Transformer adapters for the multi- and cross-lingual prediction of human reading behavior. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 136–144, Dublin, Ireland. Association for Computational Linguistics.
Sandeep Mathias, Diptesh Kanojia, Abhijit Mishra, and Pushpak Bhattacharyya. 2020. A survey on using gaze behaviour for natural language processing. *Proceedings of IJCAI*.

Danny Merkx and Stefan L. Frank. 2021. Human sentence processing: Recurrence or attention? In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 12–22, Online. Association for Computational Linguistics.

James A Michaelov, Megan D Bardolph, Seana Coulson, and Benjamin Bergen. 2021. Different kinds of cognitive plausibility: Why are transformers better than RNNs at predicting N400 amplitude? In *Proceedings of the Annual Meeting of the Cognitive Science Society*, volume 43.

Shingo Nahatame. 2021. Text readability and processing effort in second language reading: A computational and eye-tracking investigation. *Language learning*.

Gustavo Paetzold and Lucia Specia. 2016. Inferring psycholinguistic properties of words. In *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 435–440, San Diego, California. Association for Computational Linguistics.

Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations*, pages 101–108, Online. Association for Computational Linguistics.

Keith Rayner. 1977. Visual attention in reading: Eye movements reflect cognitive processes. *Memory & Cognition*, 5(4):443–448.

Keith Rayner, Kathryn H Chace, Timothy J. Slattery, and Jane Ashby. 2006. Eye movements as reflections of comprehension processes in reading. *Scientific Studies of Reading*, 10(3):241–255.

Keith Rayner, Sara C Sereno, Robin K Morris, A Rene Schmauder, and Charles Clifton Jr. 1989. Eye movements and on-line language comprehension processes. *Language and Cognitive Processes*, 4(3-4):S121–S149.

Lavinia Salicchi, Rong Xiang, and Yu-Yin Hsu. 2022. HkAmsters at CMCL 2022 shared task: Predicting eye-tracking data from a gradient boosting framework with linguistic features. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 114–120, Dublin, Ireland. Association for Computational Linguistics.

Gabriele Sarti, Dominique Brunato, and Felice Dell'Orletta. 2021. That looks hard: Characterizing linguistic complexity in humans and language

models. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 48–60, Online. Association for Computational Linguistics.

A René Schmauder, Robin K Morris, and David V Poynor. 2000. Lexical processing and text integration of function and content words: Evidence from priming and eye fixations. *Memory & Cognition*, 28(7):1098–1108.

Martin Schrimpf, Idan A Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy G Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2020. Artificial neural networks accurately predict language processing in the brain. *BioRxiv*.

Wei Shen and Xingshan Li. 2016. Processing and representation of ambiguous words in Chinese reading: Evidence from eye movements. *Frontiers in Psychology*, 7:1713.

Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brysbaert, Daria Chernova, et al. 2022. Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO). *Behavior Research Methods*, pages 1–21.

Abhinav Deep Singh, Poojan Mehta, Samar Husain, and Rajkumar Rajakrishnan. 2016. Quantifying sentence complexity based on eye-tracking measures. In *Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC)*, pages 202–212, Osaka, Japan. The COLING 2016 Organizing Committee.

Robyn Speer, Joshua Chin, Andrew Lin, Sara Jewett, and Lance Nathan. 2018. LuminosoInsight/wordfreq: v2.2.

Harshvardhan Srivastava. 2022. Poirot at CMCL 2022 shared task: Zero shot crosslingual eye-tracking data prediction using multilingual transformer models. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 102–107, Dublin, Ireland. Association for Computational Linguistics.

Michalina Strzyz, David Vilares, and Carlos Gómez-Rodríguez. 2019. Towards making a dependency parser see. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 1500–1506, Hong Kong, China. Association for Computational Linguistics.

Ece Takmaz. 2022. Team DMG at CMCL 2022 shared task: Transformer adapters for the multi- and cross-lingual prediction of human reading behavior. In *Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics*, pages 136–144, Dublin, Ireland. Association for Computational Linguistics.
Greta Tuckute, Aalok Sathe, Mingye Wang, Harley Yoder, Cory Shain, and Evelina Fedorenko. 2022. *SentSpace: Large-scale benchmarking and evaluation of text using cognitively motivated lexical, syntactic, and semantic features.* In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations*, pages 99–113, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.

Sowmya Vajjala Balakrishna. 2015. *Analyzing Text Complexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications*. Ph.D. thesis, Universität Tübingen.

Walter JB Van Heuven, Pawel Mandera, Emmanuel Keuleers, and Marc Brysbaert. 2014. SUBTLEX-UK: A new and improved word frequency database for British English. *Quarterly Journal of Experimental Psychology*, 67(6):1176–1190.

Shravan Vasishth, Titus von der Malsburg, and Felix Engelmann. 2013. What eye movements can tell us about sentence comprehension. *Wiley Interdisciplinary Reviews: Cognitive Science*, 4(2):125–134.

Daniel Wiechmann, Yu Qiao, Elma Kerz, and Justus Mattern. 2022. *Measuring the impact of (psycholinguistic and readability features and their spill over effects on the prediction of eye movement patterns)*. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 5276–5290, Dublin, Ireland. Association for Computational Linguistics.

Michael Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. *Behavior Research Methods, Instruments, & Computers*, 20(1):6–10.
Greta Tuckute, Aalok Sathe, Mingye Wang, Harley Yoder, Cory Shain, and Evelina Fedorenko. 2022. *SentSpace: Large-scale benchmarking and evaluation of text using cognitively motivated lexical, syntactic, and semantic features.* In *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations*, pages 99–113, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.

Sowmya Vajjala Balakrishna. 2015. *Analyzing Text Complexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications*. Ph.D. thesis, Universität Tübingen.

Walter JB Van Heuven, Pawel Mandera, Emmanuel Keuleers, and Marc Brysbaert. 2014. SUBTLEX-UK: A new and improved word frequency database for British English. *Quarterly Journal of Experimental Psychology*, 67(6):1176–1190.

Shravan Vasishth, Titus von der Malsburg, and Felix Engelmann. 2013. What eye movements can tell us about sentence comprehension. *Wiley Interdisciplinary Reviews: Cognitive Science*, 4(2):125–134.

Daniel Wiechmann, Yu Qiao, Elma Kerz, and Justus Mattern. 2022. *Measuring the impact of (psycholinguistic and readability features and their spill over effects on the prediction of eye movement patterns)*. In *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 5276–5290, Dublin, Ireland. Association for Computational Linguistics.

Michael Wilson. 1988. MRC psycholinguistic database: Machine-usable dictionary, version 2.00. *Behavior Research Methods, Instruments, & Computers*, 20(1):6–10.
# A Appendix

## A.1 Eye Tracking Corpora

The details of the datasets used in this work are presented in Table 5.

<table><caption>Table 5: Descriptive statistics of all eye tracking datasets. Sentence length and word length are expressed as the mean with the min-max range in parentheses. The last column shows the Flesch Reading Ease score (Flesch, 1948) which ranges from 0 to 100 (higher score indicates easier to read). Adaptations of the Flesch score were used for Dutch (NL), German (DE) and Russian (RU).</caption><thead><tr><th>Language</th><th>Corpus</th><th>Subjs.</th><th>Sents.</th><th>Sent. length</th><th>Tokens</th><th>Types</th><th>Word length</th><th>Flesch</th></tr></thead><tbody><tr><td rowspan="3">English</td><td>Dundee</td><td>10</td><td>2,379</td><td>21.7 (1–87)</td><td>51,497</td><td>9,488</td><td>4.9 (1–20)</td><td>53.3</td></tr><tr><td>GECO</td><td>14</td><td>5,373</td><td>10.5 (1–69)</td><td>56,410</td><td>5,916</td><td>4.6 (1–33)</td><td>77.4</td></tr><tr><td>ZuCo</td><td>30</td><td>1,053</td><td>19.5 (1–68)</td><td>20,545</td><td>5,560</td><td>5.0 (1–29)</td><td>50.6</td></tr><tr><td>Dutch</td><td>GECO</td><td>19</td><td>5,190</td><td>11.64 (1–60)</td><td>59,716</td><td>5,575</td><td>4.5 (1–22)</td><td>57.5</td></tr><tr><td>German</td><td>PoTeC</td><td>30</td><td>97</td><td>19.5 (5–51)</td><td>1,895</td><td>847</td><td>6.5 (2–33)</td><td>36.4</td></tr><tr><td>Russian</td><td>RSC</td><td>103</td><td>144</td><td>9.4 (5–13)</td><td>1,357</td><td>993</td><td>5.7 (1–18)</td><td>64.7</td></tr></tbody></table>

## A.2 Eye Movement Features

The values of the eye tracking features vary over different ranges (see Figure 1, left-most subplots). FFD, for example, is measured in milliseconds, and average values are around 200 ms, whereas REPROP is a proportional measure, and therefore assumes floating-point values between 0 and 1. We standardize all eye tracking features independently (range: 0–100), so that the loss can be calculated uniformly over all feature dimensions.

## A.3 Additional Correlation Results

In Table 6, we present additional correlations of the readability measures in relation to the eye tracking prediction errors.

<table><caption>Table 6: Correlations between percentage error and readability measures (age of acquisition, distance to head, and ambiguity level).</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">AOA</th><th colspan="2">D2H</th><th colspan="2">AMB</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>0.00</td><td>0.02</td><td>-0.03</td><td>-0.05</td><td>0.04</td><td>0.01</td></tr><tr><td>MFD</td><td>-0.09</td><td>-0.04</td><td>0.00</td><td>-0.03</td><td>0.04</td><td>0.00</td></tr><tr><td>FPROP</td><td>-0.18</td><td>-0.12</td><td>-0.03</td><td>-0.04</td><td>0.13</td><td>0.05</td></tr><tr><td>FFD</td><td>-0.09</td><td>-0.04</td><td>-0.01</td><td>-0.03</td><td>0.02</td><td>-0.01</td></tr><tr><td>FPD</td><td>-0.06</td><td>-0.01</td><td>0.00</td><td>-0.02</td><td>0.00</td><td>-0.04</td></tr><tr><td>TRT</td><td>0.02</td><td>-0.05</td><td>-0.02</td><td>-0.04</td><td>0.01</td><td>0.01</td></tr><tr><td>NREFIX</td><td>-0.01</td><td>0.01</td><td>0.00</td><td>-0.03</td><td>0.14</td><td>0.09</td></tr><tr><td>REPROP</td><td>0.05</td><td>0.01</td><td>0.01</td><td>-0.01</td><td>0.13</td><td>0.12</td></tr></tbody></table>

## A.4 Error Rate by Part-of-Speech

Table 7 shows the percentage error for each part-of-speech class and each eye tracking feature.
# A Appendix

## A.1 Eye Tracking Corpora

The details of the datasets used in this work are presented in Table 5.

<table><caption>Table 5: Descriptive statistics of all eye tracking datasets. Sentence length and word length are expressed as the mean with the min-max range in parentheses. The last column shows the Flesch Reading Ease score (Flesch, 1948) which ranges from 0 to 100 (higher score indicates easier to read). Adaptations of the Flesch score were used for Dutch (NL), German (DE) and Russian (RU).</caption><thead><tr><th>Language</th><th>Corpus</th><th>Subjs.</th><th>Sents.</th><th>Sent. length</th><th>Tokens</th><th>Types</th><th>Word length</th><th>Flesch</th></tr></thead><tbody><tr><td rowspan="3">English</td><td>Dundee</td><td>10</td><td>2,379</td><td>21.7 (1–87)</td><td>51,497</td><td>9,488</td><td>4.9 (1–20)</td><td>53.3</td></tr><tr><td>GECO</td><td>14</td><td>5,373</td><td>10.5 (1–69)</td><td>56,410</td><td>5,916</td><td>4.6 (1–33)</td><td>77.4</td></tr><tr><td>ZuCo</td><td>30</td><td>1,053</td><td>19.5 (1–68)</td><td>20,545</td><td>5,560</td><td>5.0 (1–29)</td><td>50.6</td></tr><tr><td>Dutch</td><td>GECO</td><td>19</td><td>5,190</td><td>11.64 (1–60)</td><td>59,716</td><td>5,575</td><td>4.5 (1–22)</td><td>57.5</td></tr><tr><td>German</td><td>PoTeC</td><td>30</td><td>97</td><td>19.5 (5–51)</td><td>1,895</td><td>847</td><td>6.5 (2–33)</td><td>36.4</td></tr><tr><td>Russian</td><td>RSC</td><td>103</td><td>144</td><td>9.4 (5–13)</td><td>1,357</td><td>993</td><td>5.7 (1–18)</td><td>64.7</td></tr></tbody></table>

## A.2 Eye Movement Features

The values of the eye tracking features vary over different ranges (see Figure 1, left-most subplots). FFD, for example, is measured in milliseconds, and average values are around 200 ms, whereas REPROP is a proportional measure, and therefore assumes floating-point values between 0 and 1. We standardize all eye tracking features independently (range: 0–100), so that the loss can be calculated uniformly over all feature dimensions.

## A.3 Additional Correlation Results

In Table 6, we present additional correlations of the readability measures in relation to the eye tracking prediction errors.

<table><caption>Table 6: Correlations between percentage error and readability measures (age of acquisition, distance to head, and ambiguity level).</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">AOA</th><th colspan="2">D2H</th><th colspan="2">AMB</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>0.00</td><td>0.02</td><td>-0.03</td><td>-0.05</td><td>0.04</td><td>0.01</td></tr><tr><td>MFD</td><td>-0.09</td><td>-0.04</td><td>0.00</td><td>-0.03</td><td>0.04</td><td>0.00</td></tr><tr><td>FPROP</td><td>-0.18</td><td>-0.12</td><td>-0.03</td><td>-0.04</td><td>0.13</td><td>0.05</td></tr><tr><td>FFD</td><td>-0.09</td><td>-0.04</td><td>-0.01</td><td>-0.03</td><td>0.02</td><td>-0.01</td></tr><tr><td>FPD</td><td>-0.06</td><td>-0.01</td><td>0.00</td><td>-0.02</td><td>0.00</td><td>-0.04</td></tr><tr><td>TRT</td><td>0.02</td><td>-0.05</td><td>-0.02</td><td>-0.04</td><td>0.01</td><td>0.01</td></tr><tr><td>NREFIX</td><td>-0.01</td><td>0.01</td><td>0.00</td><td>-0.03</td><td>0.14</td><td>0.09</td></tr><tr><td>REPROP</td><td>0.05</td><td>0.01</td><td>0.01</td><td>-0.01</td><td>0.13</td><td>0.12</td></tr></tbody></table>

## A.4 Error Rate by Part-of-Speech

Table 7 shows the percentage error for each part-of-speech class and each eye tracking feature.
<table><caption>Table 7: Percentage error for each part-of-speech class and each eye tracking feature.</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">FUNC</th><th colspan="2">ADJ</th><th colspan="2">ADV</th><th colspan="2">NOUN</th><th colspan="2">PROPN</th><th colspan="2">VERB</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>68.54</td><td>66.10</td><td>4.69</td><td>5.44</td><td>4.63</td><td>4.82</td><td>9.38</td><td>15.76</td><td>6.38</td><td>8.26</td><td>6.38</td><td>7.88</td></tr><tr><td>MFD</td><td>68.86</td><td>62.16</td><td>5.32</td><td>6.32</td><td>4.57</td><td>5.94</td><td>9.63</td><td>17.39</td><td>4.69</td><td>7.50</td><td>6.94</td><td>8.19</td></tr><tr><td>FPROP</td><td>72.17</td><td>69.17</td><td>4.57</td><td>4.75</td><td>4.50</td><td>4.44</td><td>8.13</td><td>14.82</td><td>4.38</td><td>6.82</td><td>6.25</td><td>6.82</td></tr><tr><td>FFD</td><td>67.98</td><td>62.48</td><td>5.25</td><td>6.25</td><td>4.63</td><td>5.88</td><td>10.44</td><td>16.89</td><td>4.75</td><td>7.13</td><td>6.94</td><td>8.51</td></tr><tr><td>FPD</td><td>68.36</td><td>58.91</td><td>5.25</td><td>7.32</td><td>4.44</td><td>6.19</td><td>9.82</td><td>18.89</td><td>5.19</td><td>8.51</td><td>6.94</td><td>8.69</td></tr><tr><td>TRT</td><td>68.36</td><td>66.60</td><td>16.60</td><td>6.57</td><td>14.03</td><td>5.19</td><td>31.03</td><td>13.07</td><td>16.40</td><td>6.94</td><td>21.94</td><td>8.57</td></tr><tr><td>NREFIX</td><td>75.88</td><td>78.70</td><td>3.71</td><td>3.71</td><td>3.61</td><td>3.61</td><td>7.46</td><td>7.46</td><td>2.82</td><td>2.82</td><td>6.52</td><td>6.52</td></tr><tr><td>REPROP</td><td>75.88</td><td>78.70</td><td>3.71</td><td>3.71</td><td>3.61</td><td>3.61</td><td>7.46</td><td>7.46</td><td>2.82</td><td>2.82</td><td>6.52</td><td>6.52</td></tr></tbody></table>

## A.5 Standard Deviations of Function Words

Table 8 shows the standard deviations of the predictions of the most frequent function word classes, namely prepositions (PREP), determiners (DET), and conjunctions (CONJ).

<table><caption>Table 8: Standard deviation of the most frequent function words.</caption><thead><tr><th>Lemma</th><th>NFIX</th><th>FFD</th><th>FPD</th><th>TRT</th><th>MFD</th><th>FPROP</th><th>NREFIX</th><th>REPROP</th></tr></thead><tbody><tr><td rowspan="10">PREP</td><td>at</td><td>0.20</td><td>33.87</td><td>35.96</td><td>42.79</td><td><strong>4562.23</strong></td><td>0.15</td><td>10.10</td><td>10.10</td></tr><tr><td>by</td><td>0.25</td><td>43.47</td><td>44.57</td><td>51.42</td><td><strong>5109.68</strong></td><td>0.19</td><td>0.09</td><td>0.07</td></tr><tr><td>for</td><td>0.26</td><td><strong>8365.60</strong></td><td>42.11</td><td><strong>14199.50</strong></td><td><strong>6347.25</strong></td><td>0.19</td><td>0.13</td><td>27.87</td></tr><tr><td>from</td><td>0.25</td><td>39.08</td><td>40.07</td><td><strong>12828.73</strong></td><td>38.53</td><td>0.18</td><td>0.11</td><td>35.43</td></tr><tr><td>in</td><td>23.15</td><td><strong>1859.10</strong></td><td><strong>1859.05</strong></td><td>41.57</td><td><strong>2782.60</strong></td><td>23.15</td><td>0.07</td><td>0.06</td></tr><tr><td>of</td><td>12.92</td><td><strong>4158.18</strong></td><td><strong>4158.11</strong></td><td><strong>3839.08</strong></td><td><strong>5614.73</strong></td><td>12.92</td><td>4.31</td><td>4.31</td></tr><tr><td>on</td><td>41.64</td><td>35.31</td><td>36.95</td><td><strong>8120.68</strong></td><td><strong>6406.01</strong></td><td>0.16</td><td>8.33</td><td>8.33</td></tr><tr><td>to</td><td>7.07</td><td>35.79</td><td>37.38</td><td><strong>2073.26</strong></td><td><strong>3562.01</strong></td><td>7.07</td><td>0.08</td><td>0.07</td></tr><tr><td>up</td><td>0.23</td><td>46.17</td><td>48.59</td><td>52.53</td><td>46.42</td><td>0.20</td><td>0.07</td><td>0.06</td></tr><tr><td>with</td><td>0.31</td><td>43.47</td><td>47.71</td><td>62.36</td><td><strong>18589.94</strong></td><td>0.18</td><td>0.19</td><td>0.13</td></tr><tr><td rowspan="10">DET</td><td>a</td><td>12.48</td><td><strong>1168.31</strong></td><td><strong>1168.30</strong></td><td><strong>1168.27</strong></td><td><strong>6726.54</strong></td><td>17.64</td><td>0.07</td><td>0.05</td></tr><tr><td>all</td><td>0.24</td><td>43.44</td><td>47.06</td><td>57.88</td><td><strong>11703.01</strong></td><td>0.18</td><td>0.12</td><td>0.08</td></tr><tr><td>another</td><td>0.44</td><td>56.89</td><td>70.55</td><td>98.66</td><td>54.99</td><td>0.23</td><td>0.29</td><td>0.18</td></tr><tr><td>any</td><td>0.23</td><td>40.18</td><td>42.81</td><td>50.25</td><td>39.31</td><td>0.17</td><td>0.12</td><td>0.11</td></tr><tr><td>every</td><td>0.33</td><td>40.36</td><td>46.56</td><td>48.93</td><td><strong>36800.71</strong></td><td>0.15</td><td>0.25</td><td>0.14</td></tr><tr><td>no</td><td>0.25</td><td>39.11</td><td>40.87</td><td>50.50</td><td>38.53</td><td>0.18</td><td>0.10</td><td>0.08</td></tr><tr><td>some</td><td>0.31</td><td>46.57</td><td>49.31</td><td>60.57</td><td><strong>26367.42</strong></td><td>0.18</td><td>0.20</td><td>0.11</td></tr><tr><td>that</td><td>0.26</td><td>41.07</td><td>45.10</td><td>53.25</td><td>40.90</td><td>0.18</td><td>0.15</td><td>0.11</td></tr><tr><td>the</td><td>9.06</td><td><strong>2060.29</strong></td><td>919.18</td><td><strong>2349.22</strong></td><td><strong>5173.09</strong></td><td>9.06</td><td>0.11</td><td>0.08</td></tr><tr><td>this</td><td>0.32</td><td>47.54</td><td>53.64</td><td>67.83</td><td><strong>26854.66</strong></td><td>0.19</td><td>0.18</td><td>0.12</td></tr><tr><td rowspan="10">CONJ</td><td>and</td><td>0.27</td><td>37.11</td><td>41.45</td><td>48.52</td><td><strong>5318.58</strong></td><td>32.99</td><td>0.11</td><td>0.08</td></tr><tr><td>as</td><td>0.22</td><td>38.20</td><td>39.10</td><td>44.08</td><td><strong>10602.03</strong></td><td>0.17</td><td>0.08</td><td>0.06</td></tr><tr><td>because</td><td>0.32</td><td>38.95</td><td>42.76</td><td>53.55</td><td>38.80</td><td>0.15</td><td>0.24</td><td>0.12</td></tr><tr><td>but</td><td>0.31</td><td>44.79</td><td>50.00</td><td>63.22</td><td><strong>5564.67</strong></td><td>0.20</td><td>0.15</td><td>0.09</td></tr><tr><td>if</td><td>0.25</td><td>47.90</td><td>49.16</td><td>56.45</td><td>47.71</td><td>0.21</td><td>0.09</td><td>0.07</td></tr><tr><td>of</td><td>0.25</td><td>44.95</td><td>51.17</td><td>56.85</td><td>45.99</td><td>0.20</td><td>0.07</td><td>0.07</td></tr><tr><td>or</td><td>0.22</td><td>44.45</td><td>48.68</td><td>50.86</td><td>44.59</td><td>0.18</td><td>0.08</td><td>0.07</td></tr><tr><td>that</td><td>0.26</td><td>41.08</td><td>44.46</td><td>56.21</td><td><strong>21359.14</strong></td><td>0.18</td><td>0.13</td><td>0.10</td></tr><tr><td>when</td><td>113.83</td><td>35.28</td><td>44.65</td><td>63.01</td><td><strong>18228.19</strong></td><td>0.17</td><td>0.21</td><td>0.14</td></tr><tr><td>where</td><td>0.39</td><td>48.09</td><td>57.04</td><td>68.93</td><td>47.88</td><td>0.19</td><td>0.25</td><td>0.16</td></tr></tbody></table>
<table><caption>Table 7: Percentage error for each part-of-speech class and each eye tracking feature.</caption><thead><tr><th rowspan="2">Feature</th><th colspan="2">FUNC</th><th colspan="2">ADJ</th><th colspan="2">ADV</th><th colspan="2">NOUN</th><th colspan="2">PROPN</th><th colspan="2">VERB</th></tr><tr><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th><th>BERT</th><th>XLM</th></tr></thead><tbody><tr><td>NFIX</td><td>68.54</td><td>66.10</td><td>4.69</td><td>5.44</td><td>4.63</td><td>4.82</td><td>9.38</td><td>15.76</td><td>6.38</td><td>8.26</td><td>6.38</td><td>7.88</td></tr><tr><td>MFD</td><td>68.86</td><td>62.16</td><td>5.32</td><td>6.32</td><td>4.57</td><td>5.94</td><td>9.63</td><td>17.39</td><td>4.69</td><td>7.50</td><td>6.94</td><td>8.19</td></tr><tr><td>FPROP</td><td>72.17</td><td>69.17</td><td>4.57</td><td>4.75</td><td>4.50</td><td>4.44</td><td>8.13</td><td>14.82</td><td>4.38</td><td>6.82</td><td>6.25</td><td>6.82</td></tr><tr><td>FFD</td><td>67.98</td><td>62.48</td><td>5.25</td><td>6.25</td><td>4.63</td><td>5.88</td><td>10.44</td><td>16.89</td><td>4.75</td><td>7.13</td><td>6.94</td><td>8.51</td></tr><tr><td>FPD</td><td>68.36</td><td>58.91</td><td>5.25</td><td>7.32</td><td>4.44</td><td>6.19</td><td>9.82</td><td>18.89</td><td>5.19</td><td>8.51</td><td>6.94</td><td>8.69</td></tr><tr><td>TRT</td><td>68.36</td><td>66.60</td><td>16.60</td><td>6.57</td><td>14.03</td><td>5.19</td><td>31.03</td><td>13.07</td><td>16.40</td><td>6.94</td><td>21.94</td><td>8.57</td></tr><tr><td>NREFIX</td><td>75.88</td><td>78.70</td><td>3.71</td><td>3.71</td><td>3.61</td><td>3.61</td><td>7.46</td><td>7.46</td><td>2.82</td><td>2.82</td><td>6.52</td><td>6.52</td></tr><tr><td>REPROP</td><td>75.88</td><td>78.70</td><td>3.71</td><td>3.71</td><td>3.61</td><td>3.61</td><td>7.46</td><td>7.46</td><td>2.82</td><td>2.82</td><td>6.52</td><td>6.52</td></tr></tbody></table>

## A.5 Standard Deviations of Function Words

Table 8 shows the standard deviations of the predictions of the most frequent function word classes, namely prepositions (PREP), determiners (DET), and conjunctions (CONJ).

<table><caption>Table 8: Standard deviation of the most frequent function words.</caption><thead><tr><th>Lemma</th><th>NFIX</th><th>FFD</th><th>FPD</th><th>TRT</th><th>MFD</th><th>FPROP</th><th>NREFIX</th><th>REPROP</th></tr></thead><tbody><tr><td rowspan="10">PREP</td><td>at</td><td>0.20</td><td>33.87</td><td>35.96</td><td>42.79</td><td><strong>4562.23</strong></td><td>0.15</td><td>10.10</td><td>10.10</td></tr><tr><td>by</td><td>0.25</td><td>43.47</td><td>44.57</td><td>51.42</td><td><strong>5109.68</strong></td><td>0.19</td><td>0.09</td><td>0.07</td></tr><tr><td>for</td><td>0.26</td><td><strong>8365.60</strong></td><td>42.11</td><td><strong>14199.50</strong></td><td><strong>6347.25</strong></td><td>0.19</td><td>0.13</td><td>27.87</td></tr><tr><td>from</td><td>0.25</td><td>39.08</td><td>40.07</td><td><strong>12828.73</strong></td><td>38.53</td><td>0.18</td><td>0.11</td><td>35.43</td></tr><tr><td>in</td><td>23.15</td><td><strong>1859.10</strong></td><td><strong>1859.05</strong></td><td>41.57</td><td><strong>2782.60</strong></td><td>23.15</td><td>0.07</td><td>0.06</td></tr><tr><td>of</td><td>12.92</td><td><strong>4158.18</strong></td><td><strong>4158.11</strong></td><td><strong>3839.08</strong></td><td><strong>5614.73</strong></td><td>12.92</td><td>4.31</td><td>4.31</td></tr><tr><td>on</td><td>41.64</td><td>35.31</td><td>36.95</td><td><strong>8120.68</strong></td><td><strong>6406.01</strong></td><td>0.16</td><td>8.33</td><td>8.33</td></tr><tr><td>to</td><td>7.07</td><td>35.79</td><td>37.38</td><td><strong>2073.26</strong></td><td><strong>3562.01</strong></td><td>7.07</td><td>0.08</td><td>0.07</td></tr><tr><td>up</td><td>0.23</td><td>46.17</td><td>48.59</td><td>52.53</td><td>46.42</td><td>0.20</td><td>0.07</td><td>0.06</td></tr><tr><td>with</td><td>0.31</td><td>43.47</td><td>47.71</td><td>62.36</td><td><strong>18589.94</strong></td><td>0.18</td><td>0.19</td><td>0.13</td></tr><tr><td rowspan="10">DET</td><td>a</td><td>12.48</td><td><strong>1168.31</strong></td><td><strong>1168.30</strong></td><td><strong>1168.27</strong></td><td><strong>6726.54</strong></td><td>17.64</td><td>0.07</td><td>0.05</td></tr><tr><td>all</td><td>0.24</td><td>43.44</td><td>47.06</td><td>57.88</td><td><strong>11703.01</strong></td><td>0.18</td><td>0.12</td><td>0.08</td></tr><tr><td>another</td><td>0.44</td><td>56.89</td><td>70.55</td><td>98.66</td><td>54.99</td><td>0.23</td><td>0.29</td><td>0.18</td></tr><tr><td>any</td><td>0.23</td><td>40.18</td><td>42.81</td><td>50.25</td><td>39.31</td><td>0.17</td><td>0.12</td><td>0.11</td></tr><tr><td>every</td><td>0.33</td><td>40.36</td><td>46.56</td><td>48.93</td><td><strong>36800.71</strong></td><td>0.15</td><td>0.25</td><td>0.14</td></tr><tr><td>no</td><td>0.25</td><td>39.11</td><td>40.87</td><td>50.50</td><td>38.53</td><td>0.18</td><td>0.10</td><td>0.08</td></tr><tr><td>some</td><td>0.31</td><td>46.57</td><td>49.31</td><td>60.57</td><td><strong>26367.42</strong></td><td>0.18</td><td>0.20</td><td>0.11</td></tr><tr><td>that</td><td>0.26</td><td>41.07</td><td>45.10</td><td>53.25</td><td>40.90</td><td>0.18</td><td>0.15</td><td>0.11</td></tr><tr><td>the</td><td>9.06</td><td><strong>2060.29</strong></td><td>919.18</td><td><strong>2349.22</strong></td><td><strong>5173.09</strong></td><td>9.06</td><td>0.11</td><td>0.08</td></tr><tr><td>this</td><td>0.32</td><td>47.54</td><td>53.64</td><td>67.83</td><td><strong>26854.66</strong></td><td>0.19</td><td>0.18</td><td>0.12</td></tr><tr><td rowspan="10">CONJ</td><td>and</td><td>0.27</td><td>37.11</td><td>41.45</td><td>48.52</td><td><strong>5318.58</strong></td><td>32.99</td><td>0.11</td><td>0.08</td></tr><tr><td>as</td><td>0.22</td><td>38.20</td><td>39.10</td><td>44.08</td><td><strong>10602.03</strong></td><td>0.17</td><td>0.08</td><td>0.06</td></tr><tr><td>because</td><td>0.32</td><td>38.95</td><td>42.76</td><td>53.55</td><td>38.80</td><td>0.15</td><td>0.24</td><td>0.12</td></tr><tr><td>but</td><td>0.31</td><td>44.79</td><td>50.00</td><td>63.22</td><td><strong>5564.67</strong></td><td>0.20</td><td>0.15</td><td>0.09</td></tr><tr><td>if</td><td>0.25</td><td>47.90</td><td>49.16</td><td>56.45</td><td>47.71</td><td>0.21</td><td>0.09</td><td>0.07</td></tr><tr><td>of</td><td>0.25</td><td>44.95</td><td>51.17</td><td>56.85</td><td>45.99</td><td>0.20</td><td>0.07</td><td>0.07</td></tr><tr><td>or</td><td>0.22</td><td>44.45</td><td>48.68</td><td>50.86</td><td>44.59</td><td>0.18</td><td>0.08</td><td>0.07</td></tr><tr><td>that</td><td>0.26</td><td>41.08</td><td>44.46</td><td>56.21</td><td><strong>21359.14</strong></td><td>0.18</td><td>0.13</td><td>0.10</td></tr><tr><td>when</td><td>113.83</td><td>35.28</td><td>44.65</td><td>63.01</td><td><strong>18228.19</strong></td><td>0.17</td><td>0.21</td><td>0.14</td></tr><tr><td>where</td><td>0.39</td><td>48.09</td><td>57.04</td><td>68.93</td><td>47.88</td><td>0.19</td><td>0.25</td><td>0.16</td></tr></tbody></table>
<figure><img src="image_8.png" /><figcaption>Figure 5: Standard deviations of the eye tracking features of the most frequent English **determiners**. Note that the y-axis scales change in each subplot.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6: Standard deviations of the eye tracking features of the most frequent English **conjunctions**. Note that the y-axis scales change in each subplot.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 5: Standard deviations of the eye tracking features of the most frequent English **determiners**. Note that the y-axis scales change in each subplot.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6: Standard deviations of the eye tracking features of the most frequent English **conjunctions**. Note that the y-axis scales change in each subplot.</figcaption></figure>
(In)Alienable Possession in Mandarin Relative Clauses

Deran Kong and Yu-Yin Hsu

The Hong Kong Polytechnic University, Hung Hom, Hong Kong

yu-yin.hsu@polyu.edu.hk

**Abstract**

Inalienable possession differs from alienable possession in that, in the former – e.g., kinships and part-whole relations – there is an intrinsic semantic dependency between the possessor and possessum. This paper reports two studies that used acceptability-judgment tasks to investigate whether native Mandarin speakers experienced different levels of interpretational costs while resolving different types of possessive relations, i.e., inalienable possessions (kinship terms and body parts) and alienable ones, expressed within relative clauses. The results show that sentences received higher acceptability ratings when body parts were the possessum as compared to sentences with alienable possessum, indicating that the inherent semantic dependency facilitates the resolution. However, inalienable kinship terms received the lowest acceptability ratings. We argue that this was because the kinship terms, which had the [+human] feature and appeared at the beginning of the experimental sentences, tended to be interpreted as the subject in shallow processing; these features contradicted the semantic-syntactic requirements of the experimental sentences.

# 1 Introduction

Possessive relations are fundamental in human languages because they associate nouns to express specific relationships. Questions around the alienability between the possessor and the possessum in possessive relations has garnered considerable attention in linguistics (Vergnaud and Zubizarreta, 1992). Such alienability can be categorized in several ways. Semantically, when a possessive relation is inalienable, there is an inherent dependency between the possessor and possessum, which does not exist in the case of alienable possessions (Vergnaud and Zubizarreta, 1992). Typical examples of inalienably possessed nouns include kinship terms and body parts; for example, *Mary's*

brother and *Mary's hand*, respectively. The family relation and whole-part relation are often considered intrinsic and cannot be transferred. In contrast, an alienable possession, such as *Mary's desk*, does not present such an inherent semantic dependency between the two nouns, and this type of possessive relation needs to be acquired, and can be transferred (Seiler, 1983). Several languages have distinct morphological markings of alienable and inalienable possessions (Dixon, 2000; Meyerhoff, 2002; Gebregziabher, 2012), and various syntactic structures have been proposed for them (Alexiadou, 2003). While Mandarin does not make such distinctions through morphology, some syntactic and pragmatic distinctions between these two classes of possessions exist in it (Hsu and Ting, 2006).

Normally, in Mandarin, the possessor precedes the possessed noun, and they are linearly close to each other; for example, in *laoban de gebo* ('boss's arm'), *laoban* ('boss') is the possessor, and the next noun to appear, *gebo* ('arm'), is the possessum. However, in Mandarin possessive relative clauses (PRCs), as shown in (1), the possessum and possessor are *not* adjacent. PRCs therefore provide us with a useful opportunity to examine the processing cost of resolving different types of possessions, because readers must resolve such a long-distance association to arrive at the possessive relationship between the two nouns.

(1) Fangzi/Fuqin/Gebo bei daitu house/father/arm PASS criminal jizhong de laoban hen shengqi. shot DE boss very angry 'The boss whose house/father/arm was shot by the criminal was angry.'

Moreover, the possessor-possessum position is reversed in (1); i.e., the possessum appears before the possessor. It is expected that when readers encounter a kinship term or a body part, they may
(In)Alienable Possession in Mandarin Relative Clauses

Deran Kong and Yu-Yin Hsu

The Hong Kong Polytechnic University, Hung Hom, Hong Kong

yu-yin.hsu@polyu.edu.hk

**Abstract**

Inalienable possession differs from alienable possession in that, in the former – e.g., kinships and part-whole relations – there is an intrinsic semantic dependency between the possessor and possessum. This paper reports two studies that used acceptability-judgment tasks to investigate whether native Mandarin speakers experienced different levels of interpretational costs while resolving different types of possessive relations, i.e., inalienable possessions (kinship terms and body parts) and alienable ones, expressed within relative clauses. The results show that sentences received higher acceptability ratings when body parts were the possessum as compared to sentences with alienable possessum, indicating that the inherent semantic dependency facilitates the resolution. However, inalienable kinship terms received the lowest acceptability ratings. We argue that this was because the kinship terms, which had the [+human] feature and appeared at the beginning of the experimental sentences, tended to be interpreted as the subject in shallow processing; these features contradicted the semantic-syntactic requirements of the experimental sentences.

# 1 Introduction

Possessive relations are fundamental in human languages because they associate nouns to express specific relationships. Questions around the alienability between the possessor and the possessum in possessive relations has garnered considerable attention in linguistics (Vergnaud and Zubizarreta, 1992). Such alienability can be categorized in several ways. Semantically, when a possessive relation is inalienable, there is an inherent dependency between the possessor and possessum, which does not exist in the case of alienable possessions (Vergnaud and Zubizarreta, 1992). Typical examples of inalienably possessed nouns include kinship terms and body parts; for example, *Mary's*

brother and *Mary's hand*, respectively. The family relation and whole-part relation are often considered intrinsic and cannot be transferred. In contrast, an alienable possession, such as *Mary's desk*, does not present such an inherent semantic dependency between the two nouns, and this type of possessive relation needs to be acquired, and can be transferred (Seiler, 1983). Several languages have distinct morphological markings of alienable and inalienable possessions (Dixon, 2000; Meyerhoff, 2002; Gebregziabher, 2012), and various syntactic structures have been proposed for them (Alexiadou, 2003). While Mandarin does not make such distinctions through morphology, some syntactic and pragmatic distinctions between these two classes of possessions exist in it (Hsu and Ting, 2006).

Normally, in Mandarin, the possessor precedes the possessed noun, and they are linearly close to each other; for example, in *laoban de gebo* ('boss's arm'), *laoban* ('boss') is the possessor, and the next noun to appear, *gebo* ('arm'), is the possessum. However, in Mandarin possessive relative clauses (PRCs), as shown in (1), the possessum and possessor are *not* adjacent. PRCs therefore provide us with a useful opportunity to examine the processing cost of resolving different types of possessions, because readers must resolve such a long-distance association to arrive at the possessive relationship between the two nouns.

(1) Fangzi/Fuqin/Gebo bei daitu house/father/arm PASS criminal jizhong de laoban hen shengqi. shot DE boss very angry 'The boss whose house/father/arm was shot by the criminal was angry.'

Moreover, the possessor-possessum position is reversed in (1); i.e., the possessum appears before the possessor. It is expected that when readers encounter a kinship term or a body part, they may
expect a possessor later in the sentence, and there-
fore, the inherent semantic dependency should fa-
cilitate the resolution of inalienable possessions.
In contrast, an alienable possesseum may not enjoy
this facilitation, because the association between
the possessor and the alienable possesseum is not
inherently salient, and this may result in a greater
processing load when resolving the possessive re-
lation becomes necessary in a later phase of sen-
tence comprehension.

Following a review on previous studies about
possessive relations and possessive structures in
Mandarin (Section 2), we present two experiments
(Section 3) in which an acceptability-judgment
task (AJT) is used to measure the acceptability
of Mandarin PRCs with three different types of
possessums: alienable possessums, kinship terms,
and body parts. Our results partially support the
hypothesis that the inherent semantic dependency
of the inalienable possessions facilitate the inte-
gration of the possessor-possessum relations, re-
flected by the higher acceptability of the body
parts as the possessum than the alienable noun
as the possessum. However, the other inalien-
able condition, kinship terms as the possessum, re-
ceived unexpected low ratings. Section 4 briefly
concludes this paper, and includes some possible
explanations of its unexpected findings.

2 **Alienable and Inalienable Possessions**
**in Mandarin**

While Mandarin does not require overt morpho-
logical markings on (in)alienability, contrasts be-
tween alienable and inalienable possessions are re-
flected at the syntactic, semantic, and discourse
levels (Landau, 1999; Hsu and Ting, 2006). In
this section, three examples are used to illustrate
some of such differences: *de* omission, semantics
in the *ba*-construction, and discourse-contextual
demands.

Mandarin uses a particle, *de*, to link two nouns
to form certain semantic associations, and posses-
sive relation is one of them (Li and Thompson,
1989; Hsu, 2009; Li, 2012). Example (2) is an
alienable possessive phrase, and (3) is an inalien-
able one. When two nouns express a family rela-
tion and the possessor is a pronoun, the particle
*de* can be omitted, e.g., (2). However, omitting
the particle *de* may be ungrammatical if the rela-
tion is alienable and when the phrase is presented
in isolation, as shown in (3). While kinship rela-

tions seem to consistently allow the *de* omission
and to enjoy a special status in possessive phrases
in the literature, the above-mentioned contrast be-
tween (2) and (3) does not always hold. For exam-
ple, when the inalienable possession is evaluated
within context, as shown in example (4), the omis-
sion vs. non-omission of *de* does not influence its
acceptability.

(2) wo (de) baba
I DE father
‘my father’

(3) wo *(de) zhuozi
I DE desk
‘my desk’

(4) Wo (de) xuexiao zhengzai juxing
I DE school currently hold
yanjiang bisai.
speech contest
‘My school is holding a speech contest.’

The term alienability implies a property or entity
that can be conveyed from one individual to an-
other. Putting alienable and inalienable posses-
sions in the *ba*-construction, which often involves
a meaning related to disposal, highlights these dif-
ferences around transferring possession, as well as
how such differences of (in)alienability affect sen-
tence acceptability. In (5), the *wallet* originally be-
longed to *Zhangsan*, establishing an alienable pos-
sessive relation, and after a giving event, the pos-
session of the *wallet* is transferred to *Lisi*. In con-
trast, inalienable possessive phrases exhibit more
resistance to transferring possession via the *ba*-
construction. For example, sentence (6) is gram-
matical, only in specific circumstances whereby
*Zhangsan’s* sending of his brother to *Lisi* makes
sense. Importantly, in this case, the kinship is
not actually transferred; that is, the brother is still
*Zhangsan’s*. Body parts, on the other hand, can-
not be accepted under normal conditions, leading
to the unacceptability of (7). <fnref n="1" />

(5) Zhangsan ba qianbao songgeile Lisi.
Zhangsan BA wallet give-ASP Lisi
‘Zhangsan has given (his) wallet to Lisi (as a
gift).’

<footnote n="1">¹"?" indicates that a sentence sounds odd, and "*" indi-
cates that a sentence is ungrammatical.</footnote>
expect a possessor later in the sentence, and there-
fore, the inherent semantic dependency should fa-
cilitate the resolution of inalienable possessions.
In contrast, an alienable possesseum may not enjoy
this facilitation, because the association between
the possessor and the alienable possesseum is not
inherently salient, and this may result in a greater
processing load when resolving the possessive re-
lation becomes necessary in a later phase of sen-
tence comprehension.

Following a review on previous studies about
possessive relations and possessive structures in
Mandarin (Section 2), we present two experiments
(Section 3) in which an acceptability-judgment
task (AJT) is used to measure the acceptability
of Mandarin PRCs with three different types of
possessums: alienable possessums, kinship terms,
and body parts. Our results partially support the
hypothesis that the inherent semantic dependency
of the inalienable possessions facilitate the inte-
gration of the possessor-possessum relations, re-
flected by the higher acceptability of the body
parts as the possessum than the alienable noun
as the possessum. However, the other inalien-
able condition, kinship terms as the possessum, re-
ceived unexpected low ratings. Section 4 briefly
concludes this paper, and includes some possible
explanations of its unexpected findings.

## 2 Alienable and Inalienable Possessions in Mandarin

While Mandarin does not require overt morpho-
logical markings on (in)alienability, contrasts be-
tween alienable and inalienable possessions are re-
flected at the syntactic, semantic, and discourse
levels (Landau, 1999; Hsu and Ting, 2006). In
this section, three examples are used to illustrate
some of such differences: *de* omission, semantics
in the *ba*-construction, and discourse-contextual
demands.

Mandarin uses a particle, *de*, to link two nouns
to form certain semantic associations, and posses-
sive relation is one of them (Li and Thompson,
1989; Hsu, 2009; Li, 2012). Example (2) is an
alienable possessive phrase, and (3) is an inalien-
able one. When two nouns express a family rela-
tion and the possessor is a pronoun, the particle
*de* can be omitted, e.g., (2). However, omitting
the particle *de* may be ungrammatical if the rela-
tion is alienable and when the phrase is presented
in isolation, as shown in (3). While kinship rela-

tions seem to consistently allow the *de* omission
and to enjoy a special status in possessive phrases
in the literature, the above-mentioned contrast be-
tween (2) and (3) does not always hold. For exam-
ple, when the inalienable possession is evaluated
within context, as shown in example (4), the omis-
sion vs. non-omission of *de* does not influence its
acceptability.

(2) wo (de) baba
I DE father
‘my father’

(3) wo *(de) zhuozi
I DE desk
‘my desk’

(4) Wo (de) xuexiao zhengzai juxing
I DE school currently hold
yanjiang bisai.
speech contest
‘My school is holding a speech contest.’

The term alienability implies a property or entity
that can be conveyed from one individual to an-
other. Putting alienable and inalienable posses-
sions in the *ba*-construction, which often involves
a meaning related to disposal, highlights these dif-
ferences around transferring possession, as well as
how such differences of (in)alienability affect sen-
tence acceptability. In (5), the *wallet* originally be-
longed to *Zhangsan*, establishing an alienable pos-
sessive relation, and after a giving event, the pos-
session of the *wallet* is transferred to *Lisi*. In con-
trast, inalienable possessive phrases exhibit more
resistance to transferring possession via the *ba*-
construction. For example, sentence (6) is gram-
matical, only in specific circumstances whereby
*Zhangsan’s* sending of his brother to *Lisi* makes
sense. Importantly, in this case, the kinship is
not actually transferred; that is, the brother is still
*Zhangsan’s*. Body parts, on the other hand, can-
not be accepted under normal conditions, leading
to the unacceptability of (7). <fnref n="1" />

(5) Zhangsan ba qianbao songgeile Lisi.
Zhangsan BA wallet give-ASP Lisi
‘Zhangsan has given (his) wallet to Lisi (as a
gift).’

<footnote n="1">¹"?" indicates that a sentence sounds odd, and "*" indi-
cates that a sentence is ungrammatical.</footnote>
(6) Zhangsan ba didi songgeile Lisi.
Zhangsan BA brother give-ASP Lisi
‘Zhangsan has given (his) brother to Lisi.’

(7) \*Zhangsan ba gebo songgeile Lisi.
Zhangsan BA arm give-ASP Lisi
‘Zhangsan has given (his) arm to Lisi.’

The effect of (in)alienability on sentence acceptability can also be observed at the discourse level. The semantic distinction between inalienable and alienable possessive nouns is that the former implies a specific possessor in the interlocutors' common ground, whereas the latter does not necessarily trigger such a possessive association, and can stand alone. For example, (8) is acceptable even though it is not specified whose plant it is. In contrast, (9) and (10) are less acceptable if the possessors are not indicated in the discourse.<fnref n="2" />

(8) Zhiwu bei taiyang shaisile。
plant PASS sun burn-die-ASP
‘The plant was killed by the sun.’

(9) ?Sunnv bei taiyang
granddaughter PASS sun
shaishangle。
burn-hurt-ASP
‘(Someone’s) granddaughter was sunburnt.’

(10) ?Gebo bei taiyang shaishangle。
arm PASS sun burn-hurt-ASP
‘(Someone’s) arm was sun-burnt.’

To summarize, inalienable and alienable possessions trigger different syntactic, semantic, and discourse requirements and these can affect the acceptability of sentences. Nouns related to kinship terms and body parts often trigger a possessive dependency and increase contextual demands, whereas typical nouns allowing alienable possessions behave differently. The following section presents two AJT experiments using Mandarin PRCs to test the effects of (in)alienability on sentences' acceptability.

# 3 Experiments

## 3.1 Experiment 1

### 3.1.1 Participants and Procedure

Our participants in Experiment 1 were Mainland Chinese college students studying in Hong Kong, all of whom were native speakers of Mandarin. Their average self-rated Mandarin proficiency on a scale of 1 to 7 (with 1 = 'not fluent at all', and 7 = 'extremely fluent') was 6.75. Attention filters were included in the experiment to ensure that the participants understood the task and finished it conscientiously. Of the initial pool of 114 participants, 17 failed the attention filters, leaving 97 participants' data for analysis.

The participants rated the acceptability of the sentences on a 5-point Likert scale, ranging from 1 as 'totally unacceptable' to 5 as 'totally acceptable'. The data were collected using an online questionnaire platform.

### 3.1.2 Materials

Mandarin PRCs were adopted as the basis for our investigation of the cost of the possessor-possessum integration among three different possessive conditions. In the PRCs used in Experiment 1, the possessum was at the beginning of the realtive clause, and the possessor – immediately following the relativizer particle *de* – was the head noun of the relative clause, locating at the end of the relative clause. The possessor also served as the matrix subject of the experimental sentence. Because the possessor and the possessum were not adjacent, the participants reading these sentences needed to form long-distance associations. Example (11) consists of one set of the experimental conditions, in which Condition A is the alienable condition, and Conditions B and C are inalienable ones, for kinship terms and body parts, respectively. In addition, a short context sentence was provided before each of the PRC target sentences, to help them read more naturally. In all, Twenty-four sets were constructed and were distributed into four lists <fnref n="3" />; each participant only read one condition from each set.

<footnote n="2">²An anonymous reviewer pointed out that if sentence (9) is used in a conversation, it sounds acceptable because a rational/cooperative listener will assume the speaker to be the possessor. We agree with this observation, and with the same reviewer's observation that in future research, it would therefore be worthwhile to design dialogues with multiple agents expressing sentences like (8) to (10), and use them to investigate people's understandings of them. Here, we originally intended to argue that sentences like (9) and (10) are less acceptable when no possessor can be associated with the subject.</footnote>

<footnote n="3">³Because filler items from other studies were included, there were four lists instead of three.</footnote>
(6) Zhangsan ba didi songgeile Lisi.
Zhangsan BA brother give-ASP Lisi
‘Zhangsan has given (his) brother to Lisi.’

(7) \*Zhangsan ba gebo songgeile Lisi.
Zhangsan BA arm give-ASP Lisi
‘Zhangsan has given (his) arm to Lisi.’

The effect of (in)alienability on sentence acceptability can also be observed at the discourse level. The semantic distinction between inalienable and alienable possessive nouns is that the former implies a specific possessor in the interlocutors’ common ground, whereas the latter does not necessarily trigger such a possessive association, and can stand alone. For example, (8) is acceptable even though it is not specified whose plant it is. In contrast, (9) and (10) are less acceptable if the possessors are not indicated in the discourse.<fnref n="2" />

(8) Zhiwu bei taiyang shaisile。
plant PASS sun burn-die-ASP
‘The plant was killed by the sun.’

(9) ?Sunnv bei taiyang
granddaughter PASS sun
shaishangle。
burn-hurt-ASP
‘(Someone’s) granddaughter was sunburnt.’

(10) ?Gebo bei taiyang shaishangle。
arm PASS sun burn-hurt-ASP
‘(Someone’s) arm was sun-burnt.’

To summarize, inalienable and alienable possessions trigger different syntactic, semantic, and discourse requirements and these can affect the acceptability of sentences. Nouns related to kinship terms and body parts often trigger a possessive dependency and increase contextual demands, whereas typical nouns allowing alienable possessions behave differently. The following section presents two AJT experiments using Mandarin PRCs to test the effects of (in)alienability on sentences’ acceptability.

# 3 Experiments

## 3.1 Experiment 1

### 3.1.1 Participants and Procedure

Our participants in Experiment 1 were Mainland Chinese college students studying in Hong Kong, all of whom were native speakers of Mandarin. Their average self-rated Mandarin proficiency on a scale of 1 to 7 (with 1 = ‘not fluent at all’, and 7 = ‘extremely fluent’) was 6.75. Attention filters were included in the experiment to ensure that the participants understood the task and finished it conscientiously. Of the initial pool of 114 participants, 17 failed the attention filters, leaving 97 participants’ data for analysis.

The participants rated the acceptability of the sentences on a 5-point Likert scale, ranging from 1 as ‘totally unacceptable’ to 5 as ‘totally acceptable’. The data were collected using an online questionnaire platform.

### 3.1.2 Materials

Mandarin PRCs were adopted as the basis for our investigation of the cost of the possessor-possessum integration among three different possessive conditions. In the PRCs used in Experiment 1, the possessum was at the beginning of the realtive clause, and the possessor – immediately following the relativizer particle *de* – was the head noun of the relative clause, locating at the end of the relative clause. The possessor also served as the matrix subject of the experimental sentence. Because the possessor and the possessum were not adjacent, the participants reading these sentences needed to form long-distance associations. Example (11) consists of one set of the experimental conditions, in which Condition A is the alienable condition, and Conditions B and C are inalienable ones, for kinship terms and body parts, respectively. In addition, a short context sentence was provided before each of the PRC target sentences, to help them read more naturally. In all, Twenty-four sets were constructed and were distributed into four lists <fnref n="3" />; each participant only read one condition from each set.

<footnote n="2">²An anonymous reviewer pointed out that if sentence (9) is used in a conversation, it sounds acceptable because a rational/cooperative listener will assume the speaker to be the possessor. We agree with this observation, and with the same reviewer’s observation that in future research, it would therefore be worthwhile to design dialogues with multiple agents expressing sentences like (8) to (10), and use them to investigate people’s understandings of them. Here, we originally intended to argue that sentences like (9) and (10) are less acceptable when no possessor can be associated with the subject.</footnote>

<footnote n="3">³Because filler items from other studies were included, there were four lists instead of three.</footnote>
(11) (Context) In this extremely hot weather,

a. Zhiwu bei taiyang shaisi de plant PASS sun burn-die DE A-Yong gandao shifen shangxin. A-Yong feel very sad 'A-Yong, whose plant was killed by the sun, felt very sad.'

b. Sunnv bei taiyang granddaughter PASS sun shaishang de A-Yong gandao burn-hurt DE A-Yong feel shifen shangxin. very sad 'A-Yong, whose granddaughter was sunburnt, felt very sad.'

c. Gebo bei taiyang shaishang de arm PASS sun burn-hurt DE A-Yong gandao shifen shangxin. A-Yong feel very sad 'A-Yong, whose arms were sunburnt, felt very sad.'

It has been argued that sentences with higher complexity are likely be harder to process, and consequently, readers tend to rate their acceptability as lower (Chomsky and Miller, 1968; Fanselow and Frisch, 2006). Our items were designed to be completely uniform in their structural complexity. Yet, as compared to inalienable possessions, alienable ones may be more costly to integrate, as doing so calls for the possessor to acquire the possessor-possessum relations (Alexiadou, 2003). Therefore, we predicted that the inalienable conditions (Conditions B and C) would receive higher AJT ratings than the alienable one (Condition A).

### 3.1.3 Results

The mean rating and standard deviation (SD) for each condition are listed in Table 1. Condition C, in which the body parts were the possessed nouns, received the highest mean acceptability rating. Condition A, the alienable condition, was rated as less acceptably on average than Condition C was. But surprisingly, the other inalienable condition, Condition B, received the lowest acceptability rating. The results of the three conditions are visualized in Figure 1, in which the box represents 50% of the central data, and the line inside it representing the median. The whiskers are the range of the data excluding outliers, which are indicated by the small black dots. The three large

<table><caption>Table 1: Average ratings and standard deviations for each condition.</caption><thead><tr><td>Condition</td><td>Mean</td><td>SD</td></tr></thead><tbody><tr><td>A(alienable nouns)</td><td>4.07</td><td>1.22</td></tr><tr><td>B(kinship terms)</td><td>3.36</td><td>1.44</td></tr><tr><td>C(body parts)</td><td>4.43</td><td>0.87</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>Figure 1: Ratings for different conditions, Experiment 1.</figcaption></figure>

black dots are the average ratings of each condition. The colored dots are the average ratings by item. The lines connecting the colored dots indicate that items are from the same sentence set.

To test whether the differences among the conditions were significant, a cumulative link mixed model was fit using the clmm() function in the ordinal package in R. The outcome variable was RATING, and the predictor was CONDITION, which had three levels: Condition A, Condition B, and Condition C. The random variables were PARTICIPANT and ITEM, including varying intercepts. A pairwise post hoc analysis showed that the differences among the conditions were significant (Table 2).

To summarize, the inalienable condition in which body parts were the possessed nouns being rated highest, and was significantly higher than the alienable condition.

However, the kinship-terms condition's rating

<table><caption>Table 2: Contrasts among conditions; Model: Rating Condition + (1|Participant) + (1|Item)</caption><thead><tr><td>Contrast</td><td>Estimate</td><td>SE</td><td>z.ratio</td><td>p.value</td></tr></thead><tbody><tr><td>A - B</td><td>-1.397</td><td>0.125</td><td>-11.166</td><td>&lt;.0001</td></tr><tr><td>A - C</td><td>0.837</td><td>0.133</td><td>6.297</td><td>&lt;.0001</td></tr><tr><td>B - C</td><td>2.235</td><td>0.136</td><td>16.395</td><td>&lt;.0001</td></tr></tbody></table>
(11) (Context) In this extremely hot weather,

a. Zhiwu bei taiyang shaisi de plant PASS sun burn-die DE A-Yong gandao shifen shangxin. A-Yong feel very sad 'A-Yong, whose plant was killed by the sun, felt very sad.'

b. Sunnv bei taiyang granddaughter PASS sun shaishang de A-Yong gandao burn-hurt DE A-Yong feel shifen shangxin. very sad 'A-Yong, whose granddaughter was sunburnt, felt very sad.'

c. Gebo bei taiyang shaishang de arm PASS sun burn-hurt DE A-Yong gandao shifen shangxin. A-Yong feel very sad 'A-Yong, whose arms were sunburnt, felt very sad.'

It has been argued that sentences with higher complexity are likely be harder to process, and consequently, readers tend to rate their acceptability as lower (Chomsky and Miller, 1968; Fanselow and Frisch, 2006). Our items were designed to be completely uniform in their structural complexity. Yet, as compared to inalienable possessions, alienable ones may be more costly to integrate, as doing so calls for the possessor to acquire the possessor-possessum relations (Alexiadou, 2003). Therefore, we predicted that the inalienable conditions (Conditions B and C) would receive higher AJT ratings than the alienable one (Condition A).

### 3.1.3 Results

The mean rating and standard deviation (SD) for each condition are listed in Table 1. Condition C, in which the body parts were the possessed nouns, received the highest mean acceptability rating. Condition A, the alienable condition, was rated as less acceptably on average than Condition C was. But surprisingly, the other inalienable condition, Condition B, received the lowest acceptability rating. The results of the three conditions are visualized in Figure 1, in which the box represents 50% of the central data, and the line inside it representing the median. The whiskers are the range of the data excluding outliers, which are indicated by the small black dots. The three large

<table><caption>Table 1: Average ratings and standard deviations for each condition.</caption><thead><tr><td>Condition</td><td>Mean</td><td>SD</td></tr></thead><tbody><tr><td>A(alienable nouns)</td><td>4.07</td><td>1.22</td></tr><tr><td>B(kinship terms)</td><td>3.36</td><td>1.44</td></tr><tr><td>C(body parts)</td><td>4.43</td><td>0.87</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>Figure 1: Ratings for different conditions, Experiment 1.</figcaption></figure>

black dots are the average ratings of each condition. The colored dots are the average ratings by item. The lines connecting the colored dots indicate that items are from the same sentence set.

To test whether the differences among the conditions were significant, a cumulative link mixed model was fit using the clmm() function in the ordinal package in R. The outcome variable was RATING, and the predictor was CONDITION, which had three levels: Condition A, Condition B, and Condition C. The random variables were PARTICIPANT and ITEM, including varying intercepts. A pairwise post hoc analysis showed that the differences among the conditions were significant (Table 2).

To summarize, the inalienable condition in which body parts were the possessed nouns being rated highest, and was significantly higher than the alienable condition.

However, the kinship-terms condition's rating

<table><caption>Table 2: Contrasts among conditions; Model: Rating Condition + (1|Participant) + (1|Item)</caption><thead><tr><td>Contrast</td><td>Estimate</td><td>SE</td><td>z.ratio</td><td>p.value</td></tr></thead><tbody><tr><td>A - B</td><td>-1.397</td><td>0.125</td><td>-11.166</td><td>&lt;.0001</td></tr><tr><td>A - C</td><td>0.837</td><td>0.133</td><td>6.297</td><td>&lt;.0001</td></tr><tr><td>B - C</td><td>2.235</td><td>0.136</td><td>16.395</td><td>&lt;.0001</td></tr></tbody></table>
was significantly lower than that of the other two conditions. We wonder whether this was because some kinship terms used in the materials, such as, *zhier* ('brother's son'), and *waisun* ('daughter's son'), are not as frequently encountered in day-to-day Madnarin speech as the other kinship terms. Specifically, the occurrence of the kinship terms in the BCC corpus (http://bcc.blcu.edu.cn/; Mean: 34086, Range: 2555-144007) was also markedly lower than those of the alienable nouns (Mean: 584435, Range: 3067-172563), and body parts (Mean: 45939, Range: 10875-232015). Hence, we balanced both the frequencies of words across the three conditions, and then conducted another round of AJT, as reported below.

## 3.2 Experiment 2

### 3.2.1 Participants and Procedure

Sixty-six college students from Mainland China who had not participated in Experiment 1 were recruited for Experiment 2. All self-reported Madnarin as their native language. The procedure of this AJT experiment was the same for Experiment 1, except that, after the participants finished the experiment *per se*, we interviewed some of them who had given low ratings to most of the Condition B items (kinship terms).

### 3.2.2 Materials

Experiment 2's materials were similar to those of Experiment 1, except in the following two respects. First, some of the critical words (i.e., the possessum) were changed to maintain a balance counts of strokes across conditions (Range and Mean for each condition: A: 9-22, 14.63; B: 5-26, 14.54; C: 8-28 16.67) and the frequencies of words (Range and Mean for each condition: A: 3067-172563, 51034; B: 4331-144007, 38814; C: 10152-232015, 61621; frequency is according to the BCC corpus). Second, to lower processing demand, the predicate of each experimental sentence was shortened, from 16 characters (as in Experiment 1), to 14 characters (the separate context sentences were unaffected). For example, *gandao* ('feel') in (11) was dropped, and the sentence remained grammatical.

### 3.2.3 Results

The results of Experiment 2 closely replicated the findings in Experiment 1. That is, body parts as the possessed nouns (Condition C) received the highest rating (Mean = 4.15, SD = 1.03), the alienable

possession (Condition A) was rated lower (Mean = 3.85, SD = 1.13); and the kinship terms as the possessum (Condition B) once more received the lowest rating (Mean = 3.16, SD = 1.27). The clmm() model and pairwise comparisons again showed the differences among conditions were significant (*ps* < .0001).

Importantly, the group variance of the kinship-terms condition was also the largest, which is another repetition of the findings of Experiment 1. To ensure that this unexpected result did not arise because a few items received extremely low ratings while others were acceptable, we looked for systematic differences among the average ratings for each item.<fnref n="4" /> However, this item-by-item analysis revealed no such differences. Indeed, among the 24 kinship terms we tested, 21 received average ratings lower than 3.5, as against overall average ratings for the alienable-nouns and body-parts conditions of 3.85 and 4.15, respectively. Specifically, the majority of kinship terms used as the possessum were rated as 'probably unacceptable' or 'not sure'.

We also conducted individual-level analysis, which revealed that not all participants assigned low ratings to Condition B. That is, a subset of them consistently rejected Condition B, while another tended to find its items acceptable. Therefore, we decided to reexamine the results in terms of the participants' tendency to accept items in each condition. In Experiment 2, all participants read eight sentences from each condition, and we deemed them to have rejected a given condition if they rated at least six out of the eight as '1-totally unacceptable' or '2-probably unacceptable'. Conversely, if a participant rated six out of the eight sentences in a condition as '4-probably acceptable' or '5-totally acceptable', they were counted as accepting that condition. Other cases were classified as 'not sure'. Table 3 summarizes the numbers and percentages of participants who rejected, accepted, or were unsure about each condition. No participants consistently rejected Condition A or Condition C, and indeed, the majority of them consistently accepted these two conditions. But twelve participants consistently rejected Condition B. For that reason, we conducted a post-experiment interviews with these 12 participants. Data from the interviews will be presented and dis-

<footnote n="4">⁴This was done at the suggestion of an anonymous reviewer. We appreciate this advice.</footnote>
was significantly lower than that of the other two conditions. We wonder whether this was because some kinship terms used in the materials, such as, *zhier* ('brother's son'), and *waisun* ('daughter's son'), are not as frequently encountered in day-to-day Madnarin speech as the other kinship terms. Specifically, the occurrence of the kinship terms in the BCC corpus (http://bcc.blcu.edu.cn/; Mean: 34086, Range: 2555-144007) was also markedly lower than those of the alienable nouns (Mean: 584435, Range: 3067-172563), and body parts (Mean: 45939, Range: 10875-232015). Hence, we balanced both the frequencies of words across the three conditions, and then conducted another round of AJT, as reported below.

## 3.2 Experiment 2

### 3.2.1 Participants and Procedure

Sixty-six college students from Mainland China who had not participated in Experiment 1 were recruited for Experiment 2. All self-reported Madnarin as their native language. The procedure of this AJT experiment was the same for Experiment 1, except that, after the participants finished the experiment *per se*, we interviewed some of them who had given low ratings to most of the Condition B items (kinship terms).

### 3.2.2 Materials

Experiment 2's materials were similar to those of Experiment 1, except in the following two respects. First, some of the critical words (i.e., the possessum) were changed to maintain a balance counts of strokes across conditions (Range and Mean for each condition: A: 9-22, 14.63; B: 5-26, 14.54; C: 8-28 16.67) and the frequencies of words (Range and Mean for each condition: A: 3067-172563, 51034; B: 4331-144007, 38814; C: 10152-232015, 61621; frequency is according to the BCC corpus). Second, to lower processing demand, the predicate of each experimental sentence was shortened, from 16 characters (as in Experiment 1), to 14 characters (the separate context sentences were unaffected). For example, *gandao* ('feel') in (11) was dropped, and the sentence remained grammatical.

### 3.2.3 Results

The results of Experiment 2 closely replicated the findings in Experiment 1. That is, body parts as the possessed nouns (Condition C) received the highest rating (Mean = 4.15, SD = 1.03), the alienable

possession (Condition A) was rated lower (Mean = 3.85, SD = 1.13); and the kinship terms as the possessum (Condition B) once more received the lowest rating (Mean = 3.16, SD = 1.27). The clmm() model and pairwise comparisons again showed the differences among conditions were significant (*ps* < .0001).

Importantly, the group variance of the kinship-terms condition was also the largest, which is another repetition of the findings of Experiment 1. To ensure that this unexpected result did not arise because a few items received extremely low ratings while others were acceptable, we looked for systematic differences among the average ratings for each item.<fnref n="4" /> However, this item-by-item analysis revealed no such differences. Indeed, among the 24 kinship terms we tested, 21 received average ratings lower than 3.5, as against overall average ratings for the alienable-nouns and body-parts conditions of 3.85 and 4.15, respectively. Specifically, the majority of kinship terms used as the possessum were rated as ‘probably unacceptable’ or ‘not sure’.

We also conducted individual-level analysis, which revealed that not all participants assigned low ratings to Condition B. That is, a subset of them consistently rejected Condition B, while another tended to find its items acceptable. Therefore, we decided to reexamine the results in terms of the participants’ tendency to accept items in each condition. In Experiment 2, all participants read eight sentences from each condition, and we deemed them to have rejected a given condition if they rated at least six out of the eight as ‘1-totally unacceptable’ or ‘2-probably unacceptable’. Conversely, if a participant rated six out of the eight sentences in a condition as ‘4-probably acceptable’ or ‘5-totally acceptable’, they were counted as accepting that condition. Other cases were classified as ‘not sure’. Table 3 summarizes the numbers and percentages of participants who rejected, accepted, or were unsure about each condition. No participants consistently rejected Condition A or Condition C, and indeed, the majority of them consistently accepted these two conditions. But twelve participants consistently rejected Condition B. For that reason, we conducted a post-experiment interviews with these 12 participants. Data from the interviews will be presented and dis-

<footnote n="4">⁴This was done at the suggestion of an anonymous reviewer. We appreciate this advice.</footnote>
<table><caption>Table 3: Numbers and percentages (in parentheses) of participants rejecting and accepting each condition.</caption><thead><tr><td>Condition</td><td>Rejecting</td><td>Accepting</td><td>Not Sure</td></tr></thead><tbody><tr><td>A</td><td>0 (0.00)</td><td>34 (51.52)</td><td>32 (48.48)</td></tr><tr><td>B</td><td>12 (18.18)</td><td>20 (30.30)</td><td>34 (51.52)</td></tr><tr><td>C</td><td>0 (0.00)</td><td>50 (75.76)</td><td>16 (24.24)</td></tr></tbody></table>

cussed in the next section.

## 4 Discussion and Concluding Remarks

Our study sought to explore whether different types of possessive relations affect the integration of long-distance dependency in Mandarin relative clauses. This study supports the (in)alienability effect in some respects. The fact that our participants gave their highest acceptability ratings to PRCs with body parts as their possessed nouns suggests that inherent part-whole association facilitates the resolution of the possessor-possessum relations inside of relative clauses. In other words, when readers encounter a term for a body part, they expect to find a possessor in the sentence to fulfill the semantic dependency. Thus, Condition C of body parts being rated more acceptable than the inalienable Condition A was consistent with our prediction, and supports the idea that inherent semantic association facilitates the resolution of long-distance dependency.

When we consider subtypes of alienable possessive relationships, however, we found some unexpected results. Given the operation of (in)alienability effects in the resolution of long-distance dependency, it would be reasonable to predict that kinship terms as the possessed nouns should also receive ratings higher than those in the alienable condition, just as we found with nouns in the condition of body parts. However, our results contradicted this prediction: Condition B received the lowest ratings. This may be related to an essential characteristic of Condition B: that both the possessor and possessum are humans, unlike in the other two conditions in which the possessum is inanimate. The same [+human] feature may interfere with the resolution of ‘who did what to whom’ in a relative clause that contains multiple animate references (Mak et al., 2002; Gordon et al., 2001, 2002). Moreover, kinship terms in our experimental items appear at the beginning of the clause, which tends to be regarded as the subject in shallow processing (Christianson et al.,

2001; Qian et al., 2018; Ferreira et al., 2002). Coupled with the [+human] feature, the clause’s initial position gives the kinship term great salience, but as the sentence unfolds, it turns out that it is possessed by the head noun and is not the subject of the sentence. This contradiction requires a proper reanalysis of thematic roles, resulting in a demand of additional processing demand, and therefore lower ratings of acceptability ratings.

Interview comments made by those Experiment 2 participants who rated Condition B as having low acceptability supported these views. As Table 3 shows, twelve participants consistently rated PRCs with kinship terms as the possessum as unacceptable. When we asked them why, one participant commented that she rated sentences like (12) low because their meanings did not correspond to her expectations: since it was the *qinqi* ('relative') who was hit, it should be that *qinqi* rather than the other person (*Laofeng*) should be sympathized with. Another participant reported that she found such sentences illogical because they mixed important information from the insignificant nouns. These comments support our conjecture that the two human nouns in Condition B require readers to decide whether the main event is expressed by the possessum or the possessor, complicating the comprehension process for the sentence as a whole. It is important to note that these concerns could not arise in the other two conditions because the possessed nouns were inanimate, and thus did not fit the descriptions of the predicate.

(12) Qinqi bei meiti dashang de relative PASS press hit-hurt DE
Laofeng shiren tongqing.
Laofeng evoke sympathy
‘Laofeng, whose relatives were hit and hurt by the press, evoked sympathy.’

Some participants seemed to adopt the ‘good-enough’ processing strategy (Qian et al., 2018) when they rated the sentences;<fnref n="5" /> therefore, the additional processing efforts required by items in Condition B rendered them unable to comprehend such sentences. One participant reported that

<footnote n="5">⁵One anonymous reviewer suggested that we conduct a follow-up study in which the level of processing (shallow or deep) is manipulated by including a secondary task. We appreciated this suggestion very much, and plan to conduct an eye-tracking experiment in which the participants answer comprehension questions, designed to trigger different levels of processing, after reading the sentences.</footnote>
<table><caption>Table 3: Numbers and percentages (in parentheses) of participants rejecting and accepting each condition.</caption><thead><tr><td>Condition</td><td>Rejecting</td><td>Accepting</td><td>Not Sure</td></tr></thead><tbody><tr><td>A</td><td>0 (0.00)</td><td>34 (51.52)</td><td>32 (48.48)</td></tr><tr><td>B</td><td>12 (18.18)</td><td>20 (30.30)</td><td>34 (51.52)</td></tr><tr><td>C</td><td>0 (0.00)</td><td>50 (75.76)</td><td>16 (24.24)</td></tr></tbody></table>

cussed in the next section.

## 4 Discussion and Concluding Remarks

Our study sought to explore whether different types of possessive relations affect the integration of long-distance dependency in Mandarin relative clauses. This study supports the (in)alienability effect in some respects. The fact that our participants gave their highest acceptability ratings to PRCs with body parts as their possessed nouns suggests that inherent part-whole association facilitates the resolution of the possessor-possessum relations inside of relative clauses. In other words, when readers encounter a term for a body part, they expect to find a possessor in the sentence to fulfill the semantic dependency. Thus, Condition C of body parts being rated more acceptable than the inalienable Condition A was consistent with our prediction, and supports the idea that inherent semantic association facilitates the resolution of long-distance dependency.

When we consider subtypes of alienable possessive relationships, however, we found some unexpected results. Given the operation of (in)alienability effects in the resolution of long-distance dependency, it would be reasonable to predict that kinship terms as the possessed nouns should also receive ratings higher than those in the alienable condition, just as we found with nouns in the condition of body parts. However, our results contradicted this prediction: Condition B received the lowest ratings. This may be related to an essential characteristic of Condition B: that both the possessor and possessum are humans, unlike in the other two conditions in which the possessum is inanimate. The same [+human] feature may interfere with the resolution of ‘who did what to whom’ in a relative clause that contains multiple animate references (Mak et al., 2002; Gordon et al., 2001, 2002). Moreover, kinship terms in our experimental items appear at the beginning of the clause, which tends to be regarded as the subject in shallow processing (Christianson et al.,

2001; Qian et al., 2018; Ferreira et al., 2002). Coupled with the [+human] feature, the clause’s initial position gives the kinship term great salience, but as the sentence unfolds, it turns out that it is possessed by the head noun and is not the subject of the sentence. This contradiction requires a proper reanalysis of thematic roles, resulting in a demand of additional processing demand, and therefore lower ratings of acceptability ratings.

Interview comments made by those Experiment 2 participants who rated Condition B as having low acceptability supported these views. As Table 3 shows, twelve participants consistently rated PRCs with kinship terms as the possessum as unacceptable. When we asked them why, one participant commented that she rated sentences like (12) low because their meanings did not correspond to her expectations: since it was the *qinqi* ('relative') who was hit, it should be that *qinqi* rather than the other person (*Laofeng*) should be sympathized with. Another participant reported that she found such sentences illogical because they mixed important information from the insignificant nouns. These comments support our conjecture that the two human nouns in Condition B require readers to decide whether the main event is expressed by the possessum or the possessor, complicating the comprehension process for the sentence as a whole. It is important to note that these concerns could not arise in the other two conditions because the possessed nouns were inanimate, and thus did not fit the descriptions of the predicate.

(12) Qinqi bei meiti dashang de relative PASS press hit-hurt DE
Laofeng shiren tongqing.
Laofeng evoke sympathy
‘Laofeng, whose relatives were hit and hurt by the press, evoked sympathy.’

Some participants seemed to adopt the ‘good-enough’ processing strategy (Qian et al., 2018) when they rated the sentences;<fnref n="5" /> therefore, the additional processing efforts required by items in Condition B rendered them unable to comprehend such sentences. One participant reported that

<footnote n="5">⁵One anonymous reviewer suggested that we conduct a follow-up study in which the level of processing (shallow or deep) is manipulated by including a secondary task. We appreciated this suggestion very much, and plan to conduct an eye-tracking experiment in which the participants answer comprehension questions, designed to trigger different levels of processing, after reading the sentences.</footnote>