\pdfoutput=1

\documentclass[11pt]{article}
\usepackage{acl}

\usepackage{amsmath} %
\usepackage{graphicx}
\usepackage{hyperref}
\usepackage[capitalize]{cleveref}
\Crefformat{figure}{#2Fig.~#1#3}
\Crefmultiformat{figure}{Figs.~#2#1#3}{ and~#2#1#3}{, #2#1#3}{ and~#2#1#3}
\usepackage{times}
\usepackage{latexsym}

\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}

\usepackage{microtype}

\usepackage{xcolor}
\newcommand\XXX[1]{\textcolor{red}{XXX #1}}
\usepackage[normalem]{ulem} %
\def\repl#1#2{\textcolor{red}{XXX \sout{#1}}\textcolor{blue}{\uline{#2}}}
\title{Team \'{U}FAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models}

\author{Sunit Bhattacharya, Rishu Kumar \and Ond\v{r}ej Bojar \\
        Charles University \\ 
        Faculty Of Mathematics and Physics \\
        Insititute of Formal and Applied Linguistics \\
        \texttt{{bhattacharya,kumar,bojar}@ufal.mff.cuni.cz} \\
  }

\begin{document}
\maketitle

\begin{abstract}
Eye-Tracking data is a very useful source of information to study cognition and especially language comprehension in humans. In this paper, we describe our systems for the CMCL 2022 shared task on predicting eye-tracking information. We describe our experiments with pretrained models like BERT and XLM and the different ways in which we used those representations to predict four eye-tracking features. Along with analysing the effect of using two different kinds of pretrained multilingual language models and different ways of pooling the token-level representations, we also explore how contextual information affects the performance of the systems. Finally, we also explore if factors like augmenting linguistic information affect the predictions. Our submissions achieved
an average MAE of 5.72 and ranked $5^{th}$ in the shared task. The average MAE showed further reduction to 5.25 in post task evaluation.
\end{abstract}

\section{Introduction and Motivation}
\label{intro}
In the last decade that has seen rapid developments in AI research, the emergence of the Transformer architecture \cite{vaswani2017attention} marked a pivotal point in Natural Language Processing (NLP). Fine-tuning pretrained language models to work on various downstream tasks has become a dominant method of obtaining state-of-the-art performance in different areas. Their capability to capture linguistic knowledge and learn powerful contextual word embeddings \cite{liu2019linguistic} have made the transformer based models the work-horses in many NLP tasks. Pretrained models like the multilingual BERT \cite{devlin2019bert} and XLM \cite{conneau2020unsupervised} have also shown state-of-the-art performance on cross-lingual understanding tasks \cite{wu-dredze-2019-beto,artetxe2019cross}. In some cases like machine translation, there are even claims that deep learning systems reach translation qualities that are comparable to professional translators \cite{popel2020transforming}.

Language processing and its links with cognition is a very old research problem which has revealed how cognitive data (eg. gaze, fMRI) can be used to investigate human cognition. Attempts at using computational methods for such studies \cite{mitchell2008predicting,dehghani2017decoding} have also shown encouraging results. However recently, there have been a number of works that have tried to incorporate human cognitive data collected during reading for improving the performance of NLP systems \cite{hollenstein2019advancing}. The CMCL 2022 Shared Task of multilingual and cross-lingual prediction of human reading behavior \cite{hollenstein2022shared} explores how eye-gaze attributes can be algorithmically predicted given reading data in multilingual settings. 

Informed by the previous attempts at using pretrained multilingual language models to predict human reading behavior \cite{hollenstein-etal-2021-multilingual} we experiment with multilingual BERT and XLM based models to test which fares better in this task. For the experiments with the pretrained models, we use the trained weights from Huggingface~\cite{wolf-etal-2020-transformers} and perform the rest of our experiments using PyTorch\footnote{https://pytorch.org/}.
Inspired by the psycholinguistic research on investigating context length during processing \cite{wochna2013context}, we experiment how different contexts affect model performance. Finally, we merged the principles of the "classical" approach of feature-based prediction with the pretrained-language model based prediction for further analysis. In the following sections, we present our results from a total of 48 different models. 

\section{Task Description}
\label{taskdescription}
The CMCL 2022 Shared Task of Multilingual and Cross-lingual prediction of human reading behavior frames the task of predicting eye-gaze attributes associated with reading sentences as a regression task. The data for the task was comprised of eye movements corresponding to reading sentences in six languages (Chinese, Dutch, English, German, Hindi, Russian). The training data for the task contained 1703 sentences while the development set and test set contained 104 and 324 sentences respectively. The data was presented in a way such that for each word in a sentence there were four associated eye-tracking features in the form of the mean and standard deviation scores of the Total Reading Time (TRT) and First Fixation Duration (FFD). The features in the data were scaled in the range between 0 and 100 to facilitate evaluation via the mean absolute average (MAE).   
\section{Experiments}
A total of 48 models of different configurations were trained with the data provided for the shared task. The different configurations used to construct the models are based on intuition and literature survey.

Thee models were primarily categorized as System-1 (sys1) and System-2 (sys2) models. For some word corresponding to a sentence in the dataset, System-1 models provided no additional context information. System-2 models on the other hand, contained the information of all the words in the sentence that preceded the current word, providing additional context. This setting was inspired by works \cite{khandelwal2018sharp,clark2019does} on how context is used by language models. 

All systems under the System-1/2 labels were further trained as a BERT (bert) based system or a XLM (xlm) based system. BERT embeddings were previously used by  \citet{choudhary2021mtl782_iitd} for the eye-tracking feature prediction task in CMCL 2021.

Corresponding to each such language models (bert and xlm), the impact of different fine-tuning strategies\cite{sun2019fine} on system performance was studied. Hence, for one setting, only the contextualized word representation (CWR) was utilized by freezing the model weights and putting a learnable regression layer on top of the model output layer (classifier). Alternatively, the models were fine-tuned with the regression layer on top of them (whole). This setting is similar to the one used by \citet{li2021torontocl}. However in our case, we experiment with a BERT and XLM pretrained model. 

Additionally, we also performed experiments with pooling strategies for the layer representations by either using the final hidden representation of the first sub-word encoding of the input (first) or aggregating the representations of all sub-words using mean-pooling (mean) or sum-pooling (sum). The rationale behind using different pooling strategies was to have a sentence-level representation of the input tokens. The impact of different pooling strategies has previously been studied \cite{shao2019transformer,lee2019set} for different problems. In this paper, we analyze the effect of pooling feature-space embeddings in the context of eye-tracking feature prediction.       

Finally, for the experiments where we augmented additional lexical features (augmented) to the neural features for regression, we used word length and word-frequency as the additional information following \citet{vickers-etal-2021-cognlp}.

Constructing the experiments in this manner provided us with models with a diverse set of properties and in turn provided insights into how well the model behaves when all other things stay the same, and only one aspect of learning is changed. 
\section{Results}
The results corresponding to the top 10 systems based on the experiments described above are shown in \cref{table:1}. 

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_augmented\_sum\_classifier & 5.251 \\
\hline
bert\_sys2\_unaugmented\_first\_classifier & 5.267 \\
\hline
bert\_sys2\_augmented\_mean\_classifier & 5.272 \\
\hline
bert\_sys1\_augmented\_mean\_classifier & 5.279 \\
\hline
bert\_sys2\_augmented\_first\_classifier & 5.295 \\
\hline
xlm\_sys1\_augmented\_first\_classifier & 5.341 \\
\hline
xlm\_sys2\_augmented\_first\_whole & 5.346 \\
\hline
bert\_sys1\_augmented\_sum\_classifier & 5.353 \\
\hline
bert\_sys2\_augmented\_sum\_whole & 5.367\\
\hline
xlm\_sys2\_augmented\_first\_classifier & 5.373 \\
\hline
\end{tabular}
\caption{Top 10 best performing systems}
\label{table:1}
\end{table}

It was observed that the maximum MAE scores (and the maximum variance of scores)  for all the models was obtained for the attribute "TRT\_Avg". The attribute wise variances corresponding to the test-data for all the models are shown in \cref{table:2}. Similarly, the mean values of the attributes for all models are shown in \cref{table:3}.
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|} 
\hline
FFD\_Avg & FFD\_Std & TRT\_Avg & TRT\_Std  \\
\hline
0.194 & 0.403 & 0.637 & 0.489\\
\hline
\end{tabular}
\caption{Attribute wise variance of scores for all models}
\label{table:2}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|} 
\hline
FFD\_Avg & FFD\_Std & TRT\_Avg & TRT\_Std  \\
\hline
5.691 & 2.646 & 8.633 &	5.806\\
\hline
\end{tabular}
\caption{Attribute wise mean of scores for all models}
\label{table:3}
\end{table}

An analysis of the models based on the different experimental configurations are described in the following sections.

\subsection{System-1 vs System-2}
\cref{table:12} shows the average model performance across System-1 and System-2 configurations for both BERT and XLM based models (based on the average MAE values of the configurations). We see that for the BERT based models, the average MAE for System-1 is lower than that of System-2. But for XLM-based models, the difference is almost non-existent. 

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & Average MAE across models  \\
\hline
Sys1\_BERT & 5.66 \\
\hline
Sys1\_XLM & 5.70 \\
\hline
Sys2\_BERT & 5.72 \\
\hline
Sys2\_XLM & 5.69 \\
\hline
\end{tabular}
\caption{System-1 vs System-2 performance across models}
\label{table:12}
\end{table}

However, it should be noted that 12 out of the first 20 best performing models were System-2 models. Hence we posit that although the availability of the full sentence context is a factor for having more efficient systems, independently the factor does not seem to boost the overall performance much.  
\subsection{BERT vs XLM}
\cref{table:13} shows that there is only a tiny difference in average MAE for all four attributes (FFD\_$\mu$, FFD\_$\sigma$, TRT\_$\mu$, TRT\_$\sigma$) for all BERT vs XLM models . However, a brief look at \cref{table:4} and \cref{table:5} reveal that it was the XLM models that were responsible for slightly decreased MAE scores for 3 of the 4 attributes that were being predicted.
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & Average MAE across models  \\
\hline
BERT & 5.6920 \\
\hline
XLM & 5.6960 \\
\hline
\end{tabular}
\caption{BERT vs XLM performance across models}
\label{table:13}
\end{table}
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$  \\
\hline
BERT & 0.141 & 0.776 & 0.952 & 0.792\\
\hline
XLM & 0.236 & 0.045 & 0.349 & 0.204 \\
\hline
\end{tabular}
\caption{Attribute wise variance of scores for all BERT and XLM based models}
\label{table:4}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$ \\
\hline
BERT & 5.592 & 2.679 & 8.645 & 5.852\\
\hline
XLM & 5.789 & 2.612 & 8.622 & 5.760\\
\hline
\end{tabular}
\caption{Attribute wise mean of scores for all BERT and XLM based models}
\label{table:5}
\end{table}

We also see that the amount of variance for XLM based models was also smaller for 3 of the 4 attributes. 

\subsection{Augmented vs Un-Augmented models}

\cref{fig:aug_uaug} shows that augmented models. i.e. models that were fed information like word-frequency and word-length along with the neural representation information before being fed to the regression layer performed better than models that used only contextual word embeddings resulting from pretrained language models. \cref{table:14} and \cref{table:15} show the 5 best performing models of this category sorted by their MAE. 

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_unaugmented\_first\_classifier & 5.267\\
\hline
bert\_sys2\_unaugmented\_mean\_classifier & 5.405\\
\hline
xlm\_sys1\_unaugmented\_mean\_classifier & 5.5\\
\hline
xlm\_sys2\_unaugmented\_mean\_classifier & 5.55\\
\hline
xlm\_sys1\_unaugmented\_mean\_classifier & 5.557 \\
\hline
\end{tabular}
\caption{Performance of 5 best Un-Augmented models.}
\label{table:14}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_augmented\_sum\_classifier&5.251\\
\hline
bert\_sys2\_augmented\_mean\_classifier&5.272\\
\hline
bert\_sys1\_augmented\_mean\_classifier&5.279\\
\hline
bert\_sys2\_augmented\_first\_classifier&5.295\\
\hline
xlm\_sys1\_augmented\_first\_classifier&5.341\\
\hline
\end{tabular}
\caption{Performance of 5 best Augmented models}
\label{table:15}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$  \\
\hline
Aug  & 5.502 & 2.511 & 8.181 & 5.436 \\
\hline
Uaug  & 5.88 & 2.78 & 9.086 & 6.176\\
\hline
\end{tabular}
\caption{Attribute wise mean of scores for all Augmented and Un-augmented models}
\label{table:6}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$ \\
\hline
Aug  & 0.017 & 0.004 & 0.015 & 0.007 \\
\hline
Uaug  & 0.292 & 0.749 & 0.823 & 0.678 \\
\hline
\end{tabular}
\caption{Attribute wise variance of scores for all Augmented and Un-augmented models}
\label{table:7}
\end{table}

The mean and variance of attributes across models of these families presented in \cref{table:6} \& \ref{table:7} show that augmented models show way less variance in their predictions in comparison with neural-representation only model families.
\begin{figure}[h!]
    \centering
    \includegraphics[width=7cm]{images/augvuaug.png}
    \caption{Augmented vs Un-augmented model performance. The x-axis represents the 24 different models of each category. The y-axis shows the MAE corresponding to each model.}
    \label{fig:aug_uaug}
\end{figure}

\subsection{Nature of representation of input tokens (Pooling strategies)}

\cref{fig:cls_mean_sum} shows that using the first sub-word token or the mean-pooled representation of the entire input gives lesser MAE scores than the sum-pooled representations. It was also observed that for System-2 family of models, the mean-pooled representations were associated with lesser MAE scores in comparison to the first sub-word representation. The attribute wise mean in \cref{table:8} and attribute wise variance of model MAEs shown in \cref{table:9} illustrates this point. \cref{table:16},\cref{table:17} and \cref{table:18} show the 5 best performing models of this category sorted by their MAE. 
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_unaugmented\_first\_classifier&5.267\\
\hline
bert\_sys2\_augmented\_first\_classifier&5.295\\
\hline
xlm\_sys1\_augmented\_first\_classifier&5.341\\
\hline
xlm\_sys2\_augmented\_first\_whole&5.346\\
\hline
xlm\_sys2\_augmented\_first\_classifier&5.373\\
\hline
\end{tabular}
\caption{Performance of 5 best first models}
\label{table:16}
\end{table}
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_augmented\_mean\_classifier&5.272\\
\hline
bert\_sys1\_augmented\_mean\_classifier&5.279\\
\hline
bert\_sys2\_augmented\_mean\_whole&5.375\\
\hline
bert\_sys2\_unaugmented\_mean\_classifier&5.405\\
\hline
xlm\_sys1\_augmented\_mean\_whole&5.413\\
\hline
\end{tabular}
\caption{Performance of 5 best Mean models}
\label{table:17}
\end{table}
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|} 
\hline
Model & MAE \\
\hline
bert\_sys2\_augmented\_sum\_classifier&5.251\\
\hline
bert\_sys1\_augmented\_sum\_classifier&5.353\\
\hline
bert\_sys2\_augmented\_sum\_whole&5.367\\
\hline
bert\_sys1\_augmented\_sum\_whole&5.402\\
\hline
xlm\_sys2\_augmented\_sum\_classifier&5.456\\
\hline
\end{tabular}
\caption{Performance of 5 best Sum models}
\label{table:18}
\end{table}
\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$  \\
\hline
first  & 5.549 & 2.505 & 8.434 & 5.615 \\
\hline
Mean  & 5.57 & 2.538 & 8.416 & 5.636 \\
\hline
Sum  & 5.954 & 2.894 & 9.05 & 6.167 \\
\hline
\end{tabular}
\caption{Attribute wise mean of scores for models with different input token representations}
\label{table:8}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$ \\
\hline
first  & 0.036 & 0.004 & 0.118 & 0.054 \\
\hline
Mean  & 0.047 & 0.005 & 0.118 & 0.048 \\
\hline
Sum  & 0.383 & 1.082 & 1.374 & 1.139 \\
\hline
\end{tabular}
\caption{Attribute wise variance of scores for models with different input token representations}
\label{table:9}
\end{table}
\begin{figure}[h!]
    \centering
    \includegraphics[width=7cm]{images/clsvmeanvsum.png}
    \caption{Model performance based on the nature of representation of input tokens.The x-axis represents the 16 different models of each category. The y-axis shows the MAE corresponding to each model.}
    \label{fig:cls_mean_sum}
\end{figure}

\subsection{Fine-tuning}

Fine-tuning on large pretrained language models has become the standard way to conduct NLP research after the widespread adoption of the transformer architecture. And unsurprisingly, our experiments reveal (\cref{fig:finetune}) that fine-tuning of models give smaller MAE scores than training only the regression layers. The stark difference in the variance for the predicted attributes between fine-tuned models and regression only models (as illustrated in \cref{table:10}-\ref{table:11}) further demonstrates the advantage of fine-tuning.

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$  \\
\hline
Aug  & 5.502 & 2.511 & 8.181 & 5.436 \\
\hline
Uaug  & 5.88 & 2.78 & 9.086 & 6.176\\
\hline
\end{tabular}
\caption{Attribute wise variance of scores for fine-tuned models vs regression-layer only models}
\label{table:10}
\end{table}

\begin{table}[h!]
\centering
\begin{tabular}{|c|c|c|c|c|} 
\hline
Model & FFD\_$\mu$ & FFD\_$\sigma$ & TRT\_$\mu$ & TRT\_$\sigma$ \\
\hline
Aug  & 0.017 & 0.004 & 0.015 & 0.007 \\
\hline
Uaug  & 0.292 & 0.749 & 0.823 & 0.678 \\
\hline
\end{tabular}
\caption{Attribute wise mean of scores for fine-tuned models vs regression-layer only models}
\label{table:11}
\end{table}
\begin{figure}[h!]
    \centering
    \includegraphics[width=7cm]{images/regvfine.png}
    \caption{Fine-tuning vs training only regression layer in the models. The x-axis represents the 24 different models of each category. The y-axis shows the MAE corresponding to each model.}
    \label{fig:finetune}
\end{figure}
\section{Conclusion}
In this paper, we have described our experiments with different kinds of models that were trained on the data provided for this shared-task. We have identified five ways in which we can make better systems to predict eye-tracking features based on eye-tracking data from a multilingual corpus. First, the experiments demonstrate that the inclusion of context (previous words occurring in the sentence) helps the models to predict eye-tracking attributes better. This reaffirms previous observations made with language models that more context is always helpful. Second, we find that XLM based models perform relatively better than the BERT based models. Third, our experiments show the advantages of augmenting additional linguistic features (word length and word frequency information in this case) to the contextual word representations to make better systems. This is in agreement with the findings from eye-tracking prediction tasks from last iterations of CMCL. Fourth, we see how different pooling methods applied on the input token representations affect the final performance of the systems. Finally, the experiments re-validate the approach of fine-tuning pretrained language models for specific tasks. Hence we conclude that contextualized word representations from language models pretrained with many different languages, if carefully augmented, engineered, and fine-tuned, can predict eye-tracking features quite successfully. 
\section{Acknowledgement}
This work has been funded from the grant 19-26934X (NEUREM3) of the Czech Science Foundation. 
\bibliography{anthology,custom}
\bibliographystyle{acl_natbib}
\end{document}

Peer review:
Meta-review
The paper presents the system description of Team ÚFAL for the CMCL 2022 Shared Task on Multilingual and Crosslingual Prediction of Human Reading Behavior. The authors present a range of model comparisons for eye-tracking prediction. As pointed out by the reviewers, the paper has potential but requires some improvements. The descriptions should be more precise, especially regarding the motivation of the chosen model architectures and the discussion/analysis of the results.

We urge the authors to take the feedback from the reviewers into account and to improve their paper for the camera-ready deadline.
Rating: 6: Marginally above acceptance threshold
Confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature