Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading

In this paper, we develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. The alignment of the two sequences is achieved by a cross-sequence attention mechanism. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths. We provide an extensive within- and across-data set evaluation on different languages. An ablation study and qualitative analysis support an in-depth understanding of the model's behavior.

The overview above describes the original authors' work. This project uses the original Eyettention implementation, reorganized as the Eyettention package, with raw-text inference, scanpath-prefix replay, an endpoint handler, and a Gradio interface.

Setup

For this checkout, run setup and inference commands from the project root (the directory containing Eyettention/). Install dependencies:

python -m pip install -r Eyettention/requirements.txt

The requirements retain historical PyTorch and Transformers pins that may need adjustment for your Python version and platform. The local preprocessing code also requires LAC and a compatible PaddlePaddle installation; BSC Excel loading requires openpyxl. Install gradio if using the web interface. Use a separate environment from ScanDL2 because their dependency versions differ.

Dataset

For the CELER dataset, follow the instructions at the CELER repository. In order to run experiments, place the downloaded dataset in Eyettention/Data/celer/. The local loaders expect data_v2.0/sent_fix.tsv, data_v2.0/sent_ia.tsv, and metadata.tsv beneath that directory.

The Chinese Beijing Sentence Corpus (BSC) belongs in Eyettention/Data/beijing-sentence-corpus/, with BSC.Word.Info.v2.xlsx and BSC.EMD/BSC.EMD.txt. See utils.py for the dataset loaders.

Raw-text inference uses the trained checkpoints and normalization files already stored under results/ and Data/; it does not require the full training corpora.

Run Experiments

The experiment scripts use paths relative to Eyettention/. Starting at the project root:

cd Eyettention
mkdir -p results/BSC results/CELER

Keep the project root on PYTHONPATH when running the package modules.

For Chinese BSC dataset

PYTHONPATH=.. python -m Eyettention.main_BSC --test_mode text --gpu 0
PYTHONPATH=.. python -m Eyettention.main_BSC --test_mode subject --gpu 0
PYTHONPATH=.. python -m Eyettention.main_BSC_NRS_setting --gpu 0
PYTHONPATH=.. python -m Eyettention.main_BSC_reader_identifier --gpu 0

For English CELER dataset

PYTHONPATH=.. python -m Eyettention.main_celer --test_mode text --gpu 0
PYTHONPATH=.. python -m Eyettention.main_celer --test_mode subject --gpu 0
PYTHONPATH=.. python -m Eyettention.main_celer_NRS_setting --gpu 0
PYTHONPATH=.. python -m Eyettention.main_celer_reader_identifier --gpu 0

text evaluates new sentences; subject evaluates new readers. Select an available GPU with --gpu. Several historical experiment scripts call CUDA directly, so use a CUDA-enabled environment for these commands. Review output paths before running; --save_data_folder selects the results directory.

Raw-text inference

Run the following from the project root:

from Eyettention import EyettentionRawTextInference

model = EyettentionRawTextInference(
    checkpoint_path="Eyettention/results/CELER/Eyettention_english.pth",
    dataset="celer",
    device="cpu",
)
scanpath, density = model.generate_from_english_text(
    "The quick brown fox jumps.",
    max_pred_len=20,
    # Optional: replay observed fixation positions before sampling.
    # previous_scanpath=[0, 1, 2],
)
print(scanpath[0].tolist())

For Chinese, use dataset="BSC", checkpoint Eyettention/results/BSC/Eyettention_chinese.pth, and generate_from_chinese_text(...). Dataset names are case-sensitive. Use device="cuda" for GPU inference. BERT assets must be downloadable from Hugging Face or cached locally.

Keep Data/feature_norm_celer.pickle and Data/feature_norm_BSC.pickle alongside the package. The output contains fixation indices and per-step probability distributions, not fixation durations. Index 0 denotes CLS, positions 1..N refer to words or Chinese characters, and N+1 denotes SEP; stop at the first SEP when interpreting fixations. Use short sentences within the checkpoint's configured input limits. Prefix replay extends the generation loop, so max_pred_len is not a strict total-length cap when a prefix is provided.

Endpoint handler

The added handler.py wraps inference in a dictionary-based interface:

from Eyettention.handler import EndpointHandler

handler = EndpointHandler()
result = handler({
    "inputs": "The quick brown fox jumps.",
    "parameters": {
        "dataset": "celer",
        "device": "cpu",
        "max_pred_len": 20,
    },
})

It returns scanpath and density_steps. Optional parameters include checkpoint_path and previous_scanpath. The handler defaults to BSC on CPU and does not itself start an HTTP server.

Gradio interface

The added app.py provides a web interface. From the project root, point it to the local checkpoints and launch:

export EYETTENTION_BSC_CHECKPOINT="Eyettention/results/BSC/Eyettention_chinese.pth"
export EYETTENTION_CELER_CHECKPOINT="Eyettention/results/CELER/Eyettention_english.pth"
python -m Eyettention.app

Open http://localhost:7860, choose the dataset, enter text, and click Run. The interface also supports an observed scanpath prefix. It selects CUDA when available, otherwise CPU. Set PORT to change the default port; the app binds to 0.0.0.0.

Cite our work

If you use this code for your research, please consider citing the original authors' paper:

@article{deng2023eyettention,
  title={Eyettention: {A}n Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading},
  author={Deng, Shuwen and Reich, David R and Prasse, Paul and Haller, Patrick and Scheffer, Tobias and J{\"a}ger, Lena A},
  journal={Proceedings of the {ACM} on Human-Computer Interaction},
  volume={7},
  number={ETRA},
  pages={1--24},
  year={2023},
  publisher={ACM New York, NY, USA}
}

License

The original code is provided under the MIT License, copyright © 2023 AEye. Include the copyright and permission notice when redistributing copies or substantial portions of the software. See the license file for the full terms and warranty disclaimer. Datasets and dependencies retain their own licenses.