Package: scandy
IntroductionScanDyis a modular and mechanistic computational framework for simulating realisticscanpaths indynamic real-world scenes. It is specifically designed to quantitatively test hypotheses about eye-movement behavior in videos.
Specifically, it can be used to demonstrate the influence of object-representations on gaze behavior by comparing object-based and location-based models.For a visual guide of howScanDyworks, have a look at theinteractive notebook(also onColab) and theexample usecases.Software architectureThe structure ofScanDyis inspired by theneurolibframework, which is also used for parameter optimization and exploration.Scanpath models inherit from the `Model` base class, whose functionality includes initializing and running model simulations and the evaluation and visualization of the resulting scanpaths. Models are implemented in a modular way, consiting of moules for (I) Scene features, (II) Visual sensitivity, (III) Scanpath history, (IV) Decision making, and (V) Gaze update.InstallationYou can installScanDyas pypi package usingpip:pip install scandyWe however reccomend that you clone (or fork) this repository and install all dependencies withgit clone https://github.com/rederoth/ScanDy.git
cd ScanDy/
pip install -r requirements.txt
pip install .This gives you more freedom to modify the existing models and run the examples.CAVEAT: There is currently an incompatability with Python 3.11 and thenumbapackage (required byneurolib), seenumba/numba#8304. We therefore recommend using Python <=3.10 or manually installingnumba/neurolib.DatasetThe scanpath models require precomputed maps of the video data. We use the VidCom dataset (Li et al., 2011), for which we provide all the required data on OSF (https://www.doi.org/10.17605/OSF.IO/83XUC).To prepare the dataset, we used the following resources:VidCom- Video and eye-tracking datadeep_em_classifier- Eye movement classificationdetectron2- Frame-wise object segmentationdeep_sort- Object trackingdynamic-proto-object-saliency- Low-level saliency mapsTASED-Net- High-level saliency mapsPWC-Net- Optical flow calculationIf you only want to play around with a single video, we uploaded a version of the dataset only containing the "field03" video toGoogle drive.ExamplesWe prepared a number ofIPython Notebooksfor you to explore the framework.To get started withScanDy, have a look at ourinteractive guide, where you can explore the effect of individual model parameters.Additionally, we show instructive usecases, including:Example 1, onColab: Scanpath simulation and visualization for a single videoExample 2, onColab: Evolutionary optimization of model parametersExample 3, onColab: Extending on existing models: Location-based model with object-based sensitivityAll figures from our manuscript (Roth et al., 2023) can be reproduced withthis notebook, which is also executable onColab.More informationHow to citeIfScanDyis useful for your research, please cite our preprint:Roth, N., Rolfs, M., Hellwich, O., & Obermayer, K. (2023). Objects guide human gaze behavior in dynamic real-world scenes.bioRxiv, 2023-03.@article{roth2023objects,title={Objects Guide Human Gaze Behavior in Dynamic Real-World Scenes},author={Roth, Nicolas and Rolfs, Martin and Hellwich, Olaf and Obermayer, Klaus},elocation-id={2023.03.14.532608},year={2023},doi={10.1101/2023.03.14.532608},publisher={Cold Spring Harbor Laboratory},}ContactIf you have feedback, questions, and/or ideas, feel free to send amailto Nico.Nicolas Roth,
PhD Student at Science of Intelligence;
Neural Information Processing Group,
Fakultaet IV, Technische Universitaet Berlin,
MAR 5-6, Marchstr. 23, 10587 BerlinAcknowledgmentsThis work was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy – EXC 2002/1 "Science of Intelligence" – project number 390523135.