Repository: aeye-lab/pymovements
Pull request #1102 (2025-04-17 20:30:59)

dataset: add CoLAGaze dataset
there is slight problem in the raw files, will chat with @annabondar3


------

```console
ValueError: Experiment metadata does not match the metadata in the ASC file:
- Right eye tracked: True != False
```

one can reproduce it when changing

```diff
diff --git a/src/pymovements/datasets/colagaze.yaml b/src/pymovements/datasets/colagaze.yaml
index d460bda..73d5b44 100644
--- a/src/pymovements/datasets/colagaze.yaml
+++ b/src/pymovements/datasets/colagaze.yaml
@@ -4,7 +4,7 @@ has_files:
   # need to fix bug:
   #     ValueError: Experiment metadata does not match the metadata in the ASC file:
   #     - Right eye tracked: True != False
-  gaze: false
+  gaze: true
   precomputed_events: true
   precomputed_reading_measures: true
```

and running

```python
import pymovements as pm

dataset = pm.Dataset('CoLAGaze', 'data/CoLAGaze').download().load()
```

maybe @saeub can help me

### Context

#### requires:
- #1151

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1102?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 99.94%. Comparing base [(`e6c074e`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/e6c074e632f5dfefcfe8717cff7b69adb954b9f9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) to head [(`252e808`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/252e8083842f53288d17d2f9762fca653df36d9a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##              main    #1102      +/-   ##
===========================================
- Coverage   100.00%   99.94%   -0.06%     
===========================================
  Files           87       88       +1     
  Lines         3718     3737      +19     
  Branches       638      638              
===========================================
+ Hits          3718     3735      +17     
- Misses           0        1       +1     
- Partials         0        1       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1102?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
SiQube: some more information @dkrako @saeub 

in the experiment, @annabondar3 tracked the dominant eye, thus monocular but changing eyes => the tracked eye is not defined in the experiment.

however, after the first ASCII file, the eye tracker is also defined within the dataset definition experiment. the mutability of the DatasetDefinition/Experiment is a curse here.

saeub: I see the problem. I think there are several possible solutions:
1. Decouple the GazeDataFrame-level experiment definition from the Dataset-level experiment definition, i.e., create a copy of the existing Experiment object [here](https://github.com/aeye-lab/pymovements/blob/e6c074e632f5dfefcfe8717cff7b69adb954b9f9/src/pymovements/gaze/io.py#L498). Personally I don't really like this, because this means that depending on the dataset, there will be several `Experiment` objects and it's not obvious which one contains information at which level.
2. Remove the tracked eye from the `EyeTracker` definition, since this is session-level metadata. After #1072 is implemented, the information about tracked eye should be integrated in the GazeDataFrame directly anyway. Not sure what implications this would have for other parts of the code. I think this would be the cleanest solution, but it would be a breaking change and would take more time to implement.
3. Redefine `EyeTracker.left` and `EyeTracker.right` to mean "has the left/right eye been tracked in *some part* of the dataset". I think this would be the easiest fix, as we would just have to change the logic [here](https://github.com/aeye-lab/pymovements/blob/e6c074e632f5dfefcfe8717cff7b69adb954b9f9/src/pymovements/gaze/io.py#L526-L533), but it probably makes the `left`/`right` attributes less useful, so we might as well go with option 2. directly.
SiQube: I prefer 2., but @dkrako has the bigger picture in mind
SiQube: we can merge the current dataset, after approval, and open an issue for the gaze sample data
saeub: An additional note: The automatic detection of right/left eye only works for ASC files. For non-ASC datasets, the tracked eye is only stored in the `EyeTracker` object, so if we go for option 2, we lose that information for other datasets. But then again, I'm not sure we *should* store it at the dataset-level at all, since it can be different for every participant/session, or even within one session.
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1102?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 99.94%. Comparing base [(`e6c074e`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/e6c074e632f5dfefcfe8717cff7b69adb954b9f9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) to head [(`dc574f7`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/dc574f7af1e30cdfbda349bd5ecadb1ba8afb0e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##              main    #1102      +/-   ##
===========================================
- Coverage   100.00%   99.94%   -0.06%     
===========================================
  Files           87       88       +1     
  Lines         3718     3736      +18     
  Branches       638      638              
===========================================
+ Hits          3718     3734      +16     
- Misses           0        1       +1     
- Partials         0        1       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1102?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
dkrako: > I prefer 2., but @dkrako has the bigger picture in mind

I also prefer the 2.
dkrako: > An additional note: The automatic detection of right/left eye only works for ASC files. For non-ASC datasets, the tracked eye is only stored in the `EyeTracker` object, so if we go for option 2, we lose that information for other datasets. But then again, I'm not sure we _should_ store it at the dataset-level at all, since it can be different for every participant/session, or even within one session.

I am confused. Why do we lose the information for other datasets? How are these affected?

As long as it's consistent for the dataset, it can stay defined in the `DatasetDefinition`.

As we have eyelink asc files here, we are on the safe side for now.
saeub: > I am confused. Why do we lose the information for other datasets? How are these affected?

I think what I meant is: If we simply remove the `left`/`right` attributes from the `EyeTracker` object, there is no way to specify the tracked eye in the `DatasetDefinition` anymore. For ASC-based datasets, it can be inferred from the data files, but not for CSV-based datasets. But we can of course add the information to the `DatasetDefinition` in some other way.
dkrako: > > I am confused. Why do we lose the information for other datasets? How are these affected?
> 
> I think what I meant is: If we simply remove the `left`/`right` attributes from the `EyeTracker` object, there is no way to specify the tracked eye in the `DatasetDefinition` anymore. For ASC-based datasets, it can be inferred from the data files, but not for CSV-based datasets. But we can of course add the information to the `DatasetDefinition` in some other way.

I'm completely confused. Why should we remove the attributes from the class? They are optional. 

For csv datasets with varying experiment definitions, we should extend `DatasetDefinition` to support defining experiments per file(name pattern).
saeub: That's how I understood option 2. But it's probably easier to discuss this in a meeting :sweat_smile:
dkrako: Alright, #1151 is merged so now we're ready to merge this too (after resolving the merge conflict).
SiQube: tested via 

```console
pytest -k colagaze
```


Patch:
diff --git a/docs/source/bibliography.bib b/docs/source/bibliography.bib
index 2e8546c..6eb2b7c 100644
--- a/docs/source/bibliography.bib
+++ b/docs/source/bibliography.bib
@@ -421,6 +421,21 @@ doi = {10.1016/j.dib.2021.106909}
     abstract = {Pupil tracking data are collected through the use of an infrared camera, and a head-mounted system [1]. The head-mounted system detects the relative pupil position and adjusts the mouse cursor position accordingly. The data are available for comparison of eye tracking with saccadic movements (with the head fixed in space) versus those from smooth movements (with the head moving in space). The analysis comprises two experiments for both types of eye tracking, which are performed with ten trials each for two participants. In the first experiment, the participant attempts to place the cursor into a target boundary of varying sizes. In the second experiment, the participant attempts to move the cursor to a target location within the shortest time.}
 }
 
+@inproceedings{CoLAGaze,
+    author = {Bondar, Anna and Reich, David R. and J\"{a}ger, Lena A.},
+    title = {CoLAGaze: A Corpus of Eye Movements for Linguistic Acceptability},
+    year = {2025},
+    isbn = {979-8-4007-1487-0/2025/05},
+    publisher = {Association for Computing Machinery},
+    address = {New York, NY, USA},
+    doi = {10.1145/3715669.3723120},
+    booktitle = {2025 Symposium on Eye Tracking Research and Applications},
+    numpages = {9},
+    keywords = {Eye-tracking, reading, dataset, grammatical acceptability judgments},
+    location = {Tokyo, Japan},
+    series = {ETRA '25}
+}
+
 @article{OneStop,
     title={OneStop: A 360-Participant English Eye Tracking Dataset with Different Reading Regimes},
     author={Berzak, Yevgeni and Malmaud, Jonathan and Shubi, Omer and Meiri, Yoav and Lion, Ella and Levy, Roger},
diff --git a/docs/source/datasets/public_datasets.csv b/docs/source/datasets/public_datasets.csv
index 77f0156..8caf308 100644
--- a/docs/source/datasets/public_datasets.csv
+++ b/docs/source/datasets/public_datasets.csv
@@ -2,6 +2,7 @@ Dataset,Citation,Samples,Events,Reading Measures,# Subjects,# Sessions per Subje
 :py:class:`~pymovements.datasets.BSC`,:cite:p:`BSC`,,X,,60,1,500 Hz,Text (sentences)
 :py:class:`~pymovements.datasets.BSCII`,:cite:p:`BSCII`,,X,,60,1,1000 Hz,Text (sentences)
 :py:class:`~pymovements.datasets.CodeComprehension`,:cite:p:`CodeComprehension`,,X,,27,1-2,2000 Hz,Text (code)
+:py:class:`~pymovements.datasets.CoLAGaze`,:cite:p:`CoLAGaze`,,X,X,42,1,2000 Hz,Text (sentences)
 :py:class:`~pymovements.datasets.CopCo`,":cite:p:`CopCoL1Hollenstein,CopCoL1DysBjornsdottir,CopCoL2`",X,X,X,58,1,1000 Hz,Text (documents)
 :py:class:`~pymovements.datasets.DAEMONS`,:cite:p:`DAEMONS`,,X,,250,1,1000 Hz,Image
 :py:class:`~pymovements.datasets.DIDEC`,:cite:p:`DIDEC`,X,,,112,1,250 Hz,Image
diff --git a/src/pymovements/datasets/__init__.py b/src/pymovements/datasets/__init__.py
index 9303819..fa70f03 100644
--- a/src/pymovements/datasets/__init__.py
+++ b/src/pymovements/datasets/__init__.py
@@ -28,6 +28,7 @@
     BSC
     BSCII
     CodeComprehension
+    CoLAGaze
     CopCo
     DAEMONS
     DIDEC
@@ -62,6 +63,7 @@
 from pymovements.datasets.bsc import BSC
 from pymovements.datasets.bsc2 import BSCII
 from pymovements.datasets.codecomprehension import CodeComprehension
+from pymovements.datasets.colagaze import CoLAGaze
 from pymovements.datasets.copco import CopCo
 from pymovements.datasets.daemons import DAEMONS
 from pymovements.datasets.didec import DIDEC
@@ -90,6 +92,7 @@ __all__ = [
     'BSC',
     'BSCII',
     'CodeComprehension',
+    'CoLAGaze',
     'CopCo',
     'DAEMONS',
     'DIDEC',
diff --git a/src/pymovements/datasets/colagaze.py b/src/pymovements/datasets/colagaze.py
new file mode 100644
index 0000000..1d962cf
--- /dev/null
+++ b/src/pymovements/datasets/colagaze.py
@@ -0,0 +1,184 @@
+# Copyright (c) 2022-2025 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""Provides a definition for the CoLAGaze dataset."""
+from __future__ import annotations
+
+from dataclasses import dataclass
+from dataclasses import field
+from typing import Any
+
+from pymovements.dataset.dataset_definition import DatasetDefinition
+from pymovements.gaze.experiment import Experiment
+
+
+@dataclass
+class CoLAGaze(DatasetDefinition):
+    """CoLAGaze dataset :cite:p:`CoLAGaze`.
+
+    This dataset includes eye-tracking data from native speakers of English reading
+    sentences from the CoLA dataset. Eye movements are recorded at a sampling frequency of 2,000 Hz
+    using an EyeLink 1000 eye tracker and are provided as pixel coordinates.
+
+    Check the respective paper for details :cite:p:`CoLAGaze`.
+
+    Attributes
+    ----------
+    name: str
+        The name of the dataset.
+
+    long_name: str
+        The entire name of the dataset.
+
+    has_files: dict[str, bool]
+        Indicate whether the dataset contains 'gaze', 'precomputed_events', and
+        'precomputed_reading_measures'.
+
+    resources: dict[str, list[dict[str, str]]]
+        A list of dataset gaze_resources. Each list entry must be a dictionary with the following
+        keys:
+        - `resource`: The url suffix of the resource. This will be concatenated with the mirror.
+        - `filename`: The filename under which the file is saved as.
+        - `md5`: The MD5 checksum of the respective file.
+
+    extract: dict[str, bool]
+        Decide whether to extract the data.
+
+    experiment: Experiment
+        The experiment definition.
+
+    filename_format: dict[str, str]
+        Regular expression which will be matched before trying to load the file. Namedgroups will
+        appear in the `fileinfo` dataframe.
+
+    filename_format_schema_overrides: dict[str, dict[str, type]]
+        If named groups are present in the `filename_format`, this makes it possible to cast
+        specific named groups to a particular datatype.
+
+    custom_read_kwargs: dict[str, dict[str, Any]]
+        If specified, these keyword arguments will be passed to the file reading function.
+
+    Examples
+    --------
+    Initialize your :py:class:`~pymovements.dataset.Dataset` object with the
+    :py:class:`~pymovements.datasets.CoLAGaze` definition:
+
+    >>> import pymovements as pm
+    >>>
+    >>> dataset = pm.Dataset("CoLAGaze", path='data/CoLAGaze')
+
+    Download the dataset resources:
+
+    >>> dataset.download()# doctest: +SKIP
+
+    Load the data into memory:
+
+    >>> dataset.load()# doctest: +SKIP
+    """
+
+    # pylint: disable=similarities
+    # The PublicDatasetDefinition child classes potentially share code chunks for definitions.
+
+    name: str = 'CoLAGaze'
+
+    long_name: str = 'Corpus of Eye Movements for Linguistic Acceptability'
+
+    has_files: dict[str, bool] = field(
+        default_factory=lambda: {
+            'gaze': True,
+            'precomputed_events': True,
+            'precomputed_reading_measures': True,
+        },
+    )
+
+    resources: dict[str, list[dict[str, str]]] = field(
+        default_factory=lambda: {
+            'gaze': [
+                {
+                    'resource':
+                    'https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/'
+                    '67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=',
+                    'filename': 'raw_data.zip',
+                    'md5': None,  # type: ignore
+                },
+            ],
+            'precomputed_events': [
+                {
+                    'resource':
+                    'https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/'
+                    '67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=',
+                    'filename': 'fixations.zip',
+                    'md5': None,  # type: ignore
+                },
+            ],
+            'precomputed_reading_measures': [
+                {
+                    'resource':
+                    'https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/'
+                    '67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=',
+                    'filename': 'measures.zip',
+                    'md5': None,  # type: ignore
+                },
+            ],
+        },
+    )
+
+    extract: dict[str, bool] = field(
+        default_factory=lambda: {
+            'gaze': True,
+            'precomputed_events': True,
+            'precomputed_reading_measures': True,
+        },
+    )
+
+    experiment: Experiment = field(
+        default_factory=lambda: Experiment(
+            screen_width_px=1280,
+            screen_height_px=1024,
+            screen_width_cm=54.37,
+            screen_height_cm=30.26,
+            distance_cm=60,
+            origin='bottom left',
+            sampling_rate=2000,
+        ),
+    )
+
+    filename_format: dict[str, str] = field(
+        default_factory=lambda: {
+            'gaze': '{subject_id:d}.asc',
+            'precomputed_events': 'fixations_report_{subject_id:d}.csv',
+            'precomputed_reading_measures': 'raw_measures_for_features{subject_id:d}.csv',
+        },
+    )
+
+    filename_format_schema_overrides: dict[str, dict[str, type]] = field(
+        default_factory=lambda: {
+            'gaze': {'subject_id': int},
+            'precomputed_events': {'subject_id': int},
+            'precomputed_reading_measures': {'subject_id': int},
+        },
+    )
+
+    custom_read_kwargs: dict[str, dict[str, Any]] = field(
+        default_factory=lambda: {
+            'gaze': {},
+            'precomputed_events': {},
+            'precomputed_reading_measures': {},
+        },
+    )
diff --git a/src/pymovements/datasets/colagaze.yaml b/src/pymovements/datasets/colagaze.yaml
new file mode 100644
index 0000000..83197d0
--- /dev/null
+++ b/src/pymovements/datasets/colagaze.yaml
@@ -0,0 +1,54 @@
+name: "CoLAGaze"
+
+long_name: 'Corpus of Eye Movements for Linguistic Acceptability'
+
+has_files:
+  gaze: true
+  precomputed_events: true
+  precomputed_reading_measures: true
+
+resources:
+  gaze:
+    - resource: https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=
+      filename: "raw_data.zip"
+      md5: null
+  precomputed_events:
+    - resource: https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=
+      filename: "fixations.zip"
+      md5: null
+  precomputed_reading_measures:
+    - resource: https://files.au-1.osf.io/v1/resources/gj2uk/providers/osfstorage/67e14ce0f392601163f33215/?view_only=a8ac6e0091e64d0a81d5b1fdec9bab6e&zip=
+      filename: "measures.zip"
+      md5: null
+
+extract:
+  gaze: true
+  precomputed_events: true
+  precomputed_reading_measures: true
+
+experiment:
+  screen_width_px: 1280
+  screen_height_px: 1024
+  screen_width_cm: 54.37
+  screen_height_cm: 30.26
+  distance_cm: 60
+  origin: "bottom left"
+  sampling_rate: 2000
+
+filename_format:
+  gaze: '{subject_id:d}.asc'
+  precomputed_events: 'fixations_report_{subject_id:d}.csv'
+  precomputed_reading_measures: 'raw_measures_for_features{subject_id:d}.csv'
+
+filename_format_schema_overrides:
+  gaze:
+    subject_id: !int
+  precomputed_events:
+    subject_id: !int
+  precomputed_reading_measures:
+    subject_id: !int
+
+custom_read_kwargs:
+  gaze: {}
+  precomputed_events: {}
+  precomputed_reading_measures: {}
diff --git a/src/pymovements/datasets/datasets.yaml b/src/pymovements/datasets/datasets.yaml
index 482b03f..16a2c8e 100644
--- a/src/pymovements/datasets/datasets.yaml
+++ b/src/pymovements/datasets/datasets.yaml
@@ -1,6 +1,7 @@
 - bsc
 - bsc2
 - codecomprehension
+- colagaze
 - copco
 - daemons
 - didec


Test patch:
diff --git a/tests/unit/datasets/datasets_test.py b/tests/unit/datasets/datasets_test.py
index 6c3f6e5..d5d79f6 100644
--- a/tests/unit/datasets/datasets_test.py
+++ b/tests/unit/datasets/datasets_test.py
@@ -33,6 +33,7 @@ from pymovements import DatasetLibrary
         pytest.param(pm.datasets.BSC, 'BSC', id='BSC'),
         pytest.param(pm.datasets.BSCII, 'BSCII', id='BSCII'),
         pytest.param(pm.datasets.CodeComprehension, 'CodeComprehension', id='CodeComprehension'),
+        pytest.param(pm.datasets.CoLAGaze, 'CoLAGaze', id='CoLAGaze'),
         pytest.param(pm.datasets.CopCo, 'CopCo', id='CopCo'),
         pytest.param(pm.datasets.DAEMONS, 'DAEMONS', id='DAEMONS'),
         pytest.param(pm.datasets.DIDEC, 'DIDEC', id='DIDEC'),