Repository: aeye-lab/pymovements
Pull request #473 (2023-08-10 18:31:44)

feat: Add trial_columns and trialize transform() method

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/473?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`1b716eb`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/1b716eb97857ff598f0a22b13ecc7da503e7d3ea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) 100.00% compared to head [(`00b5d01`)](https://app.codecov.io/gh/aeye-lab/pymovements/pull/473?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) 100.00%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main      #473   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           47        47           
  Lines         2100      2101    +1     
  Branches       504       504           
=========================================
+ Hits          2100      2101    +1     
```


| [Files Changed](https://app.codecov.io/gh/aeye-lab/pymovements/pull/473?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) | Coverage Δ | |
|---|---|---|
| [src/pymovements/gaze/gaze\_dataframe.py](https://app.codecov.io/gh/aeye-lab/pymovements/pull/473?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab#diff-c3JjL3B5bW92ZW1lbnRzL2dhemUvZ2F6ZV9kYXRhZnJhbWUucHk=) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/473?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).


Patch:
diff --git a/docs/source/tutorials/local-dataset.ipynb b/docs/source/tutorials/local-dataset.ipynb
index bc708d6..a74b28b 100644
--- a/docs/source/tutorials/local-dataset.ipynb
+++ b/docs/source/tutorials/local-dataset.ipynb
@@ -169,15 +169,48 @@
     "}"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "9e63f355",
+   "metadata": {},
+   "source": [
+    "## Column Definitions"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "18b5b563",
+   "metadata": {},
+   "source": [
+    "The `trial_columns` argument can be used to specify which columns define a single trial.\n",
+    "\n",
+    "This is important for correctly applying all preprocessing methods.\n",
+    "\n",
+    "For this very small single user dataset a trial ist just defined by `text_id` and `page_id`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5f5e4789",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "trial_columns = ['text_id', 'page_id']"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "e039de9b",
    "metadata": {},
    "source": [
-    "The `time_column` and `pixel_columns` arguments can be used to specify your columns according to their content.\n",
-    "This is needed for correctly applying all preprocessing methods.\n",
+    "The `time_column` and `pixel_columns` arguments can be used to correctly map the columns in your dataframes.\n",
+    "\n",
+    "Depending on the content of your dataset, you can alternatively also provide `position_columns`, `velocity_columns` and `acceleration_columns`.\n",
+    "\n",
+    "Specifying these columns is needed for correctly applying preprocessing methods. For example, if you want to apply the `pix2deg` method, you will need to specify `pixel_columns` accordingly.\n",
     "\n",
-    "Depending on the content of your dataset, you can alternatively also provide `position_columns`, `velocity_columns` and `acceleration_columns`."
+    "If your dataset has gaze positions available only in degrees of visual angle, you have to specify the `position_columns` instead."
    ]
   },
   {
diff --git a/src/pymovements/dataset/dataset_definition.py b/src/pymovements/dataset/dataset_definition.py
index a7017a8..bf01bd9 100644
--- a/src/pymovements/dataset/dataset_definition.py
+++ b/src/pymovements/dataset/dataset_definition.py
@@ -79,6 +79,7 @@ class DatasetDefinition:
 
     column_map: dict[str, str] = field(default_factory=dict)
 
+    trial_columns: list[str] | None = None
     time_column: str | None = None
     pixel_columns: list[str] | None = None
     position_columns: list[str] | None = None
diff --git a/src/pymovements/dataset/dataset_files.py b/src/pymovements/dataset/dataset_files.py
index 66ad4d0..a76607b 100644
--- a/src/pymovements/dataset/dataset_files.py
+++ b/src/pymovements/dataset/dataset_files.py
@@ -225,6 +225,7 @@ def load_gaze_files(
             gaze_df = GazeDataFrame(
                 gaze_data,
                 experiment=definition.experiment,
+                trial_columns=definition.trial_columns,
             )
 
         elif preprocessed and extension == 'csv':
@@ -261,6 +262,7 @@ def load_gaze_files(
             gaze_df = GazeDataFrame(
                 gaze_data,
                 experiment=definition.experiment,
+                trial_columns=definition.trial_columns,
                 time_column=time_column,
                 pixel_columns=pixel_columns,
                 position_columns=position_columns,
@@ -272,6 +274,7 @@ def load_gaze_files(
             gaze_df = GazeDataFrame(
                 gaze_data,
                 experiment=definition.experiment,
+                trial_columns=definition.trial_columns,
                 time_column=definition.time_column,
                 pixel_columns=definition.pixel_columns,
                 position_columns=definition.position_columns,
diff --git a/src/pymovements/datasets/gazebase.py b/src/pymovements/datasets/gazebase.py
index d61360c..7481b77 100644
--- a/src/pymovements/datasets/gazebase.py
+++ b/src/pymovements/datasets/gazebase.py
@@ -137,6 +137,10 @@ class GazeBase(DatasetDefinition):
         },
     )
 
+    trial_columns: list[str] = field(
+        default_factory=lambda: ['round_id', 'subject_id', 'session_id', 'task_name'],
+    )
+
     time_column: str = 'n'
 
     position_columns: list[str] = field(default_factory=lambda: ['x', 'y'])
diff --git a/src/pymovements/datasets/gazebasevr.py b/src/pymovements/datasets/gazebasevr.py
index 888f2e2..f144560 100644
--- a/src/pymovements/datasets/gazebasevr.py
+++ b/src/pymovements/datasets/gazebasevr.py
@@ -138,6 +138,10 @@ class GazeBaseVR(DatasetDefinition):
         },
     )
 
+    trial_columns: list[str] = field(
+        default_factory=lambda: ['round_id', 'subject_id', 'session_id', 'task_name'],
+    )
+
     time_column: str = 'n'
 
     position_columns: list[str] = field(default_factory=lambda: ['lx', 'ly', 'rx', 'ry', 'x', 'y'])
diff --git a/src/pymovements/datasets/judo1000.py b/src/pymovements/datasets/judo1000.py
index b0ecc53..8af9d60 100644
--- a/src/pymovements/datasets/judo1000.py
+++ b/src/pymovements/datasets/judo1000.py
@@ -125,6 +125,10 @@ class JuDo1000(DatasetDefinition):
         },
     )
 
+    trial_columns: list[str] = field(
+        default_factory=lambda: ['subject_id', 'session_id', 'trial_id'],
+    )
+
     time_column: str = 'time'
     pixel_columns: list[str] = field(
         default_factory=lambda: [
diff --git a/src/pymovements/datasets/toy_dataset.py b/src/pymovements/datasets/toy_dataset.py
index 480bb71..27c2056 100644
--- a/src/pymovements/datasets/toy_dataset.py
+++ b/src/pymovements/datasets/toy_dataset.py
@@ -123,6 +123,8 @@ class ToyDataset(DatasetDefinition):
         },
     )
 
+    trial_columns: list[str] = field(default_factory=lambda: ['text_id', 'page_id'])
+
     time_column: str = 'timestamp'
 
     pixel_columns: list[str] = field(default_factory=lambda: ['x', 'y'])
diff --git a/src/pymovements/datasets/toy_dataset_eyelink.py b/src/pymovements/datasets/toy_dataset_eyelink.py
index fb1b162..4a86c9d 100644
--- a/src/pymovements/datasets/toy_dataset_eyelink.py
+++ b/src/pymovements/datasets/toy_dataset_eyelink.py
@@ -128,6 +128,8 @@ class ToyDatasetEyeLink(DatasetDefinition):
 
     column_map: dict[str, str] = field(default_factory=lambda: {})
 
+    trial_columns: list[str] = field(default_factory=lambda: ['subject_id', 'session_id'])
+
     time_column: str = 'time'
 
     pixel_columns: list[str] = field(default_factory=lambda: ['x_pix', 'y_pix'])
diff --git a/src/pymovements/gaze/gaze_dataframe.py b/src/pymovements/gaze/gaze_dataframe.py
index 1bcc0ed..79c9945 100644
--- a/src/pymovements/gaze/gaze_dataframe.py
+++ b/src/pymovements/gaze/gaze_dataframe.py
@@ -78,6 +78,7 @@ class GazeDataFrame:
             data: pl.DataFrame | None = None,
             experiment: Experiment | None = None,
             *,
+            trial_columns: str | list[str] | None = None,
             time_column: str | None = None,
             pixel_columns: list[str] | None = None,
             position_columns: list[str] | None = None,
@@ -166,6 +167,8 @@ class GazeDataFrame:
             data = data.clone()
         self.frame = data
 
+        self.trial_columns = trial_columns
+
         if time_column is not None:
             self.frame = self.frame.rename({time_column: 'time'})
 
@@ -272,7 +275,15 @@ class GazeDataFrame:
                 _check_n_components(self.n_components)
                 kwargs['n_components'] = self.n_components
 
-            self.frame = self.frame.with_columns(transform_method(**kwargs))
+            if self.trial_columns is None:
+                self.frame = self.frame.with_columns(transform_method(**kwargs))
+            else:
+                self.frame = pl.concat(
+                    [
+                        df.with_columns(transform_method(**kwargs))
+                        for group, df in self.frame.groupby(self.trial_columns, maintain_order=True)
+                    ],
+                )
 
     def pix2deg(self) -> None:
         """Compute gaze positions in degrees of visual angle from pixel position coordinates.


Test patch:
diff --git a/tests/gaze/gaze_transform_test.py b/tests/gaze/gaze_transform_test.py
index bcd52ac..66c7d24 100644
--- a/tests/gaze/gaze_transform_test.py
+++ b/tests/gaze/gaze_transform_test.py
@@ -101,10 +101,7 @@ def fixture_experiment():
                     {
                         'time': np.arange(1000, 1010, 2),
                         'x_pix': np.arange(0, 1, 0.2),
-                        'y_pix': [
-                            20.0, 20.200000000000003, 20.400000000000006,
-                            20.60000000000001, 20.80000000000001,
-                        ],
+                        'y_pix': [20.0, 20.2, 20.4, 20.6, 20.8],
                     },
                 ),
                 pixel_columns=['x_pix', 'y_pix'],
@@ -206,7 +203,7 @@ def fixture_experiment():
                         'time': [1000, 1000],
                         'x_pix': [49.5, 49.5],
                         'y_pix': [0.0, 0.0],
-                        'x_dva': [26.335410003881348, 26.335410003881348],
+                        'x_dva': [26.3354, 26.3354],
                         'y_dva': [0.0, 0.0],
                     },
                 ),
@@ -243,7 +240,7 @@ def fixture_experiment():
                         'time': [1000, 1000],
                         'x_pix': [49.5, 49.5],
                         'y_pix': [0.0, 0.0],
-                        'x_dva': [26.335410003881348, 26.335410003881348],
+                        'x_dva': [26.3354, 26.3354],
                         'y_dva': [0.0, 0.0],
                     },
                 ),
@@ -281,7 +278,7 @@ def fixture_experiment():
                         'x_pix': [49.5],
                         'y_pix': [0.0],
                         'x_dva': [0.0],
-                        'y_dva': [-26.335410003881348],
+                        'y_dva': [-26.3354],
                     },
                 ),
                 pixel_columns=['x_pix', 'y_pix'],
@@ -294,9 +291,10 @@ def fixture_experiment():
             {
                 'data': pl.from_dict(
                     {
-                        'time': [1000, 1001, 1002],
-                        'x_dva': [1.0, 1.0, 1.0],
-                        'y_dva': [1.0, 1.1, 1.2],
+                        'trial_id': [1, 1, 1, 2, 2, 2],
+                        'time': [1000, 1001, 1002, 1003, 1004, 1005],
+                        'x_dva': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
+                        'y_dva': [1.0, 1.1, 1.2, 1.0, 1.1, 1.2],
                     },
                 ),
                 'experiment': pm.Experiment(
@@ -314,11 +312,12 @@ def fixture_experiment():
             pm.GazeDataFrame(
                 data=pl.from_dict(
                     {
-                        'time': [1000, 1001, 1002],
-                        'x_dva': [1.0, 1.0, 1.0],
-                        'y_dva': [1.0, 1.1, 1.2],
-                        'x_vel': [None, 0.0, 0.0],
-                        'y_vel': [None, 100.00000000000009, 99.99999999999987],
+                        'trial_id': [1, 1, 1, 2, 2, 2],
+                        'time': [1000, 1001, 1002, 1003, 1004, 1005],
+                        'x_dva': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
+                        'y_dva': [1.0, 1.1, 1.2, 1.0, 1.1, 1.2],
+                        'x_vel': [None, 0.0, 0.0, 0.0, 0.0, 0.0],
+                        'y_vel': [None, 100.0, 100.0, -200.0, 100.0, 100.0],
                     },
                 ),
                 position_columns=['x_dva', 'y_dva'],
@@ -355,7 +354,7 @@ def fixture_experiment():
                         'x_dva': [1.0, 1.0, 1.0],
                         'y_dva': [1.0, 1.1, 1.2],
                         'x_vel': [None, 0.0, None],
-                        'y_vel': [None, 99.99999999999997, None],
+                        'y_vel': [None, 100.0, None],
                     },
                 ),
                 position_columns=['x_dva', 'y_dva'],
@@ -392,7 +391,7 @@ def fixture_experiment():
                         'x_dva': [1.0, 1.0, 1.0, 1.0, 1.0],
                         'y_dva': [1.0, 1.1, 1.2, 1.3, 1.4],
                         'x_vel': [None, None, 0.0, None, None],
-                        'y_vel': [None, None, 100.00000000000001, None, None],
+                        'y_vel': [None, None, 100.0, None, None],
                     },
                 ),
                 position_columns=['x_dva', 'y_dva'],
@@ -400,6 +399,47 @@ def fixture_experiment():
             ),
             id='pos2vel_five_point',
         ),
+
+        pytest.param(
+            {
+                'data': pl.from_dict(
+                    {
+                        'trial_id': [1, 1, 1, 2, 2, 2],
+                        'time': [1000, 1001, 1002, 1003, 1004, 1005],
+                        'x_dva': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
+                        'y_dva': [1.0, 1.1, 1.2, 1.0, 1.1, 1.2],
+                    },
+                ),
+                'experiment': pm.Experiment(
+                    sampling_rate=1000,
+                    screen_width_px=100,
+                    screen_height_px=100,
+                    screen_width_cm=100,
+                    screen_height_cm=100,
+                    distance_cm=100,
+                    origin='lower left',
+                ),
+                'position_columns': ['x_dva', 'y_dva'],
+                'trial_columns': 'trial_id',
+            },
+            'pos2vel', {'method': 'preceding'},
+            pm.GazeDataFrame(
+                data=pl.from_dict(
+                    {
+                        'trial_id': [1, 1, 1, 2, 2, 2],
+                        'time': [1000, 1001, 1002, 1003, 1004, 1005],
+                        'x_dva': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
+                        'y_dva': [1.0, 1.1, 1.2, 1.0, 1.1, 1.2],
+                        'x_vel': [None, 0.0, 0.0, None, 0.0, 0.0],
+                        'y_vel': [None, 100.0, 100.0, None, 100.0, 100.0],
+                    },
+                ),
+                position_columns=['x_dva', 'y_dva'],
+                velocity_columns=['x_vel', 'y_vel'],
+            ),
+            id='pos2vel_preceding_trialize_single_column_str',
+        ),
+
     ],
 )
 def test_gaze_transform_expected_frame(