Repository: aeye-lab/pymovements
Pull request #1076 (2025-04-01 08:30:56)

feat: add auto_column_detect to from_csv()
## Description

Adds automatic column detection to `from_csv()` as it was already available in `GazeDataFrame()`.

I simplified `dataset_files.py` and just use the `auto_column_detect` argument there.

The auto column detection should probably be improved somewhen (see my tests where I needed to rename the columns beforehand), but that's out of scope for this PR.

I also moved the `auto_column_detect` argument to the last position in `GazeDataFrame()`, as I think this way it is more clear what columns are detected.

## Implemented changes

Insert a description of the changes implemented in the pull request.

- [x] added `auto_column_detect` to `from_csv()` and forward it to `GazeDataFrame.__init__()`
- [x] move `auto_column_detect` to last position in `GazeDataFrame.__init__()`
- [x] refactored `dataset_files.py` to use `auto_column_detect`
- [x] doubled the number of similar lines for pylint

## Type of change

- [x] New feature (non-breaking change which adds functionality)

## How Has This Been Tested?

The csv tests currently only test the shape of the dataframe.
I improved this to check the schema of the dataframes in #1004 but I wouldn't like to rewrite the tests in this PR (I use passing the definition there).

- [x] test auto column detection for monocular and binocular example csv file

## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream modules
- [x] I have checked my code and corrected any misspellings


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1076?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 100.00%. Comparing base [(`37779e6`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/37779e63891e6662e9b80e7ed3c192400cc438a0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) to head [(`0ecbbf3`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/0ecbbf39d31a554ab47a457a9bd2cd62a38c14ed?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main     #1076   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           87        87           
  Lines         3717      3695   -22     
  Branches       639       635    -4     
=========================================
- Hits          3717      3695   -22     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/1076?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>

Patch:
diff --git a/pylintrc b/pylintrc
index 4bf1201..180d15e 100644
--- a/pylintrc
+++ b/pylintrc
@@ -155,7 +155,7 @@ notes=FIXME,XXX,TODO
 [SIMILARITIES]
 
 # Minimum lines number of a similarity.
-min-similarity-lines=8
+min-similarity-lines=16
 
 # Ignore comments when computing similarities.
 ignore-comments=yes
diff --git a/src/pymovements/dataset/dataset_files.py b/src/pymovements/dataset/dataset_files.py
index 9578691..4d94eea 100644
--- a/src/pymovements/dataset/dataset_files.py
+++ b/src/pymovements/dataset/dataset_files.py
@@ -335,41 +335,10 @@ def load_gaze_file(
                 filepath,
                 trial_columns=trial_columns,
                 time_unit=time_unit,
+                auto_column_detect=True,
                 add_columns=fileinfo_columns,
                 column_schema_overrides=definition.filename_format_schema_overrides['gaze'],
             )
-
-            # suffixes as ordered after using GazeDataFrame.unnest()
-            component_suffixes = ['x', 'y', 'xl', 'yl', 'xr', 'yr', 'xa', 'ya']
-
-            pixel_columns = ['pixel_' + suffix for suffix in component_suffixes]
-            pixel_columns = [c for c in pixel_columns if c in gaze_df.frame.columns]
-
-            position_columns = ['position_' + suffix for suffix in component_suffixes]
-            position_columns = [c for c in position_columns if c in gaze_df.frame.columns]
-
-            velocity_columns = ['velocity_' + suffix for suffix in component_suffixes]
-            velocity_columns = [c for c in velocity_columns if c in gaze_df.frame.columns]
-
-            acceleration_columns = ['acceleration_' + suffix for suffix in component_suffixes]
-            acceleration_columns = [c for c in acceleration_columns if c in gaze_df.frame.columns]
-
-            column_specifiers: list[list[str]] = []
-            if len(pixel_columns) > 0:
-                gaze_df.nest(pixel_columns, output_column='pixel')
-                column_specifiers.append(pixel_columns)
-
-            if len(position_columns) > 0:
-                gaze_df.nest(position_columns, output_column='position')
-                column_specifiers.append(position_columns)
-
-            if len(velocity_columns) > 0:
-                gaze_df.nest(velocity_columns, output_column='velocity')
-                column_specifiers.append(velocity_columns)
-
-            if len(acceleration_columns) > 0:
-                gaze_df.nest(acceleration_columns, output_column='acceleration')
-                column_specifiers.append(acceleration_columns)
         else:
             gaze_df = from_csv(
                 filepath,
diff --git a/src/pymovements/gaze/gaze_dataframe.py b/src/pymovements/gaze/gaze_dataframe.py
index 06fbb94..b3e0b97 100644
--- a/src/pymovements/gaze/gaze_dataframe.py
+++ b/src/pymovements/gaze/gaze_dataframe.py
@@ -52,8 +52,6 @@ class GazeDataFrame:
         The experiment definition. (default: None)
     events: pm.EventDataFrame | None
         A dataframe of events in the gaze signal. (default: None)
-    auto_column_detect: bool
-        Flag indicating if the column names should be inferred automatically. (default: False)
     trial_columns: str | list[str] | None
         The name of the trial columns in the input data frame. If the list is empty or None,
         the input data frame is assumed to contain only one trial. If the list is not empty,
@@ -88,6 +86,8 @@ class GazeDataFrame:
         in the input data frame. If specified, the column will be used for pixel to dva
         transformations. If not specified, the constant eye-to-screen distance will be taken
         from the experiment definition. This column will be renamed to ``distance``. (default: None)
+    auto_column_detect: bool
+        Flag indicating if the column names should be inferred automatically. (default: False)
 
     Attributes
     ----------
@@ -210,7 +210,6 @@ class GazeDataFrame:
             experiment: Experiment | None = None,
             events: pm.EventDataFrame | None = None,
             *,
-            auto_column_detect: bool = False,
             trial_columns: str | list[str] | None = None,
             time_column: str | None = None,
             time_unit: str | None = None,
@@ -219,6 +218,7 @@ class GazeDataFrame:
             velocity_columns: list[str] | None = None,
             acceleration_columns: list[str] | None = None,
             distance_column: str | None = None,
+            auto_column_detect: bool = False,
     ):
         if data is None:
             data = pl.DataFrame()
@@ -1408,7 +1408,6 @@ class GazeDataFrame:
 
     def _init_columns(
             self,
-            auto_column_detect: bool = False,
             trial_columns: str | list[str] | None = None,
             time_column: str | None = None,
             time_unit: str | None = None,
@@ -1417,6 +1416,7 @@ class GazeDataFrame:
             velocity_columns: list[str] | None = None,
             acceleration_columns: list[str] | None = None,
             distance_column: str | None = None,
+            auto_column_detect: bool = False,
     ) -> None:
         """Initialize dataframe columns."""
         # Initialize trial_columns.
diff --git a/src/pymovements/gaze/io.py b/src/pymovements/gaze/io.py
index 8138fc0..31166da 100644
--- a/src/pymovements/gaze/io.py
+++ b/src/pymovements/gaze/io.py
@@ -42,6 +42,7 @@ def from_csv(
         velocity_columns: list[str] | None = None,
         acceleration_columns: list[str] | None = None,
         distance_column: str | None = None,
+        auto_column_detect: bool = False,
         column_map: dict[str, str] | None = None,
         add_columns: dict[str, str] | None = None,
         column_schema_overrides: dict[str, type] | None = None,
@@ -88,6 +89,8 @@ def from_csv(
         the column will be used for pixel to dva transformations. If not specified, the
         constant eye-to-screen distance will be taken from the experiment definition.
         (default: None)
+    auto_column_detect: bool
+        Flag indicating if the column names should be inferred automatically. (default: False)
     column_map: dict[str, str] | None
         The keys are the columns to read, the values are the names to which they should be renamed.
         (default: None)
@@ -118,13 +121,13 @@ def from_csv(
 
     The supported number of component columns with the expected order are:
 
-    * zero columns: No nested component column will be created.
-    * two columns: monocular data; expected order: x-component, y-component
-    * four columns: binocular data; expected order: x-component left eye, y-component left eye,
-      x-component right eye, y-component right eye,
-    * six columns: binocular data with additional cyclopian data; expected order: x-component
+    - **zero columns**: No nested component column will be created.
+    - **two columns**: monocular data; expected order: x-component, y-component
+    - **four columns**: binocular data; expected order: x-component left eye, y-component left eye,
+      x-component right eye, y-component right eye
+    - **six columns**: binocular data with additional cyclopian data; expected order: x-component
       left eye, y-component left eye, x-component right eye, y-component right eye,
-      x-component cyclopian eye, y-component cyclopian eye,
+      x-component cyclopian eye, y-component cyclopian eye
 
 
     Examples
@@ -264,6 +267,7 @@ def from_csv(
         velocity_columns=velocity_columns,
         acceleration_columns=acceleration_columns,
         distance_column=distance_column,
+        auto_column_detect=auto_column_detect,
     )
     return gaze_df
 


Test patch:
diff --git a/tests/unit/gaze/io/csv_test.py b/tests/unit/gaze/io/csv_test.py
index 93c76fd..059642e 100644
--- a/tests/unit/gaze/io/csv_test.py
+++ b/tests/unit/gaze/io/csv_test.py
@@ -37,6 +37,20 @@ import pymovements as pm
             (10, 2),
             id='csv_mono_shape',
         ),
+
+        pytest.param(
+            {
+                'file': 'tests/files/monocular_example.csv',
+                'column_map': {
+                    'x_left_pix': 'pixel_xl',
+                    'y_left_pix': 'pixel_yl',
+                },
+                'auto_column_detect': True,
+            },
+            (10, 2),
+            id='csv_mono_shape_auto_column_detect',
+        ),
+
         pytest.param(
             {
                 'file': 'tests/files/binocular_example.csv',
@@ -48,6 +62,26 @@ import pymovements as pm
             (10, 3),
             id='csv_bino_shape',
         ),
+
+        pytest.param(
+            {
+                'file': 'tests/files/binocular_example.csv',
+                'column_map': {
+                    'x_left_pix': 'pixel_xl',
+                    'y_left_pix': 'pixel_yl',
+                    'x_right_pix': 'pixel_xr',
+                    'y_right_pix': 'pixel_yr',
+                    'x_left_pos': 'position_xl',
+                    'y_left_pos': 'position_yl',
+                    'x_right_pos': 'position_xr',
+                    'y_right_pos': 'position_yr',
+                },
+                'auto_column_detect': True,
+            },
+            (10, 3),
+            id='csv_bino_shape_auto_column_detect',
+        ),
+
         pytest.param(
             {
                 'file': 'tests/files/hbn_example.csv',
@@ -59,6 +93,7 @@ import pymovements as pm
             (10, 2),
             id='hbn_dataset_example',
         ),
+
         pytest.param(
             {
                 'file': 'tests/files/sbsat_example.csv',
@@ -70,6 +105,7 @@ import pymovements as pm
             (10, 5),
             id='sbsat_dataset_example',
         ),
+
         pytest.param(
             {
                 'file': 'tests/files/gazebase_example.csv',
@@ -81,6 +117,7 @@ import pymovements as pm
             (10, 7),
             id='gazebase_dataset_example',
         ),
+
         pytest.param(
             {
                 'file': 'tests/files/gaze_on_faces_example.csv',
@@ -92,6 +129,7 @@ import pymovements as pm
             (10, 1),
             id='gaze_on_faces_dataset_example',
         ),
+
         pytest.param(
             {
                 'file': 'tests/files/gazebase_vr_example.csv',
@@ -102,6 +140,7 @@ import pymovements as pm
             (10, 11),
             id='gazebase_vr_dataset_example',
         ),
+
         pytest.param(
             {
                 'file': 'tests/files/judo1000_example.csv',