Repository: aeye-lab/pymovements (python)

Issue:
SchemaFieldNotFoundError When Loading Preprocessed Data
## Current Behavior

When loading preprocessed data using the `Dataset.load()` method with the `preprocessed=True` flag, I encounter a `SchemaFieldNotFoundError` related to the `trialId` column. The preprocessed data already has renamed columns, and the error occurs because the renaming operation is performed again, regardless of whether the data is preprocessed or not.

## Expected Behavior

The preprocessed data should be successfully loaded without any errors.

## Minimum Acceptance Criteria

- [ ] Ensure the `dataset.load(preprocessed=True)` can correctly load preprocessed data without column renaming issues.

## Failure Information (for bugs)

### Steps to Reproduce

1. Load the JuDo1000 public dataset.
2. Perform preprocessing steps like `pix2deg` and `pos2vel`.
3. Save the preprocessed data using `dataset.save()`.
4. Attempt to reload the saved preprocessed data using `dataset.load(preprocessed=True)`.

### Code to Reproduce

```python
dataset = pm.Dataset('JuDo1000', path='/mnt/scratch/chwastek/datasets/Judo1000/')
dataset.download()
dataset.load(subset={'subject_id': 1})

dataset.pix2deg()
dataset.pos2vel(method='savitzky_golay', window_length=7, degree=2)

dataset.save()
dataset.load(preprocessed=True)
```

### Error Log

```python-traceback
---------------------------------------------------------------------------
SchemaFieldNotFoundError                  Traceback (most recent call last)
Cell In[33], line 9
      6 dataset.pos2vel(method='savitzky_golay', window_length=7, degree=2)
      8 dataset.save()
----> 9 dataset.load(preprocessed=True)

File /mnt/scratch/chwastek/anaconda3/lib/python3.11/site-packages/pymovements/dataset/dataset.py:122, in Dataset.load(self, events, preprocessed, subset, events_dirname, preprocessed_dirname, extension)
    119 self.scan()
    120 self.fileinfo = dataset_files.take_subset(fileinfo=self.fileinfo, subset=subset)
--> 122 self.load_gaze_files(
    123     preprocessed=preprocessed, preprocessed_dirname=preprocessed_dirname,
    124     extension=extension,
    125 )
    127 if events:
    128     self.load_event_files(
    129         events_dirname=events_dirname,
    130         extension=extension,
    131     )

File /mnt/scratch/chwastek/anaconda3/lib/python3.11/site-packages/pymovements/dataset/dataset.py:187, in Dataset.load_gaze_files(self, preprocessed, preprocessed_dirname, extension)
    159 """Load all available gaze data files.
    160 
    161 Parameters
   (...)
    184     If file type of gaze file is not supported.
    185 """
    186 self._check_fileinfo()
--> 187 self.gaze = dataset_files.load_gaze_files(
    188     definition=self.definition,
    189     fileinfo=self.fileinfo,
    190     paths=self.paths,
    191     preprocessed=preprocessed,
    192     preprocessed_dirname=preprocessed_dirname,
    193     extension=extension,
    194 )
    195 return self

File /mnt/scratch/chwastek/anaconda3/lib/python3.11/site-packages/pymovements/dataset/dataset_files.py:216, in load_gaze_files(definition, fileinfo, paths, preprocessed, preprocessed_dirname, extension)
    205     filepath = paths.get_preprocessed_filepath(
    206         filepath, preprocessed_dirname=preprocessed_dirname,
    207         extension=extension,
    208     )
    210 gaze_data = load_gaze_file(
    211     filepath=filepath,
    212     preprocessed=preprocessed,
    213     custom_read_kwargs=definition.custom_read_kwargs,
    214 )
--> 216 gaze_data = gaze_data.rename(definition.column_map)
    218 # Add fileinfo columns to dataframe.
    219 gaze_data = add_fileinfo(
    220     definition=definition,
    221     df=gaze_data,
    222     fileinfo=fileinfo_row,
    223 )
    .......
    SchemaFieldNotFoundError: trialId
```

### Relevant Code Snippets

The datasets `DatasetDefinition` describes a `column_map` for renaming columns in the raw input data.

```python
column_map: dict[str, str] = field(
    default_factory=lambda: {
        'trialId': 'trial_id',
        'pointId': 'point_id',
    },
)
```

I tested by running the code with removing the column map, and it worked:

```python
dataset = pm.Dataset('JuDo1000', path='/mnt/scratch/chwastek/datasets/Judo1000/')
dataset.definition.column_map = {}
dataset.load(preprocessed=True)
```

## Context

- Project Version / Commit: 0.17.0
- Operating System: Linux

## Checklist

- [x] I am running the latest version
- [x] I checked the documentation and found no answer
- [x] I checked to make sure that this issue has not already been filed
- [x] I have provided sufficient information for the team


Pull request:
fix: SchemaFieldNotFoundError When Loading Preprocessed Data
Fixes #607

Changes:

- Only rename input columns by `column_map` when loading raw data
- Test for loading preprocessed dataset. 

Working with the example files currently present in `tests/files` I had to adjust some attributes of the pupil datasets `DatasetDefinition`.

Patch:
diff --git a/src/pymovements/dataset/dataset_files.py b/src/pymovements/dataset/dataset_files.py
index 0bdd4e5..e61f411 100644
--- a/src/pymovements/dataset/dataset_files.py
+++ b/src/pymovements/dataset/dataset_files.py
@@ -213,7 +213,8 @@ def load_gaze_files(
             custom_read_kwargs=definition.custom_read_kwargs,
         )
 
-        gaze_data = gaze_data.rename(definition.column_map)
+        if not preprocessed:
+            gaze_data = gaze_data.rename(definition.column_map)
 
         # Add fileinfo columns to dataframe.
         gaze_data = add_fileinfo(


Test patch:
diff --git a/tests/functional/dataset_processing_test.py b/tests/functional/dataset_processing_test.py
new file mode 100644
index 0000000..da25f0f
--- /dev/null
+++ b/tests/functional/dataset_processing_test.py
@@ -0,0 +1,117 @@
+# Copyright (c) 2023 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""Test basic preprocessing on various datasets."""
+
+import shutil
+import pytest
+
+import pymovements as pm
+
+
+@pytest.fixture(
+    name='datasets',
+    params=[
+        'csv_monocular',
+        'csv_binocular',
+        'ipc_monocular',
+        'ipc_binocular',
+        'hbn',
+        'sbsat',
+        'gaze_on_faces',
+        'gazebase',
+        'gazebase_vr',
+        'judo1000',
+    ],
+)
+def fixture_dataset_init_kwargs(request):
+    init_param_dict = {
+        'csv_monocular':pm.dataset.DatasetDefinition(
+            time_column='time',
+            pixel_columns=['x_left_pix', 'y_left_pix'],
+            experiment=pm.Experiment(1024, 768, 38, 30, 60, 'center', 1000),
+            filename_format='monocular_example.csv',
+        ),
+        'csv_binocular': pm.dataset.DatasetDefinition(
+            filename_format='binocular_example.csv',
+            time_column='time',
+            pixel_columns=['x_left_pix', 'y_left_pix', 'x_right_pix', 'y_right_pix'],
+            position_columns=['x_left_pos', 'y_left_pos', 'x_right_pos', 'y_right_pos'],
+            experiment=pm.Experiment(1024, 768, 38, 30, 60, 'center', 1000),
+        ),
+        'ipc_monocular': pm.dataset.DatasetDefinition(
+            filename_format='monocular_example.feather',
+            experiment=pm.Experiment(1024, 768, 38, 30, 60, 'center', 1000),
+        ),
+        'ipc_binocular': pm.dataset.DatasetDefinition(
+            filename_format='binocular_example.feather',
+            experiment=pm.Experiment(1024, 768, 38, 30, 60, 'center', 1000),
+        ),
+        'hbn': pm.datasets.HBN(
+            filename_format='hbn_example.csv',
+            filename_format_dtypes={},
+            trial_columns=None
+        ),
+        'sbsat': pm.datasets.SBSAT(
+            filename_format='sbsat_example.csv',
+            filename_format_dtypes={},
+
+        ),
+        'gaze_on_faces': pm.datasets.GazeOnFaces(
+            filename_format='gaze_on_faces_example.csv',
+            filename_format_dtypes={},
+            trial_columns=None
+        ),
+        'gazebase': pm.datasets.GazeBase(
+            filename_format='gazebase_example.csv',
+            filename_format_dtypes={},
+            trial_columns=None
+        ),
+        'gazebase_vr': pm.datasets.GazeBaseVR(
+            filename_format='gazebase_vr_example.csv',
+            filename_format_dtypes={},
+            trial_columns=None,
+        ),
+        'judo1000': pm.datasets.JuDo1000(
+            filename_format='judo1000_example.csv',
+            filename_format_dtypes={},
+            trial_columns=['trial_id'],
+        )
+
+    }
+    yield pm.dataset.Dataset(
+        definition=init_param_dict[request.param],
+        path=pm.dataset.DatasetPaths(
+            root='tests',
+            dataset='.',
+            raw='files',
+        ),
+    )
+
+
+def test_dataset_save_load_preprocessed(datasets):
+    dataset = datasets
+    dataset.load()
+
+    if 'pixel' in dataset.gaze[0].frame.columns:
+        dataset.pix2deg()
+
+    dataset.pos2vel()
+    dataset.save()
+    dataset.load(preprocessed=True)