Repository: aeye-lab/pymovements
Pull request #219 (2023-03-01 11:33:01)

Dataset: Add subset of fileinfo columns to events dataframe
This now behaves consistently with the gaze dataframe which also has some of the fileinfo columns available (like subject_id)

codecov-commenter: # [Codecov](https://codecov.io/gh/aeye-lab/pymovements/pull/219?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.30`** :tada:
> Comparison is base [(`41e1474`)](https://codecov.io/gh/aeye-lab/pymovements/commit/41e147433bf98f7c8376f82eb934f2f3a048332c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) 80.99% compared to head [(`230a31f`)](https://codecov.io/gh/aeye-lab/pymovements/pull/219?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) 81.30%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #219      +/-   ##
==========================================
+ Coverage   80.99%   81.30%   +0.30%     
==========================================
  Files          21       21              
  Lines         842      845       +3     
  Branches      196      195       -1     
==========================================
+ Hits          682      687       +5     
+ Misses        139      138       -1     
+ Partials       21       20       -1     
```


| [Impacted Files](https://codecov.io/gh/aeye-lab/pymovements/pull/219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) | Coverage Δ | |
|---|---|---|
| [src/pymovements/datasets/dataset.py](https://codecov.io/gh/aeye-lab/pymovements/pull/219?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab#diff-c3JjL3B5bW92ZW1lbnRzL2RhdGFzZXRzL2RhdGFzZXQucHk=) | `95.61% <100.00%> (+0.94%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/aeye-lab/pymovements/pull/219?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).


Patch:
diff --git a/src/pymovements/datasets/dataset.py b/src/pymovements/datasets/dataset.py
index f5de4df..5c91ed3 100644
--- a/src/pymovements/datasets/dataset.py
+++ b/src/pymovements/datasets/dataset.py
@@ -311,7 +311,8 @@ class Dataset:
         gaze_dfs: list[pl.DataFrame] = []
 
         # Read gaze files from fileinfo attribute.
-        for file_id, filepath in enumerate(tqdm(self.fileinfo['filepath'])):
+        for fileinfo in tqdm(self.fileinfo.to_dicts()):
+            filepath = Path(fileinfo['filepath'])
             filepath = self.raw_rootpath / filepath
 
             if preprocessed:
@@ -327,21 +328,7 @@ class Dataset:
                 raise RuntimeError(f'data files of type {filepath.suffix} are not supported')
 
             # Add fileinfo columns to dataframe.
-            for column in self.fileinfo.columns[::-1]:
-                if column == 'filepath':
-                    continue
-
-                column_value = self.fileinfo.select(column)[file_id][0, 0]
-                gaze_df = gaze_df.select([
-                    pl.lit(column_value).alias(column),
-                    pl.all(),
-                ])
-
-            # Cast columns from fileinfo according to specification.
-            gaze_df = gaze_df.with_columns([
-                pl.col(fileinfo_key).cast(fileinfo_dtype)
-                for fileinfo_key, fileinfo_dtype in self._filename_regex_dtypes.items()
-            ])
+            gaze_df = self._add_fileinfo(gaze_df, fileinfo)
 
             gaze_dfs.append(gaze_df)
 
@@ -370,19 +357,22 @@ class Dataset:
         """
         self._check_fileinfo()
 
-        file_dfs: list[pl.DataFrame] = []
+        event_dfs: list[pl.DataFrame] = []
 
         # read and preprocess input files
-        for _, filepath in enumerate(tqdm(self.fileinfo['filepath'])):
-            filepath = self.raw_rootpath / Path(filepath)
+        for fileinfo in tqdm(self.fileinfo.to_dicts()):
+            filepath = Path(fileinfo['filepath'])
+            filepath = self.raw_rootpath / filepath
 
             filepath = self._raw_to_event_filepath(filepath, events_dirname=events_dirname)
+            event_df = pl.read_ipc(filepath)
 
-            file_df = pl.read_ipc(filepath)
+            # Add fileinfo columns to dataframe.
+            event_df = self._add_fileinfo(event_df, fileinfo)
 
-            file_dfs.append(file_df)
+            event_dfs.append(event_df)
 
-        return file_dfs
+        return event_dfs
 
     def pix2deg(self, verbose: bool = True) -> None:
         """Compute gaze positions in degrees of visual angle from pixel coordinates.
@@ -520,13 +510,17 @@ class Dataset:
 
         event_dfs: list[pl.DataFrame] = []
 
-        for gaze_df in tqdm(self.gaze, disable=disable_progressbar):
+        for gaze_df, fileinfo in tqdm(
+                zip(self.gaze, self.fileinfo.to_dicts()), disable=disable_progressbar,
+        ):
 
             positions = gaze_df.select(position_columns).to_numpy()
             velocities = gaze_df.select(velocity_columns).to_numpy()
 
-            events = method(positions=positions, velocities=velocities, **kwargs)
-            event_dfs.append(events)
+            event_df = method(positions=positions, velocities=velocities, **kwargs)
+            event_df = self._add_fileinfo(event_df, fileinfo)
+
+            event_dfs.append(event_df)
 
         if not self.events or clear:
             self.events = event_dfs
@@ -600,7 +594,7 @@ class Dataset:
                 if column in self.fileinfo.columns:
                     event_df.drop(column)
 
-            if verbose > 2:
+            if verbose >= 2:
                 print('Save file to', events_filepath)
 
             events_filepath.parent.mkdir(parents=True, exist_ok=True)
@@ -633,7 +627,7 @@ class Dataset:
                 if column in self.fileinfo.columns:
                     gaze_df = gaze_df.drop(column)
 
-            if verbose > 2:
+            if verbose >= 2:
                 print('Save file to', preprocessed_filepath)
 
             preprocessed_filepath.parent.mkdir(parents=True, exist_ok=True)
@@ -860,3 +854,34 @@ class Dataset:
         events_filename = raw_filepath.stem + '.feather'
 
         return events_file_dirpath / events_filename
+
+    def _add_fileinfo(self, df: pl.DataFrame, fileinfo: dict[str, Any]) -> pl.DataFrame:
+        """Add columns from fileinfo to dataframe.
+
+        Parameters
+        ----------
+        df : pl.DataFrame
+            Base dataframe to add fileinfo to.
+        fileinfo : dict[str, Any]
+            Dictionary of fileinfo row.
+
+        Returns
+        -------
+        pl.DataFrame:
+            Dataframe with added columns from fileinfo dictionary keys.
+        """
+
+        df = df.select(
+            [
+                pl.lit(value).alias(column)
+                for column, value in fileinfo.items()
+                if column != 'filepath'
+            ] + [pl.all()],
+        )
+
+        # Cast columns from fileinfo according to specification.
+        df = df.with_columns([
+            pl.col(fileinfo_key).cast(fileinfo_dtype)
+            for fileinfo_key, fileinfo_dtype in self._filename_regex_dtypes.items()
+        ])
+        return df
diff --git a/src/pymovements/events/engbert.py b/src/pymovements/events/engbert.py
index c28eba7..be0bed5 100644
--- a/src/pymovements/events/engbert.py
+++ b/src/pymovements/events/engbert.py
@@ -30,6 +30,7 @@ import polars as pl
 from pymovements.events.events import Saccade
 from pymovements.transforms import consecutive
 from pymovements.utils.checks import check_shapes_positions_velocities
+from pymovements.utils.filters import filter_candidates_remove_nans
 
 
 def microsaccades(
@@ -39,6 +40,7 @@ def microsaccades(
     threshold_factor: float = 6,
     minimum_duration: int = 6,
     minimum_threshold: float = 1e-10,
+    include_nan: bool = False,
 ) -> pl.DataFrame:
     """Detect micro-saccades from velocity gaze sequence.
 
@@ -66,6 +68,8 @@ def microsaccades(
     minimum_threshold : float
         minimal threshold value. Raises ValueError if calculated threshold is too low.
         Default: 1e-10
+    include_nan: bool
+        Indicator, whether we want to split events on missing/corrupt value (np.nan)
 
     Returns
     -------
@@ -100,13 +104,24 @@ def microsaccades(
     radius = threshold * threshold_factor
 
     # If value is greater than 1, point lies outside the ellipse.
-    outside_ellipse = np.greater(np.sum(np.power(velocities / radius, 2), axis=1), 1)
+    candidate_mask = np.greater(np.sum(np.power(velocities / radius, 2), axis=1), 1)
 
-    # Get all indices with velocities outside of ellipse.
-    outside_ellipse_indices = np.where(outside_ellipse)[0]
+    # Add nans to candidates if desired.
+    if include_nan:
+        candidate_mask = np.logical_or(candidate_mask, np.isnan(velocities).any(axis=1))
+
+    # Get indices of true values in candidate mask.
+    candidate_indices = np.where(candidate_mask)[0]
 
     # Get all saccade candidates by grouping all consecutive indices.
-    candidates = consecutive(arr=outside_ellipse_indices)
+    candidates = consecutive(arr=candidate_indices)
+
+    # Filter np.nan in candidates (delete starting/ending np.nans)
+    if include_nan:
+        candidates = filter_candidates_remove_nans(
+            candidates,
+            velocities,
+        )
 
     # Filter all candidates by minimum duration.
     candidates = [candidate for candidate in candidates if len(candidate) >= minimum_duration]
diff --git a/src/pymovements/events/idt.py b/src/pymovements/events/idt.py
index 24b3118..f7762d3 100644
--- a/src/pymovements/events/idt.py
+++ b/src/pymovements/events/idt.py
@@ -27,6 +27,8 @@ import polars as pl
 
 from pymovements.events.events import Fixation
 from pymovements.utils.checks import check_shapes_positions_velocities
+from pymovements.utils.filters import events_split_nans
+from pymovements.utils.filters import filter_candidates_remove_nans
 
 
 def dispersion(positions: list[list[float]] | np.ndarray) -> float:
@@ -46,7 +48,7 @@ def dispersion(positions: list[list[float]] | np.ndarray) -> float:
     dispersion: float
         Dispersion of the group of points.
     """
-    return sum(np.max(positions, axis=0) - np.min(positions, axis=0))
+    return sum(np.nanmax(positions, axis=0) - np.nanmin(positions, axis=0))
 
 
 def idt(
@@ -54,6 +56,7 @@ def idt(
         velocities: list[list[float]] | list[tuple[float, float]] | np.ndarray,
         dispersion_threshold: float,
         minimum_duration: int,
+        include_nan: bool = False,
 ) -> pl.DataFrame:
     """
     Fixation identification based on dispersion threshold.
@@ -77,6 +80,8 @@ def idt(
         Threshold for dispersion for a group of consecutive samples to be identified as fixation
     minimum_duration: int
         Minimum fixation duration in number of samples
+    include_nan: bool
+        Indicator, whether we want to split events on missing/corrupt value (np.nan)
 
     Returns
     -------
@@ -120,15 +125,44 @@ def idt(
                 if win_end == len(positions):
                     break
 
-            # Note a fixation at the centroid of the window points.
-            centroid = np.mean(positions[win_start:win_end - 1], axis=0)
-
-            fixations.append({
-                'type': 'fixation',
-                'onset': win_start,
-                'offset': win_end - 1,
-                'position': centroid.tolist(),
-            })
+            # check for np.nan values
+            if np.sum(np.isnan(positions[win_start:win_end - 1])) > 0:
+                tmp_candidates = [np.arange(win_start, win_end - 1, 1)]
+                tmp_candidates = filter_candidates_remove_nans(
+                    candidates=tmp_candidates,
+                    values=positions,
+                )
+                # split events if include_nan == False
+                if not include_nan:
+                    tmp_candidates = events_split_nans(
+                        candidates=tmp_candidates,
+                        values=positions,
+                    )
+
+                # Filter all candidates by minimum duration.
+                tmp_candidates = [
+                    candidate for candidate in tmp_candidates if len(candidate) >= minimum_duration
+                ]
+
+                for candidate in tmp_candidates:
+                    fixations.append({
+                        'type': 'fixation',
+                        'onset': candidate[0],
+                        'offset': candidate[-1],
+                        'position': np.nanmean(positions[candidate], axis=0).tolist(),
+                    })
+
+            else:
+                # Note a fixation at the centroid of the window points.
+                # the mean is computed using all values except np.nan
+                centroid = np.nanmean(positions[win_start:win_end - 1], axis=0)
+
+                fixations.append({
+                    'type': 'fixation',
+                    'onset': win_start,
+                    'offset': win_end - 1,
+                    'position': centroid.tolist(),
+                })
 
             # Remove window points from points.
             # Initialize new window excluding the previous window
diff --git a/src/pymovements/events/ivt.py b/src/pymovements/events/ivt.py
index 58df6b4..bc59cdc 100644
--- a/src/pymovements/events/ivt.py
+++ b/src/pymovements/events/ivt.py
@@ -29,6 +29,7 @@ from pymovements.events.events import Fixation
 from pymovements.transforms import consecutive
 from pymovements.transforms import norm
 from pymovements.utils.checks import check_shapes_positions_velocities
+from pymovements.utils.filters import filter_candidates_remove_nans
 
 
 def ivt(
@@ -36,6 +37,7 @@ def ivt(
         velocities: list[list[float]] | list[tuple[float, float]] | np.ndarray,
         velocity_threshold: float,
         minimum_duration: int,
+        include_nan: bool = False,
 ) -> pl.DataFrame:
     """
     Identification of fixations based on velocity-threshold
@@ -58,6 +60,8 @@ def ivt(
         velocity is below the threshold, the point is classified as a fixation.
     minimum_duration: int
         Minimum fixation duration in number of samples
+    include_nan: bool
+        Indicator, whether we want to split events on missing/corrupt value (np.nan)
 
     Returns
     -------
@@ -84,23 +88,38 @@ def ivt(
 
     # Get all indices with norm-velocities below threshold.
     velocity_norm = norm(velocities, axis=1)
-    below_threshold_indices = np.where(velocity_norm < velocity_threshold)[0]
+    candidate_mask = velocity_norm < velocity_threshold
+
+    # Add nans to candidates if desired.
+    if include_nan:
+        candidate_mask = np.logical_or(candidate_mask, np.isnan(velocities).any(axis=1))
+
+    # Get indices of true values in candidate mask.
+    candidate_ind = np.where(candidate_mask)[0]
 
     # Get all fixation candidates by grouping all consecutive indices.
-    candidates = consecutive(arr=below_threshold_indices)
+    candidates = consecutive(arr=candidate_ind)
+
+    # Filter np.nan in candidates (delete starting/ending np.nans)
+    if include_nan:
+        candidates = filter_candidates_remove_nans(
+            candidates,
+            velocities,
+        )
 
     # Filter all candidates by minimum duration.
     candidates = [candidate for candidate in candidates if len(candidate) >= minimum_duration]
 
-    # Create ficaitons from valid candidates. First channel is onset, second channel is offset.
+    # Create fixations from valid candidates. First channel is onset, second channel is offset.
     fixations = np.array([
         (candidate_indices[0], candidate_indices[-1])
         for candidate_indices in candidates
     ])
 
     # Calculate centroid positions for fixations.
+    # the mean is computed using all values except np.nan
     centroids = [
-        np.mean(positions[fixation[0]:fixation[1]], axis=0, dtype=np.float64).tolist()
+        np.nanmean(positions[fixation[0]:fixation[1]], axis=0, dtype=np.float64).tolist()
         for fixation in fixations
     ]
 
diff --git a/src/pymovements/synthetic/step_function.py b/src/pymovements/synthetic/step_function.py
index 435e5aa..bd14587 100644
--- a/src/pymovements/synthetic/step_function.py
+++ b/src/pymovements/synthetic/step_function.py
@@ -70,7 +70,7 @@ def step_function(
     ...     values=[1., 2., 3.],
     ...     start_value=0,
     ... )
-    array([0., 0., 1., 1., 1., 2., 2., 2., 2., 3.])
+    array([0., 0., 1., 1., 1., 2., 2., 2., 2., 3.], dtype=float32)
 
     >>> # multi-channel example
     >>> step_function(
@@ -88,7 +88,7 @@ def step_function(
            [ 3.,  4.],
            [ 3.,  4.],
            [ 3.,  4.],
-           [ 3.,  4.]])
+           [ 3.,  4.]], dtype=float32)
     """
     # Check that steps and values have equal length.
     if len(steps) != len(values):
@@ -136,6 +136,9 @@ def step_function(
     else:
         arr = np.tile(start_value, (length, 1))
 
+    # change type of elements to float32, to allow np.nan
+    arr = np.array(arr, dtype=np.float32)
+
     # Iterate through all steps except the last.
     for begin, end, value in zip(steps[:-1], steps[1:], values[:-1]):
         arr[begin:end] = value
diff --git a/src/pymovements/utils/filters.py b/src/pymovements/utils/filters.py
new file mode 100644
index 0000000..7039824
--- /dev/null
+++ b/src/pymovements/utils/filters.py
@@ -0,0 +1,101 @@
+# Copyright (c) 2023 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+This module holds filter specific funtions.
+"""
+from __future__ import annotations
+
+import numpy as np
+
+from pymovements.transforms import consecutive
+
+
+def filter_candidates_remove_nans(
+        candidates: list[np.ndarray],
+        values: np.ndarray,
+) -> list[np.ndarray]:
+    """
+    Filters a list of candidates for an event-detection algorithm
+
+    Removes leading and ending np.nans for all candidates in candidates
+
+    Parameters
+    ----------
+    candidates: list, list of candidates
+        List of candidates; each candidate consists of a list of indices
+    values: array-like, shape (N, 1) or shape (N, 2)
+        Corresponding continuous 1D/2D values time series.
+
+    Returns
+    -------
+    list
+        a filtered list of candidates
+    """
+    values = np.array(values)
+    return_candidates = []
+    for candidate in candidates:
+        if len(candidate) == 0:
+            continue
+        cand_values = values[np.array(candidate)]
+        start_id = 0
+        while np.sum(np.isnan(cand_values[start_id, :])) > 0:
+            start_id += 1
+        end_id = len(cand_values) - 1
+        while np.sum(np.isnan(cand_values[end_id, :])) > 0:
+            end_id -= 1
+        cur_candidate = list(candidate[start_id:end_id + 1])
+        return_candidates.append(np.array(cur_candidate))
+    return return_candidates
+
+
+def events_split_nans(
+        candidates: list[np.ndarray],
+        values: np.ndarray,
+) -> list[np.ndarray]:
+    """
+    Filters a list of candidates for an event-detection algorithm
+
+    Splits events if np.nans are within an event
+
+    Parameters
+    ----------
+    candidates: list, list of candidates
+        List of candidates; each candidate consists of a list of indices
+    values: array-like, shape (N, 1) or shape (N, 2)
+        Corresponding continuous 1D/2D values time series.
+
+    Returns
+    -------
+    list
+        a filtered list of candidates
+    """
+    values = np.array(values)
+    return_candidates = []
+    for candidate in candidates:
+        if len(candidate) == 0:
+            continue
+        cur_values = values[np.array(candidate)]
+        nan_candidates = consecutive(arr=np.where(~np.isnan(np.sum(cur_values, axis=1)))[0])
+        cand_list = [
+            np.array(candidate[candidate_indices[0]:candidate_indices[-1] + 1])
+            for candidate_indices in nan_candidates
+        ]
+        return_candidates += cand_list
+    return return_candidates


Test patch:
diff --git a/tests/datasets/test_dataset.py b/tests/datasets/test_dataset.py
index 1523649..2898d9d 100644
--- a/tests/datasets/test_dataset.py
+++ b/tests/datasets/test_dataset.py
@@ -157,11 +157,12 @@ def mock_toy(rootpath, raw_fileformat='csv'):
     for fileinfo_row in fileinfo.to_dicts():
         event_df = pl.from_dict(
             {
+                'subject_id': fileinfo_row['subject_id'],
                 'type': 'saccade',
                 'onset': np.arange(0, 901, 100),
                 'offset': np.arange(10, 911, 100),
             },
-            schema={'type': pl.Utf8, 'onset': pl.Int64, 'offset': pl.Int64},
+            schema={'subject_id': pl.Int64, 'type': pl.Utf8, 'onset': pl.Int64, 'offset': pl.Int64},
         )
         event_dfs.append(event_df)
 
@@ -417,8 +418,9 @@ def test_detect_events(detect_event_kwargs, dataset_configuration):
     dataset.pos2vel()
     dataset.detect_events(**detect_event_kwargs)
 
+    expected_schema = {'subject_id': pl.Int64, **Saccade.schema}
     for result_event_df in dataset.events:
-        assert result_event_df.schema == Saccade.schema
+        assert result_event_df.schema == expected_schema
 
 
 @pytest.mark.parametrize(
@@ -435,7 +437,7 @@ def test_detect_events(detect_event_kwargs, dataset_configuration):
                 'threshold': 1,
                 'eye': 'auto',
             },
-            Saccade.schema,
+            {'subject_id': pl.Int64, **Saccade.schema},
             id='two-saccade-runs',
         ),
         pytest.param(
@@ -449,7 +451,7 @@ def test_detect_events(detect_event_kwargs, dataset_configuration):
                 'velocity_threshold': 1,
                 'minimum_duration': 1,
             },
-            {**Saccade.schema, **Fixation.schema},
+            {'subject_id': pl.Int64, **Saccade.schema, **Fixation.schema},
             id='one-saccade-one-fixation-run',
         ),
     ],
@@ -595,6 +597,12 @@ def test_save_preprocessed(preprocessed_dirname, expected_save_dirpath, dataset_
             {},
             id='none_dirname',
         ),
+        pytest.param(
+            'preprocessed',
+            'events',
+            {'verbose': 2},
+            id='verbose_equals_2',
+        ),
         pytest.param(
             'preprocessed_test',
             'events',
@@ -628,7 +636,7 @@ def test_save(
     shutil.rmtree(dataset.path / Path(expected_save_preprocessed_path), ignore_errors=True)
     shutil.rmtree(dataset.path / Path(events_dirname), ignore_errors=True)
     shutil.rmtree(dataset.path / Path(expected_save_events_path), ignore_errors=True)
-    dataset.save(preprocessed_dirname, events_dirname)
+    dataset.save(**save_kwargs)
 
     assert (dataset.path / Path(expected_save_preprocessed_path)).is_dir(), (
         f'data was not written to {dataset.path / Path(expected_save_events_path)}'
diff --git a/tests/events/test_engbert.py b/tests/events/test_engbert.py
index f31be5e..8786039 100644
--- a/tests/events/test_engbert.py
+++ b/tests/events/test_engbert.py
@@ -125,6 +125,50 @@ def test_microsaccades_raises_error(kwargs, expected):
             ),
             id='four_steps_two_saccades',
         ),
+        pytest.param(
+            {
+                'positions': step_function(length=100, steps=[0], values=[(0, 0)]),
+                'velocities': step_function(
+                    length=100,
+                    steps=[20, 25, 28, 30, 70, 80],
+                    values=[(9, 9), (np.nan, np.nan), (9, 9), (0, 0), (9, 9), (0, 0)],
+                    start_value=(0, 0),
+                ),
+                'threshold': 1,
+                'include_nan': True,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'saccade',
+                    'onset': [20, 70],
+                    'offset': [29, 79],
+                },
+                schema=Saccade.schema,
+            ),
+            id='two_saccades_nan_delete_ending_leading_nan',
+        ),
+        pytest.param(
+            {
+                'positions': step_function(length=100, steps=[0], values=[(0, 0)]),
+                'velocities': step_function(
+                    length=100,
+                    steps=[20, 25, 28, 30, 70, 80],
+                    values=[(9, 9), (np.nan, np.nan), (9, 9), (0, 0), (9, 9), (0, 0)],
+                    start_value=(0, 0),
+                ),
+                'threshold': 1,
+                'minimum_duration': 1,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'saccade',
+                    'onset': [20, 28, 70],
+                    'offset': [24, 29, 79],
+                },
+                schema=Saccade.schema,
+            ),
+            id='three_saccades_nan_delete_ending_leading_nan',
+        ),
     ],
 )
 def test_microsaccades_detects_saccades(kwargs, expected):
diff --git a/tests/events/test_idt.py b/tests/events/test_idt.py
index 216ddb7..af3450f 100644
--- a/tests/events/test_idt.py
+++ b/tests/events/test_idt.py
@@ -181,6 +181,53 @@ def test_idt_raises_error(kwargs, expected_error):
             ),
             id='three_steps_two_fixations',
         ),
+        pytest.param(
+            {
+                'positions': step_function(
+                    length=100, steps=[10, 20, 90],
+                    values=[
+                        (np.nan, np.nan), (0, 0),
+                        (np.nan, np.nan),
+                    ],
+                ),
+                'dispersion_threshold': 1,
+                'minimum_duration': 1,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'fixation',
+                    'onset': [0, 20],
+                    'offset': [9, 89],
+                    'position': [(0.0, 0.0), (0.0, 0.0)],
+                },
+                schema=Fixation.schema,
+            ),
+            id='two_fixations_nan_delete_leading_ending',
+        ),
+        pytest.param(
+            {
+                'positions': step_function(
+                    length=100, steps=[10, 20, 90],
+                    values=[
+                        (np.nan, np.nan), (0, 0),
+                        (np.nan, np.nan),
+                    ],
+                ),
+                'dispersion_threshold': 1,
+                'minimum_duration': 1,
+                'include_nan': True,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'fixation',
+                    'onset': [0],
+                    'offset': [89],
+                    'position': [(0.0, 0.0)],
+                },
+                schema=Fixation.schema,
+            ),
+            id='one_fixation_nan_delete_leading_ending',
+        ),
     ],
 )
 def test_idt_detects_fixations(kwargs, expected):
diff --git a/tests/events/test_ivt.py b/tests/events/test_ivt.py
index bc80ef7..6cb755f 100644
--- a/tests/events/test_ivt.py
+++ b/tests/events/test_ivt.py
@@ -213,6 +213,53 @@ def test_ivt_raise_error(kwargs, expected_error):
             ),
             id='three_steps_two_fixations',
         ),
+        pytest.param(
+            {
+                'positions': step_function(
+                    length=100, steps=[10, 20, 90],
+                    values=[
+                        (np.nan, np.nan),
+                        (0, 0), (np.nan, np.nan),
+                    ],
+                ),
+                'velocity_threshold': 1,
+                'minimum_duration': 1,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'fixation',
+                    'onset': [0, 21],
+                    'offset': [9, 89],
+                    'position': [(0.0, 0.0), (0.0, 0.0)],
+                },
+                schema=Fixation.schema,
+            ),
+            id='two_fixations_nan_remove_leading_ending',
+        ),
+        pytest.param(
+            {
+                'positions': step_function(
+                    length=100, steps=[10, 20, 90],
+                    values=[
+                        (np.nan, np.nan),
+                        (0, 0), (np.nan, np.nan),
+                    ],
+                ),
+                'velocity_threshold': 1,
+                'minimum_duration': 1,
+                'include_nan': True,
+            },
+            pl.DataFrame(
+                {
+                    'type': 'fixation',
+                    'onset': [0],
+                    'offset': [89],
+                    'position': [(0.0, 0.0)],
+                },
+                schema=Fixation.schema,
+            ),
+            id='one_fixations_nan_remove_leading_ending',
+        ),
     ],
 )
 def test_ivt_detects_fixations(kwargs, expected):
diff --git a/tests/synthetic/test_step_function.py b/tests/synthetic/test_step_function.py
index 09940c1..5774cbd 100644
--- a/tests/synthetic/test_step_function.py
+++ b/tests/synthetic/test_step_function.py
@@ -99,6 +99,11 @@ from pymovements.synthetic import step_function
             {'exception': ValueError},
             id='negative_noise_raises_value_error',
         ),
+        pytest.param(
+            {'length': 4, 'steps': [2], 'values': [(np.nan, np.nan)], 'start_value': (0, 0)},
+            {'dimension': (4, 2)},
+            id='length_5_2_nan_value',
+        ),
     ],
 )
 def test_step_function(params, expected):
@@ -106,6 +111,10 @@ def test_step_function(params, expected):
         with pytest.raises(expected['exception']):
             step_function(**params)
         return
+    if 'dimension' in expected:
+        arr = step_function(**params)
+        assert expected['dimension'] == arr.shape
+        return
 
     arr = step_function(**params)
     assert np.array_equal(arr, expected['value']), f"arr = {arr}, expected = {expected['value']}"
diff --git a/tests/utils/test_filters.py b/tests/utils/test_filters.py
new file mode 100644
index 0000000..2c5ea91
--- /dev/null
+++ b/tests/utils/test_filters.py
@@ -0,0 +1,83 @@
+# Copyright (c) 2023 The pymovements Project Authors
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+Test pymovements filters.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pytest
+
+from pymovements.utils.filters import events_split_nans
+from pymovements.utils.filters import filter_candidates_remove_nans
+
+
+@pytest.mark.parametrize(
+    'params, expected',
+    [
+        pytest.param(
+            {
+                'candidates': [
+                    [0, 1, 2, 3, 4],
+                    [5, 6, 7, 8],
+                ],
+                'values': np.array([
+                    (np.nan, np.nan), (0, 0),
+                    (0, 0), (0, 0),
+                    (np.nan, np.nan),
+                    (np.nan, np.nan),
+                    (0, 0), (0, 0),
+                    (0, 0),
+                ]),
+            },
+            {'values_filter': [np.array([1, 2, 3]), np.array([6, 7, 8])]},
+            id='test_filters',
+        ),
+        pytest.param(
+            {
+                'candidates': [[0, 1, 2, 3, 4, 5, 6, 7]],
+                'values': np.array([
+                    (0, 0),
+                    (0, 0), (0, 0),
+                    (np.nan, np.nan),
+                    (np.nan, np.nan),
+                    (0, 0), (0, 0),
+                    (0, 0),
+                ]),
+            },
+            {'values_split': [np.array([0, 1, 2]), np.array([5, 6, 7])]},
+            id='test_events_split',
+        ),
+    ],
+)
+def test_filters(params, expected):
+    if 'values_filter' in expected:
+        results = filter_candidates_remove_nans(
+            params['candidates'],
+            params['values'],
+        )
+        assert np.all(np.array(expected['values_filter']) == np.array(results))
+
+    if 'values_split' in expected:
+        results = events_split_nans(
+            params['candidates'],
+            params['values'],
+        )
+        assert np.all(np.array(expected['values_split']) == np.array(results))