Repository: aeye-lab/pymovements
Pull request #746 (2024-07-05 16:22:09)

fix: cast numerical columns to Float64 when reading CSV
## Description

When a CSV file starts with >= 100 rows of missing values for the pixel, position, or other column, Polars will infer a `str` dtype for this column (see [this comment](https://github.com/pola-rs/polars/issues/12473#issuecomment-1814012038). This can happen if a recording starts with a blink, or some other kind of data loss. As a result, functions like `pix2deg()` fail.

## Implemented changes

- [x] When reading CSV, cast columns that *should* be numerical but *could* contain missing values (pixel, position, velocity, acceleration, distance) to Float64

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] This change is or requires a documentation update

## How Has This Been Tested?

- [x] Added a unit test to test dtypes in `tests/unit/gaze/io/csv_test.py` and added `tests/files/missing_values_example.csv` with missing values

## Checklist:

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged and published in downstream modules
- [x] I have checked my code and corrected any misspellings


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/aeye-lab/pymovements/pull/746?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 100.00%. Comparing base [(`6aaf84d`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/6aaf84def5eff46b17e7b32dffa2a6c06514dffb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab) to head [(`1ee511a`)](https://app.codecov.io/gh/aeye-lab/pymovements/commit/1ee511afbf2d79dd858d98ea34190aaea7127ec5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main      #746   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           60        60           
  Lines         2763      2766    +3     
  Branches       715       716    +1     
=========================================
+ Hits          2763      2766    +3     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/aeye-lab/pymovements/pull/746?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=aeye-lab).


Patch:
diff --git a/src/pymovements/gaze/io.py b/src/pymovements/gaze/io.py
index 78979ca..f770feb 100644
--- a/src/pymovements/gaze/io.py
+++ b/src/pymovements/gaze/io.py
@@ -228,6 +228,21 @@ def from_csv(
             if column not in gaze_data.columns
         ])
 
+    # Cast numerical columns to Float64 if they were incorrectly inferred to be Utf8.
+    # This can happen if the column only has missing values in the top 100 rows.
+    numerical_columns = (
+        (pixel_columns or [])
+        + (position_columns or [])
+        + (velocity_columns or [])
+        + (acceleration_columns or [])
+        + ([distance_column] if distance_column else [])
+    )
+    for column in numerical_columns:
+        if gaze_data[column].dtype == pl.Utf8:
+            gaze_data = gaze_data.with_columns([
+                pl.col(column).cast(pl.Float64),
+            ])
+
     if column_dtypes is not None:
         gaze_data = gaze_data.with_columns([
             pl.col(fileinfo_key).cast(fileinfo_dtype)


Test patch:
diff --git a/tests/files/missing_values_example.csv b/tests/files/missing_values_example.csv
new file mode 100644
index 0000000..78c2359
--- /dev/null
+++ b/tests/files/missing_values_example.csv
@@ -0,0 +1,104 @@
+time,pixel_x,pixel_y,position_x,position_y
+0,,
+1,,
+2,,
+3,,
+4,,
+5,,
+6,,
+7,,
+8,,
+9,,
+10,,
+11,,
+12,,
+13,,
+14,,
+15,,
+16,,
+17,,
+18,,
+19,,
+20,,
+21,,
+22,,
+23,,
+24,,
+25,,
+26,,
+27,,
+28,,
+29,,
+30,,
+31,,
+32,,
+33,,
+34,,
+35,,
+36,,
+37,,
+38,,
+39,,
+40,,
+41,,
+42,,
+43,,
+44,,
+45,,
+46,,
+47,,
+48,,
+49,,
+50,,
+51,,
+52,,
+53,,
+54,,
+55,,
+56,,
+57,,
+58,,
+59,,
+60,,
+61,,
+62,,
+63,,
+64,,
+65,,
+66,,
+67,,
+68,,
+69,,
+70,,
+71,,
+72,,
+73,,
+74,,
+75,,
+76,,
+77,,
+78,,
+79,,
+80,,
+81,,
+82,,
+83,,
+84,,
+85,,
+86,,
+87,,
+88,,
+89,,
+90,,
+91,,
+92,,
+93,,
+94,,
+95,,
+96,,
+97,,
+98,,
+99,,
+100,,
+101,1,2,1.0,2.0
+101,3,4,3.0,4.0
diff --git a/tests/unit/gaze/io/csv_test.py b/tests/unit/gaze/io/csv_test.py
index 736c320..d24a645 100644
--- a/tests/unit/gaze/io/csv_test.py
+++ b/tests/unit/gaze/io/csv_test.py
@@ -18,6 +18,7 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 """Test read from csv."""
+import polars as pl
 import pytest
 
 import pymovements as pm
@@ -117,3 +118,34 @@ import pymovements as pm
 def test_shapes(kwargs, shape):
     gaze_dataframe = pm.gaze.from_csv(**kwargs)
     assert gaze_dataframe.frame.shape == shape
+
+
+@pytest.mark.parametrize(
+    ('kwargs', 'dtypes'),
+    [
+        pytest.param(
+            {
+                'file': 'tests/files/monocular_example.csv',
+                'time_column': 'time',
+                'time_unit': 'ms',
+                'pixel_columns': ['x_left_pix', 'y_left_pix'],
+            },
+            [pl.Int64, pl.List(pl.Int64)],
+            id='csv_mono_dtypes',
+        ),
+        pytest.param(
+            {
+                'file': 'tests/files/missing_values_example.csv',
+                'time_column': 'time',
+                'time_unit': 'ms',
+                'pixel_columns': ['pixel_x', 'pixel_y'],
+                'position_columns': ['position_x', 'position_y'],
+            },
+            [pl.Int64, pl.List(pl.Float64), pl.List(pl.Float64)],
+            id='csv_missing_values_dtypes',
+        ),
+    ],
+)
+def test_dtypes(kwargs, dtypes):
+    gaze_dataframe = pm.gaze.from_csv(**kwargs)
+    assert gaze_dataframe.frame.dtypes == dtypes