Repository: mne-tools/mne-python

Issue:
filter.py _1d_overlap_filter DeprecationWarning on dtype conversion
Hi all. I am getting the following WARN:
filter.py:175: DeprecationWarning: Implicitly casting between incompatible kinds. In a future numpy release, this will raise an error. Use casting="unsafe" if this is intentional. 
x_filtered[seg_idx \* n_seg:seg_idx \* n_seg + n_fft] += prod
AND
filter.py:178: DeprecationWarning: Implicitly casting between incompatible kinds. In a future numpy release, this will raise an error. Use casting="unsafe" if this is intentional.
x_filtered[seg_idx \* n_seg:] += prod[:n_x - seg_idx \* n_seg]

Please note that I call two filter functions, band_stop_filter and high_pass_filter, in a loop, but I only see the two DeprecationWarning print outs once each. I did some looking into this and it seems that x_filtered is and array of 16-bit integers and prod is an array of 64-bit floating point numbers. I also see that x_filtered is cast above to be an array of zeros with the dtype of the filter input. If I set cast my input arrays of 16-bit integers to a larger array of 64-bit fps, then I don't see any of the warnings.

What's the intended behavior? Should x_filtered always be explicitly up cast to np.float64?



Discussion:
Good catch, that's a bug .

These are mostly designed for internal use. To be safest I'd prefer to
throw an error if not double, so that the user must explicitly cast (and
thus understand what is being done). I can live with automatic casting if
others like it.

I would force the dtype to be float64

I have a fix in place that upcasts inside _1d_overlap_filter.  However maybe a more appropriate place to upcast input or raise an error would be _prep_for_filtering. What do you think? Do all arrays passed for filtering need to be of dtype np.float64?

> I have a fix in place that upcasts inside _1d_overlap_filter. However maybe a more appropriate place to upcast input or raise an error would be _prep_for_filtering. What do you think?
> 
> +1
> Do all arrays passed for filtering need to be of dtype np.float64?
> 
> Yes
> —
> Reply to this email directly or view it on GitHub.

Pull request: https://github.com/mne-tools/mne-python/pull/1615

This means we need something like `raw._data = raw._data.astype('d')` before filtering, right? At least this is what I have to do here..

Any chance we could pass an `upcast=True` to filter and it can handle the conversions inside the loop over channels (to avoid doubling mem usage)?

> This means we need something like raw._data = raw._data.astype('d') before filtering, right? At least this is what I have to do here..
> 
> Any chance we could pass an upcast=True to filter and it can handle the conversions inside the loop over channels (to avoid doubling mem usage)?

you work with float32?

how come have you raw data in memory in float32?

I admit I was also surprised, but it's raw data from the read_raw_bti function.

@dengemann ?

FWIW I haven't had any problems with 4D data due to single precision.. routine filtering, ICA, etc. works fine.

hum... I assume you would be against casting bti data to double by default?

@dengemann what's your take on this?

> against casting bti data to double by default

No, I was just surprised when the filter refused to run (after a `git pull`).

@maedoc did you directly work on the data in memory after converting? If you save them first to disk and reopen they will be read as float64. I here followed the general fiff io pattern.

... or there's something I'm missing. At least all of my BTI files are float64 on reading from fiff.

@maedoc the raw reading code uses np.float as default dtype. This is expands into np.float64. I'm therefore a bit surprised how you can possible end up with single precision raw bti data in memory.

I usually load from 4D file, do some preprocessing before saving to fif, in which case it's definitely float32. After saving to fif & loading, it's float64.

> raw reading code uses np.float as default dtype

I see it reads [`astype('f4')`](https://github.com/mne-tools/mne-python/blob/master/mne/io/bti/bti.py#L922), so maybe you're referring to fif reader?

> After saving to fif & loading, it's float64.

good.

> I see it reads astype('f4'), so maybe you're referring to fif reader?

one level higher. see e.g. https://github.com/mne-tools/mne-python/blob/master/mne/io/fiff/raw.py#L496

RawBTi doesn't inherit from RawFIFF, so that doesn't apply to results of read_raw_bti, whose data are effectively preloaded, because `self._data` attr is created from the float32 array returned by bti._read_data.

> RawBTi doesn't inherit from RawFIFF

yes, exactly. This is on purpose. RawFIFF is starts with reading a fiff file. The other classes are to write a fiff file.

Then I guess I'm puzzled why these classes bother to have filter, crop etc. if they are just for conversion. Anyways, we answered the question of where float32 came from, I'll make sure to upcast as needed.

> Then I guess I'm puzzled why these classes bother to have filter, crop etc. if they are just for conversion.

sorry I don't get this comment.

they inherit from baseraw. 

I figured that all readers would up the dtype to `float64` like `RawFIF` does. We should either have them all read in as `float64`, or not. Otherwise there would have been no point in making `RawFIF` do recasting in the first place... @dengemann if you're okay with it, I'll change the reader to convert the data to `float64`.


Resolving patch:
diff --git a/mne/io/base.py b/mne/io/base.py
--- a/mne/io/base.py
+++ b/mne/io/base.py
@@ -223,6 +223,11 @@ def __init__(self, info, data=None,
                  filenames=(), rawdirs=(),
                  comp=None, orig_comp_grade=None,
                  orig_format='double', verbose=None):
+        # some functions (e.g., filtering) only work w/64-bit data
+        if data is not None:
+            if data.dtype not in (np.float64, np.complex128):
+                raise RuntimeError('datatype must be float64 or complex128, '
+                                   'not %s' % data.dtype)
         self.info = info
         self._data = data
         cals = np.empty(info['nchan'])
diff --git a/mne/io/bti/bti.py b/mne/io/bti/bti.py
--- a/mne/io/bti/bti.py
+++ b/mne/io/bti/bti.py
@@ -896,6 +896,7 @@ def _read_data(info, start=None, stop=None):
     -------
     data : ndarray
         The measurement data, a channels x time slices array.
+        The data will be cast to np.float64 for compatibility.
     """
 
     total_slices = info['total_slices']
@@ -918,7 +919,7 @@ def _read_data(info, start=None, stop=None):
     for ch in info['chs']:
         data[:, ch['index']] *= ch['cal']
 
-    return data[:, info['order']].T
+    return data[:, info['order']].T.astype(np.float64)
 
 
 class RawBTi(_BaseRaw):


Test patch: