Repository: mne-tools/mne-python

Issue:
Annotation and crop - different onset points after reloading the saved croppend data
#### Describe the bug
I suspect that this issue is similar to the #9383.
The issue only occurs when a raw instance created with Annotation. Originally labeled data as brain vision files are work fine.
Cropping the instance, saving and reloading it shows the annotations in the original place before the cropping.


#### Steps to reproduce
```
import numpy as np
import mne
CH_NUM = 32
FS = 160

data = np.zeros((CH_NUM, FS * 20))
ch_names = [f'EEG{i}' for i in range(CH_NUM)]
ch_types = ['eeg'] * len(ch_names)
onset = [8, 12, 18]
duration = [2] * len(onset)
description = list(range(len(onset)))

info = mne.create_info(ch_names, ch_types=ch_types, sfreq=FS)
raw = mne.io.RawArray(data, info)
annotation = mne.Annotations(onset, duration, description)
raw = raw.set_annotations(annotation)

sess = raw.copy()
sess.crop(5, 18)
file = 'test_file_raw.fif'
sess.save(file, overwrite=True)
sess.plot(block=False)
r = mne.io.read_raw(str(file))
r.plot(block=False)
raw.plot(block=True)
```

#### Expected results
I expect that the annotated data should look like the same before and after saving it.


#### Actual results
The annotations are shifted...


#### Additional information
Platform:      Windows-10-10.0.19041-SP0
Python:        3.8.5 (default, Sep  3 2020, 21:29:08) [MSC v.1916 64 bit (AMD64)]
Executable:    C:\Programs\Miniconda3\envs\bci2\python.exe
CPU:           Intel64 Family 6 Model 158 Stepping 10, GenuineIntel: 12 cores
Memory:        Unavailable (requires "psutil" package)
mne:           0.23.0
numpy:         1.19.3 {blas=D:\\a\\1\\s\\numpy\\build\\openblas_info, lapack=D:\\a\\1\\s\\numpy\\build\\openblas_lapack_info}
scipy:         1.6.0
matplotlib:    3.3.3 {backend=TkAgg}
sklearn:       0.24.0
numba:         Not found
nibabel:       Not found
nilearn:       Not found
dipy:          Not found
cupy:          Not found
pandas:        1.2.0
mayavi:        Not found
pyvista:       Not found
vtk:           Not found




Discussion:
![bug](https://user-images.githubusercontent.com/47258484/124893315-e8691f00-dfda-11eb-93ea-572cdcc41f6a.jpg)

I confirm the bug... you can fix it if you set a meas_date with the raw eg with:

```python
import numpy as np
import mne
from datetime import datetime, timezone

CH_NUM = 32
FS = 160

data = np.zeros((CH_NUM, FS * 20))
ch_names = [f'EEG{i}' for i in range(CH_NUM)]
ch_types = ['eeg'] * len(ch_names)
onset = [8, 12, 18]
duration = [2] * len(onset)
description = list(range(len(onset)))

info = mne.create_info(ch_names, ch_types=ch_types, sfreq=FS)
raw = mne.io.RawArray(data, info)
annotation = mne.Annotations(onset, duration, description)
raw = raw.set_annotations(annotation)

raw.set_meas_date(datetime.now(tz=timezone.utc))

sess = raw.copy()
sess.crop(5, 18)
file = 'test_file_raw.fif'
sess.save(file, overwrite=True)
sess.plot(block=False)
r = mne.io.read_raw(str(file))
r.plot(block=False)
raw.plot(block=False)
```
I think this bug has been reported in the past and I suggested a temporary
fix
which is to set a meas_date so it's not None....

>

Hi just bumping this issue because it came up today for us @pmyers16

Do we perceive this to be an easy issue to solve?

Why can't we just modify the `meas_date` in `crop()` if it is set?
@adam2392 In the post I deleted and on which @agramfort reacted, I was trying to understand how to correctly solve this issue, as I also ran into it. In crop, there is basically an if/else statement that checks if you have a `meas_date` set or not. If you have one set, the statement is correct. If you don't it enters the other statement which creates the issue. No idea yet how to properly solve this as I don't know enough about the intention behind the `first_samp` attribute (I hope I remember the name of this attribute correctly) which stores the time of the first sample? I don't get why it might be different from 0 and I don't get to what the annotations onsets are related to (i.e. what is the origin).
> I don't get why `first_samp` might be different from 0 

This is a "feature" of Neuromag systems, where there is a distinction between "starting the acquisition system" and "starting the recording"

> and I don't get to what the annotations onsets are related to (i.e. what is the origin).

the origin is the beginning of the raw object instance, regardless of what it's `first_samp` value is.  

> the origin is the beginning of the raw object instance, regardless of what its `first_samp `value is.

So would that line be the culprit? annotation setter line 693 in mne.io.Base.py

`new_annotations.onset += self._first_time`
give a try to a PR and see if this breaks any test. If not and it makes the
new test case pass you win !

>

@agramfort If I'm not opening a PR yet, it's because I know this is not the only culprit, and I can't really figure out what else is messing up the annotations. Let's take your example, with slightly renamed variables.

```
from datetime import datetime, timezone

import numpy as np
import mne


CH_NUM = 32
FS = 160

data = np.zeros((CH_NUM, FS * 20))
ch_names = [f'EEG{i+1}' for i in range(CH_NUM)]
ch_types = ['eeg'] * len(ch_names)
onset = [8, 12, 18]
duration = [2] * len(onset)
description = list(range(len(onset)))

info = mne.create_info(ch_names, ch_types=ch_types, sfreq=FS)
raw = mne.io.RawArray(data, info)
annotation = mne.Annotations(onset, duration, description)
raw = raw.set_annotations(annotation)

# raw.set_meas_date(datetime.now(tz=timezone.utc))

raw_copied = raw.copy()
raw_copied.crop(5, 18)
file = 'test_file_raw.fif'
raw_copied.save(file, overwrite=True)
raw_loaded = mne.io.read_raw(str(file))


raw.plot(block=False)
raw_copied.plot(block=False)
raw_loaded.plot(block=False)
```

If the measurement date is set, with the current code we get:

```
raw.annotations.onset
Out[2]: array([ 8., 12., 18.])

raw_copied.annotations.onset
Out[3]: array([ 8., 12., 18.])

raw_loaded.annotations.onset
Out[4]: array([ 8., 12., 18.])
```

@drammock Already, this does not make sense to me. If the annotations should be relative to the beginning of the raw object instance, then the timings for `raw_copied` and `raw_loaded` should be `-5`.

Without the measurement date, with the current code we get:

```
raw.annotations.onset
Out[6]: array([ 8., 12., 18.])

raw_copied.annotations.onset
Out[7]: array([ 8., 12., 18.])

raw_loaded.annotations.onset
Out[8]: array([13., 17.])
```

With a wrong plot for `raw_loaded`. That tells me that either `raw.save()` or `mne.io.read_raw` runs the annotations setter, adding a second time the variable `self._first_time` (5 seconds).

And finally, if we comment the line 693 in `mne.io.base.py` and thus don't add `self._first_time`:

```
raw.annotations.onset
Out[10]: array([ 8., 12., 18.])

raw_copied.annotations.onset
Out[11]: array([ 3.,  7., 13.])

raw_loaded.annotations.onset
Out[12]: array([ 3.,  7., 13.])
```

This is actually making sense to me. Both `raw_copied` and `raw_loaded` have the same onset, relative to the beginning of the instance. The durations are also similar and make sense:

```
raw.annotations.duration
Out[14]: array([2., 2., 2.])

raw_copied.annotations.duration
Out[15]: array([2.     , 2.     , 0.00625])

raw_loaded.annotations.duration
Out[16]: array([2.        , 2.        , 0.00625038])
```

However, this is the plot obtained (for both `raw_copied` and `raw_loaded`):

![Screenshot 2021-09-02 at 10 42 21](https://user-images.githubusercontent.com/73893616/131813694-42e54ce1-46d7-4358-a96e-8078443cee23.png)

I do believe this line 693 in `mne.io.base.py` is a mistake and should be removed, but something else is messing up the annotations. Probably something in the plot which is using the `first_time` which would explain why the plot shows the 2 last events of duration `2` and `0.00625038`?
A few words following discord discussion. The assumption is that:

- Annotation onset should be relative to the beginning of the data reflected by `first_samp`. -> Different from @drammock comment: `the origin is the beginning of the raw object instance, regardless of what its first_samp value is.`
- With this assumption, the problem would be limited to the I/O roundtrip.

I'll have a look when I have time.
@mscheltienne I'm inclined to bump the milestone on this from 0.24 to 1.0 so we don't have to rush a fix in the next week that will potentially break other stuff.

Resolving patch:
diff --git a/mne/io/fiff/raw.py b/mne/io/fiff/raw.py
--- a/mne/io/fiff/raw.py
+++ b/mne/io/fiff/raw.py
@@ -202,6 +202,11 @@ def _read_raw_file(self, fname, allow_maxshield, preload,
             raw = _RawShell()
             raw.filename = fname
             raw.first_samp = first_samp
+            if info['meas_date'] is None and annotations is not None:
+                # we need to adjust annotations.onset as when there is no meas
+                # date set_annotations considers that the origin of time is the
+                # first available sample (ignores first_samp)
+                annotations.onset -= first_samp / info['sfreq']
             raw.set_annotations(annotations)
 
             #   Go through the remaining tags in the directory


Test patch:
diff --git a/mne/tests/test_annotations.py b/mne/tests/test_annotations.py
--- a/mne/tests/test_annotations.py
+++ b/mne/tests/test_annotations.py
@@ -254,6 +254,21 @@ def test_crop(tmp_path):
     raw_read = read_raw_fif(fname)
     assert raw_read.annotations is not None
     assert len(raw_read.annotations.onset) == 0
+    # test saving and reloading cropped annotations in raw instance
+    info = create_info([f'EEG{i+1}' for i in range(3)],
+                       ch_types=['eeg'] * 3, sfreq=50)
+    raw = RawArray(np.zeros((3, 50 * 20)), info)
+    annotation = mne.Annotations([8, 12, 15], [2] * 3, [1, 2, 3])
+    raw = raw.set_annotations(annotation)
+    raw_copied = raw.copy().crop(5, 18)
+    fname = op.join(tempdir, 'test_raw.fif')
+    raw_copied.save(fname, overwrite=True)
+    raw_loaded = mne.io.read_raw(str(fname))
+    for attr in ('onset', 'duration'):
+        assert_allclose(getattr(raw.annotations, attr),
+                        getattr(raw_copied.annotations, attr))
+        assert_allclose(getattr(raw_copied.annotations, attr),
+                        getattr(raw_loaded.annotations, attr))
 
 
 @first_samps