Repository: mne-tools/mne-python

Issue:
Annotations lost when concatenating Raws / incorrect shift calculation in _combine_annotations
Annotations are lost when concatenating (BrainVision) raws (on ``master``).

The bug seems to be in ``mne.annotations._combine_annotations`` ( https://github.com/mne-tools/mne-python/blob/master/mne/annotations.py#L362 ).
When concatenating annotations, the time shift calculation seems to be incorrect since the annotations of the second raw are shifted both by the length of the first raw (``one_n_sampes``) and the difference in the start times of the recording (``meas_date - two.orig_time``). Hence, the annotations of the second raw are shifted "too far" and removed when cropping the annotations in ``Raw.set_annotations`` (some annotations might be kept if the lengths of the Raws differs). 

I'm not sure if I'm missing something, but since annotations are supposed to be used instead of stim channels, this bug seems rather critical to me.

#### Steps and/or code to reproduce
On a high level, the bug manifests itself as

```
import mne
import numpy as np

# vhdr_files = ...
#%% load first raw
raw1 = mne.io.read_raw_brainvision(vhdr_files[0])
events1, _ = mne.events_from_annotations(raw1)
print(len(events1)) 
# 977

#%% load second raw
raw2 = mne.io.read_raw_brainvision(vhdr_files[1])
events2, _ = mne.events_from_annotations(raw2)
print(len(events2))
 # 975

#%% combine raw
raw1.append(raw2)
events_comb, _ = mne.events_from_annotations(raw1)
print(len(events_comb))
# 979
np.testing.assert_equal(events1, events_comb[:len(events1), :])
```
All but two events of the second recording are missing.

On a lower level, the shifting becomes obvious:

```
orig_time1 = 0
duration1 = 10.
sfreq = 1000
annot1 = mne.Annotations(onset=[0, 1], duration=[0,0], description=['test', 'test'],
                         orig_time=orig_time1)
annot2 = mne.Annotations(onset=[0, 1], duration=[0,0], description=['test', 'test'],
                         orig_time=orig_time1 + duration1)
#%%
# this call corresponds to the one in Raw.append
annot_combined = mne.annotations._combine_annotations(
    annot1, annot2, one_n_samples=duration1 * sfreq,
    one_first_samp=0, two_first_samp=0, sfreq=sfreq,
    meas_date=orig_time1)
print(annot_combined.onset)
# [ 0.  1. 20. 21.]
```


#### Expected results
The onsets of the annotations should be ``[0., 1., 10. ,11.]``.

#### Actual results
The onsets are ``[ 0.  1. 20. 21.]``



Discussion:
One could just replace ``orig_time`` of the annotations of the Raw to be appended with the the initial one, but I'm not sure whether that may have any side effects...

https://github.com/mne-tools/mne-python/blob/master/mne/io/base.py#L2130
```
r_annotations = r.annotations.copy()
r_annotations.orig_time = annotations.orig_time
annotations = _combine_annotations(
    annotations, r_annotations, n_samples,
    self.first_samp, r.first_samp,
    self.info['sfreq'], self.info['meas_date'])
In order to combine annotations, they should share the same `orig_time`.
So in your example, if we do `annot1+annot2` raises an error.

what you are suggesting is that when concatenating raw files, the annotations are broken. I guess that if both raw files share meas_date, that would be no problem.

I'll try to replicate


```py
    import mne
    info = mne.create_info(ch_names=10, sfreq=10.)
    raw_A = mne.io.RawArray(data=np.empty((10, 10)), info=info, first_samp=10)
    raw_A.info['meas_date'] = 0
    annot = mne.Annotations([.5], [.2], ['dummy'], orig_time=None)
    raw_A.set_annotations(annotations=annot)

    print(raw_A)
    print(raw_A.annotations)
    for a in raw_A.annotations:
        print(a)

    # raw_B = raw_A.copy().append(raw_A.copy())
    raw_B = raw_A.copy()
    raw_B.append(raw_A.copy())

    print(raw_B)
    print(raw_B.annotations)
    for a in raw_B.annotations:
        print(a)

```
This actually works as I would expect. (apart from the issue that I cannot assign `raw_B` as I would expect).

Can you share your vhdr files?

Sorry I forgot the output. 
```
<RawArray  |  None, n_channels x n_times : 10 x 10 (0.9 sec), ~24 kB, data loaded>
<Annotations  |  1 segment : dummy (1), orig_time : 1970-01-01 00:00:00>
OrderedDict([('onset', 1.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])
<RawArray  |  None, n_channels x n_times : 10 x 20 (1.9 sec), ~25 kB, data loaded>
<Annotations  |  4 segments : dummy (2), BAD boundary (1), EDGE boundary (1), orig_time : 1970-01-01 00:00:00>
OrderedDict([('onset', 1.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])
OrderedDict([('onset', 2.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])
OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'BAD boundary'), ('orig_time', 0.0)])
OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'EDGE boundary'), ('orig_time', 0.0)])
```

`raw_A` has a single onset at 1.5 (which is at 0.5 of first_samp)

and `raw_B` has 2 bad segments + at the junction, `meas_date` of the second is ignored and the onset corresponds to 2.5 as if the buffers where concatenated.
On your second example you cannot concatenate annotations with different `orig_times`. 

```py
    orig_time1 = 0
    duration1 = 10.
    sfreq = 1000
    annot1 = mne.Annotations(onset=[0, 1], duration=[0, 0], description='test',
                             orig_time=orig_time1)
    annot2 = mne.Annotations(onset=np.array([0, 1])+duration1, duration=[0, 0],
                             description='test', orig_time=orig_time1)
    print((annot1+annot2).onset)
```

This gives you `[0., 1., 10. ,11.]`, but combining annotations with a different `orig_time` is not possible. 
Thanks for the clarification @massich ! 

My ``Raw``s have different ``meas_date`` and annotations have different ``orig_time``s [1]. Hence, in my previous example ``raw1.annotations + raw2.annotations`` throws  (``ValueError: orig_time should be the same to add/concatenate 2 annotations [...]``).
However, ``raw1.append(raw2)`` succeeds: This calls ``_combine_annotations``, which tries to compensate for the differing ``orig_time``, but in addition to that also shifts the second ``Annotation``'s onsets by the duration of the first raw => incorrect offsets for the newly appended annotations.
I extended your example to illustrate that [2].

If I understand you correctly, ``raw1.append(raw2)`` (with different ``meas_date``) should currently fail---this would be unfortunate since imho it's a frequent use case to concatenate raw recordings and the basic logic is already there.
Probably, the parameter ``one_n_samples`` should only be passed to ``_combine_annotations`` if the second annotation's ``orig_time`` is ``None``!?

[1] I'm loading an experiment where each block is a separate Brainvision recording. 
In my script, I ``append`` the different ``Raw``s of the blocks into a single one and extract events (previously using the synthesized stim channel, and I encountered this problem when switching to extracting events from annotations after the latest deprecations). During loading of the annotations, the start time of the recording is read from the marker file (for ``info['meas_date']`` in https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/brainvision.py#L478 and for the annotations in https://github.com/mne-tools/mne-python/blob/master/mne/io/brainvision/brainvision.py#L226).

[2]
```python
import mne
import numpy as np
info = mne.create_info(ch_names=10, sfreq=10.)
raw_A = mne.io.RawArray(data=np.empty((10, 10)), info=info, first_samp=10)
raw_A.info['meas_date'] = 0
annot = mne.Annotations([.5], [.2], ['dummy'], orig_time=None)
raw_A.set_annotations(annotations=annot)

print(raw_A)
print(raw_A.annotations)
for a in raw_A.annotations:
    print(a)

#%% previous example, everything works fine
raw_B = raw_A.copy()
raw_C = raw_A.copy()
raw_B.append(raw_C)

print(raw_B)
print(raw_B.annotations)
for a in raw_B.annotations:
    print(a)
# <RawArray  |  None, n_channels x n_times : 10 x 10 (0.9 sec), ~24 kB, data loaded>
# <Annotations  |  1 segment : dummy (1), orig_time : 1970-01-01 00:00:00>
# OrderedDict([('onset', 1.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])

#------------------------------
# if the second orig_time is "after the end" of the first recording, the second dummy annotation 
# is lost
raw_B = raw_A.copy()
raw_C = raw_A.copy()

raw_C.info['meas_date'] = 3
raw_C.annotations.orig_time = raw_C.info['meas_date']
raw_B.append(raw_C)
# RuntimeWarning: Omitted 1 annotation(s) that were outside data range.

print(raw_B)
print(raw_B.annotations)
for a in raw_B.annotations:
    print(a)
# <RawArray  |  None, n_channels x n_times : 10 x 20 (1.9 sec), ~25 kB, data loaded>
# <Annotations  |  3 segments : dummy (1), BAD boundary (1), EDGE boundary (1), orig_time : 1970-01-01 00:00:00>
# OrderedDict([('onset', 1.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])
# OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'BAD boundary'), ('orig_time', 0.0)])
# OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'EDGE boundary'), ('orig_time', 0.0)])

#------------------------------
# if we second orig_time is still within the duration of the first Raw, the annotation is kept,
# but will have an incorrect onset
raw_B = raw_A.copy()
raw_C = raw_A.copy()

raw_C.info['meas_date'] = 0.5
raw_C.annotations.orig_time = raw_C.info['meas_date']
raw_B.append(raw_C)
# RuntimeWarning: Limited 1 annotation(s) that were expanding outside the data range.

print(raw_B)
print(raw_B.annotations)
for a in raw_B.annotations:
    print(a)
# <RawArray  |  None, n_channels x n_times : 10 x 20 (1.9 sec), ~25 kB, data loaded>
# <Annotations  |  4 segments : dummy (2), BAD boundary (1), EDGE boundary (1), orig_time : 1970-01-01 00:00:00>
# OrderedDict([('onset', 1.5), ('duration', 0.2), ('description', 'dummy'), ('orig_time', 0.0)])
# OrderedDict([('onset', 3.0), ('duration', -1.6653345369377348e-16), ('description', 'dummy'), ('orig_time', 0.0)])
# OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'BAD boundary'), ('orig_time', 0.0)])
# OrderedDict([('onset', 2.0), ('duration', 0.0), ('description', 'EDGE boundary'), ('orig_time', 0.0)])
```

it should not fail to concat raws with different meas_date as indeed it's a
really frequent usecase.
mne inserts a bad annotation to notifiy there is file boundary and it
should just work ie the orig_time
of the appended raw file should be adjust to just work.

>


Resolving patch:
diff --git a/mne/io/base.py b/mne/io/base.py
--- a/mne/io/base.py
+++ b/mne/io/base.py
@@ -2127,8 +2127,12 @@ def append(self, raws, preload=None):
         edge_samps = list()
         for ri, r in enumerate(raws):
             n_samples = self.last_samp - self.first_samp + 1
+            r_annot = Annotations(onset=r.annotations.onset - r._first_time,
+                                  duration=r.annotations.duration,
+                                  description=r.annotations.description,
+                                  orig_time=None)
             annotations = _combine_annotations(
-                annotations, r.annotations, n_samples,
+                annotations, r_annot, n_samples,
                 self.first_samp, r.first_samp,
                 self.info['sfreq'], self.info['meas_date'])
             edge_samps.append(sum(self._last_samps) -
diff --git a/mne/io/fiff/raw.py b/mne/io/fiff/raw.py
--- a/mne/io/fiff/raw.py
+++ b/mne/io/fiff/raw.py
@@ -112,7 +112,7 @@ def __init__(self, fname, allow_maxshield=False, preload=False,
                     self.annotations, r.annotations,
                     n_samples, self.first_samp, r.first_samp,
                     r.info['sfreq'], self.info['meas_date'])
-                self.set_annotations(annotations, False)
+                self.set_annotations(annotations, emit_warning=False)
                 n_samples += r.last_samp - r.first_samp + 1
 
         # Add annotations for in-data skips


Test patch:
diff --git a/mne/io/tests/test_raw.py b/mne/io/tests/test_raw.py
--- a/mne/io/tests/test_raw.py
+++ b/mne/io/tests/test_raw.py
@@ -281,3 +281,46 @@ def test_get_data_reject():
         assert log.getvalue().strip() == msg
     assert data.shape == (len(ch_names), 2560)  # shape doesn't change
     assert np.isnan(data).sum() == 3072  # but NaNs are introduced instead
+
+
+def test_5839():
+    """Test concatenating raw objects with annotations."""
+    # Global Time 0         1         2         3         4
+    #             .
+    #      raw_A  |---------XXXXXXXXXX
+    #      annot  |--------------AA
+    #    latency  .         0    0    1    1    2    2    3
+    #             .              5    0    5    0    5    0
+    #
+    #      raw_B  .                   |---------YYYYYYYYYY
+    #      annot  .                   |--------------AA
+    #    latency  .                             0         1
+    #             .                                  5    0
+    #             .
+    #     output  |---------XXXXXXXXXXYYYYYYYYYY
+    #      annot  |--------------AA---|----AA
+    #    latency  .         0    0    1    1    2    2    3
+    #             .              5    0    5    0    5    0
+    #
+    EXPECTED_ONSET = [1.5, 2.5, 2., 2.]
+    EXPECTED_DURATION = [0.2, 0.2, 0., 0.]
+    EXPECTED_DESCRIPTION = ['dummy', 'dummy', 'BAD boundary', 'EDGE boundary']
+
+    def raw_factory(meas_date):
+        raw = RawArray(data=np.empty((10, 10)),
+                       info=create_info(ch_names=10, sfreq=10., ),
+                       first_samp=10)
+        raw.info['meas_date'] = meas_date
+        raw.set_annotations(annotations=Annotations(onset=[.5],
+                                                    duration=[.2],
+                                                    description='dummy',
+                                                    orig_time=None))
+        return raw
+
+    raw_A, raw_B = [raw_factory((x, 0)) for x in [0, 2]]
+    raw_A.append(raw_B)
+
+    assert_array_equal(raw_A.annotations.onset, EXPECTED_ONSET)
+    assert_array_equal(raw_A.annotations.duration, EXPECTED_DURATION)
+    assert_array_equal(raw_A.annotations.description, EXPECTED_DESCRIPTION)
+    assert raw_A.annotations.orig_time == 0.0